Data—the Biggest Constraint in DevOps and CI/CD

Gene Kim is the WSJ bestselling author of The Unicorn Project.  He’s also the founder and organizer of the DevOps Enterprise Summit.

Gene shares his thoughts on the role data plays in DevOps, the common thread of data-related challenges that run through DevOps and digital transformation, and what inspired him to invest in Delphix.

In the spirit of debate season, Gene and Jed discuss their differing ideologies behind which approach yields the fastest path to success for an organization. Is bottoms up or top down the right approach to innovation? Watch the interview to find out!

Jed: How did you go from a founder and CTO to an award-winning author spearheading the DevOps movement?

Gene: I've been studying high performing technology organizations since 1999, and that's a journey that started back when I was a CTO and founder of an information security company called Tripwire. The high performers were the organizations that simultaneously had the best project due date performance in development, the best operational stability, and the best posture in security compliance. We wanted to understand: How did they make their good-to-great transformation so that other organizations could replicate those amazing outcomes? The biggest surprise was how DevOps is not just being used by the tech giants: Facebook, Amazon, Netflix, Google, Microsoft. But that it is also being adopted by large complex organizations, the most well known brands across every industry vertical. And I have no doubt that when those organizations, where 18 million developers reside, are as productive as those working for the tech giants, they will create trillions of dollars of economic value per year, and so that's why I'm so excited to see how all those practices are really being adopted.

Jed: High performing organizations not only have high speed, they also have high quality and high security. Most of the time, people think these are trade offs. How do you get all three together?

Gene: One of the most astounding findings came from the State of DevOps research I did with Dr. Nicole Forsgren and Jez Humble, which was a cross-population study of 36,000 respondents that spanned over six years. For six years in a row, it has been decisive that you can be more agile and be more reliable at the same time. It says that the only way to get real reliability is to do small deployments more frequently. You also end up with better security outcomes when you integrate information security objectives into everyone's daily work. That also correlates with better organizational performance. They are more likely to exceed profitability, market share, and productivity goals. They're better able to achieve mission and organization goals—whether it's quality, quantity, or customer satisfaction.

Jed: How much do you see bottom up versus top down innovation driving change?

Gene: I think they all mostly start bottom up. Despite the fact that we have a lot more senior people presenting [at the DevOps Enterprise Summit], the sweet spot is a third or fourth line manager. I think it's because it's at that director level, where they're close enough to the work, where they can see horrendous things that shouldn't be happening. They have a notion there must be a better way to do things. They're also senior enough to see the business perspective, and can make that judgment about where is a good place to start.

Jed: In Silicon Valley, it's often product CEOs who start companies. Contrast what's happening in the world of Silicon Valley and tech giants to these mid-level, bottom-up movements you're seeing in big enterprises.

Gene: It’s not usual for leadership to come from a technology background. I think that is a fundamental handicap of these traditional enterprises. But I think that is changing. In the Harvard Business Review paper that came out of the Apple University program, they said that one of the things they looked for in Apple is a leader who could work three levels down. These technology leaders have to be close to the work.

Jed: When you look at high performing organizations compared to average or lower performing organizations, what's the divide in quality, secure output?

Gene: It is orders of magnitude. High performers are deploying multiple times per day versus monthly or quarterly deployments for non-high performers. What's been amazing over the years is that the percentage of those organizations that are high performing keep growing.

Jed: In Accelerate, you discuss some of the critical elements to be a high performing organization. One of the things you talk about is version controlling everything. But you explicitly say don't version control the data.

Gene: In an ideal world, everything should be version controlled. Version control gives us safety. It gives us the ability to know what happened, to roll back to an established, known state. I think we're just now realizing to what extent we can do that with data. One of the things that we try to explore in my most recent book The Unicorn Project is this bigger problem around data. We now live in a world where 30-50% of company employees use or manipulate data in their daily work. That's arguably an even larger population than developers. I believe the value that gets created when we can use the same software development practices for data will be even larger.

Jed: Why is immutability important when it comes to releasing software? How does that translate into data?

Gene: For 20 years, I self-identified as an ops person. It was my observation that operations where all the saves were made. It was ops who saved us from terrible developers who didn't care about quality and pushed into production anyway. But in the last four years, I've changed my mind. I now no longer self-identify as an ops person, but as a developer. That was really from learning a functional programming language called Clojure that runs on the JVM. In functional programming languages like Clojure, it forces the immutability of variables. You're not allowed to change them once you create them. You end up with a far safer world when you can never overwrite data in the database. Life is better when you have immutability, whether it's in code or in data.

Jed: Finally, tell me what made you interested in Delphix?

Gene: The aha moment for me was that DevOps has transformed almost every aspect of technologies, whether it's development, QA, or operations. There's no doubt in my mind that there are going to be equally dramatic changes happening in the data space. At the heart of every data center—whether it's in the cloud or on premise—is typically a database. It was such an eye opener to see what could be done in the database space as well. I'm really excited to see all this innovation for something so critical.