1.5 — DevOps with Nicole Reineke and Hanna Yehuda

Bill Pfeifer:

Hello and welcome to The Next Horizon, a Dell Technologies podcast. I’m Bill Pfeifer. Together, we’ll be talking about emerging technologies, their potential to impact society and what you need to know today.

Our guests today are Nicole Reineke and Hanna Yehuda, who both worked for the Dell Technologies Office of the CTO and have recently completed some original research on DevOps. We thought it might be fun to talk to them a bit about that effort. Nicole and Hanna, thank you so much for your time today and welcome to The Next Horizon. To kick off our conversation, could you tell us a bit about how you came to work together on this topic? What led you to take the time to do original research into DevOps adoption and how did you get connected to one another? I get the sense that you’ve been working very closely with one another on this topic. I assume you have complimentary skill sets. What brought you together to work on the project?

Hanna Yehuda:

I would love to answer this one. I’m working for so many years on research and always focusing on the personas in their users. The Office of the CTO start to work with simply with the user experience design team. I think this is such a blessing for the company because we bring two different perspectives when we talk to customers. When we call interview customers, she listened to the technology side of discussion. When I listen to the customer and ask question, I try to understand their challenges. The aspect of what is the best experience for them to do their work, what are the right people that needs to be in the team to be successful. Because at the end of the day, we are not providing just technology or a product or an API or just a solution. We need to look at the whole solution, the whole experience, which is what people needs to be in the team.

If for example for these DevOps research, whether the right team that needs to have the right persona inside to be successful in addition to what the technology needs to be there to come together and bring the success for this group who wants to go and develop in this way. I’ve seen so much open ideas and we found so many new directions that needs to be changed and achieve better solutions for our customers to come to this research and bring these ideas together.

Nicole Reineke:

Hanna’s absolutely right. We started working together on a completely different project. We were actually assigned to look at the user interface for managing heterogeneous data center environments. As we went out and spoke with customers, Hanna was asking her user-focused questions and I was asking technology-focused questions. We came to the realization that as customers were explaining what they needed to us, they were actually identifying problems that we hadn’t started thinking about. In many cases during this management user interface research, people started talking about infrastructure as code or automating the infrastructure. In some cases, started throwing out the term DevOps.

Hanna started pushing on the question, well, what do they really mean when they say DevOps? As a technologist, sometimes I approach a problem and I assume I know what the customer means. Hanna has this great perspective where she understands that users use language differently. As she was pushing for us to investigate the use of the word DevOps more clearly, we identified this whole other area of research and understanding how people wanted to manage their infrastructure. Not just now but in the future so that they could eliminate a lot of the bottlenecks that were making their internal consumers unhappy. That was really the genesis of this partnership is us realizing that each of us brought such a unique perspective that what we were able to create together far exceeded what either of us could create alone.

Hanna Yehuda:

Absolutely.

Nicole Reineke:

If you can’t tell, she’s a delight. This has been quite a wonderful experience and we’re looking forward to bringing the partnership to so many new areas of research.

Bill Pfeifer:

Sounds like a really great partnership. I’m glad you guys got together. I love the research that came out of it. In reading through some of the documents around the study, I caught your mention that customers are looking to Dell Technologies to help in their transition from manual operations to automated and DevOps operations. Now I’d imagine that through the course of your research, you came across some common stumbling blocks on the path to DevOps. Could you talk some about what some of those stumbling blocks might be and how folks who were thinking about moving to DevOps but haven’t yet could build their strategy to avoid those pitfalls or take them into account?

Nicole Reineke:

Some of the most common stumbling blocks we came across involved the ability to clearly articulate the value. Someone was trying to identify or trying to bring to their organization when they were adopting these new technologies. There are patterns where you adopt a new technology for the sake of adopting a new technology or for bringing a buzzword into your organization. We see that a lot with some new startup comes along, people buy the software and they don’t necessarily implement it and use it in the long run. We found that the same was happening with this infrastructure automation capability that there are some people who are looking at the technology trying to adopt it, not really understanding the motivation or the value that it was going to bring to the organization, so not committing 100%. the automation they did put in place didn’t add enough value to continue to support for the long run. That was a big stumbling block.

Another stumbling block was identifying the tooling that needed to be put in place for success. They didn’t clearly understand or evaluate which things they needed to use in order to make this kind of an initiative successful. This is a massive initiative adopting infrastructure as code or adopting DevOps or adopting some kind of automation process. While it seems simple at first, it’s actually a big investment. You have to understand what things have to go together in order to make it successful. Not clearly articulating the tool set that you were committing to in order to implement it on site or within your organization was a big stumbling block.

Additionally, not having people in the right places was a big stumbling block. Not having executive buy-in or not having an IT director in place who was owning that tooling or owning the vision. Sometimes not having an architect in place who could help the rest of the team build things that worked together in the long run, that was a massive stumbling block. Again, having the right organization or the right size teams or identifying the right projects to do adoption on first was a big stumbling block. If you try to automate everything, you’re unlikely to succeed. If you can siphon off or identify functions or services that need to be automated first and put in place goals and teams to work together to siphon off those small portions and build success on success, that was more likely to succeed.

Hanna Yehuda:

Another thing to add is about the AIOps. Companies who just automate realize that it’s not enough just to automate. They have to also go into automate also and add more AI into their decision making to reduce the time and the bottlenecks off their whole process from engineering requesting changes from IT. The bottlenecks come mostly in the IT side where they need a process to get decisions making and so on. Where the AIOps come into place here is to reduce that time and to automate and come to conclusion quickly and fast.

Nicole Reineke:

One example of this type of a story is there was an IT team that provided infrastructure as code. The way they did that was when somebody made a request into ServiceNow, the IT team would approve that request and then the backend would automate the provisioning of what the user requested. From the IT perspective, that was automation. When they reached out to their end users to see if it increased their end user satisfaction, their end users were just as dissatisfied as they were before that automation was put in place because the IT team didn’t identify that the user experience started at the point the user requested the provisioning of the application. They assumed that the automation of their job being easier would increase the satisfaction of the end user. End user had no insight into that automation adding value. That is a massive bottleneck that they didn’t identify. It is the difference between somebody doing infrastructure automation, simple DevOps or some kind of an intelligence DevOps, which some people refer to as AIOps.

Bill Pfeifer:

Good perspective. Looking a little bit in a different direction at the skill set that we need, a few times in the report you bring up SREs, site reliability engineers. In one spot, you refer to them as unicorns because there aren’t nearly as many of them in the industry as companies want to hire. Based on your interactions, do you have any suggestions for folks who might be interested in becoming SREs? How do they earn those unicorn horns? And then for customers that are looking for SREs, where should they look to hire or develop or build their own SREs?

Hanna Yehuda:

Maybe before we drill into this answer, we need to explain to our audience who is in SRE. In a nutshell, it is a combination of an expert IT generalist with the ability to code as a very advanced developer and knows how to write scripts and understanding repetitive areas that he needs to go and automate. This combination is very unique. I think that’s the first thing that we need to understand.

Nicole Reineke:

As Hanna said, it is a skilled developer and a skilled infrastructure specialist. We have seen examples of successful site reliability engineers come from those two different places. We have absolutely seen cases where an IT specialist does a transitional step into understanding automation and then continues their education along the coding path so that they can become a site reliability engineer that provides the automation value as well as the continual monitoring and improvement of that pipeline. We have also seen many cases where an application developer or a software engineer is developing software and starts to understand more information about the infrastructure themselves. They actually move a secondary path where they begin to learn more about networking or storage or servers and develop that skill set and become a site reliability engineer from that perspective.

You can come at it from many different ways. What we loved is that there were several large organizations we spoke with that actually proactively identified engineers within their organization from many different groups and started providing them the training to implement coding or to implement infrastructure as code and build up that skill set that way and move them into a site reliability engineering role. That was exceptional and we really applaud the efforts that they were making. Several of them have formal processes in place and formal programs in place to enable that. They do realize it’s incredibly hard to find site reliability engineers in the wild. Our recommendation is to build on the skill set that you already have. If you come from the IT infrastructure perspective, start to build up that coding skill set and take advantage of opportunities to participate in the open source community if you don’t have the opportunity to do it within your own organization.

Hanna Yehuda:

What was also surprising to us in the research when we talk to customers is that every company that we talked to, they said, “We don’t want to replace our people. We actually appreciate our IT generalist or server administrators. They’re really important for us to help them to grow.” The same as in the development area where you worked in their waterfall procedure and now we moved to agile. You had to go and train your developers, you didn’t get rid of them and replaced them with new ones. Right? In this situation, the same thing. They really want to help you guys to grow and to start to automate slowly with small pieces where you can see low hanging fruits and then from there grow and get more and more knowledge on how to do it across other domains than your expertise.

Nicole Reineke:

We did have cases where we spoke with the organizations that tried to hire in SREs from the outside to build the organization from scratch. There was one organization that we spoke with that expressed frustration at a mandate to hire site reliability engineers in from the outside. They spent 18 months with racks open for this position before they started a infrastructure as code or a DevOps initiative. What they found was they could not fill the position because they did not have access to enough candidates who had all of the skills that the position opening was requiring. When they did find that resource, the pay scale was out of range with what the opening was willing to pay. After 18 months of not being able to fill this position, the company finally agreed to upskill the existing team members and they were able to start doing an implementation. They were starting to see success using the existing team members.

That was one example of a great shift in perspective where we once were seeing this idea that you had to hire in from the outside. We’re now seeing proof points that you can in fact upskill and train your existing team members and increase the value all around.

Bill Pfeifer:

Well, it’s pretty encouraging. It’s good to hear that the best practice and maybe one of the more common ones is to have companies supporting their own internal folks and building SREs and moving people forward rather than just hiring new folks from outside who don’t have the background and the culture and all of that good stuff. That’s how you avoid the negative side of getting from regular operations to DevOps with SREs, but are there maybe some recipes that customers typically follow to get into DevOps or some common patterns that you’ve seen customers use successfully to get started down the DevOps path?

Nicole Reineke:

If you have low or minimal skill set in doing automation or in doing coding, you can create or adopt flow-based interfaces. There are partners of ours who have created these. If you have a VMware only environment, VMware does have a flow based interface where you can start automation, or if you have a mixed environment, there are tools that we provide that can help create a flow based implementation. If you have people on your team who are expert developers or who have experience in it, you can look at the complexity of your environment and make an informed decision. If you have highly expert developers with a complex environment that’s mixed, you can start to do some script based implementations where you’re using things like APIs. You don’t have to adopt what’s commonly referred to as an orchestration platform in order to achieve this.

If you have a fairly simple environment and you have expert developers, you can do manifest based or other implementations of this automation. There are a lot of different options. Don’t think that if you’re going to adopt DevOps that it has to be incredibly complex. There are different ways to implement it to meet the needs of the organization and also match up with the current skill set of the organization.

Bill Pfeifer:

From a company perspective, you also mentioned later that of the customers who participated in the research, over 50% are pursuing DevOps. Now, I think there’s a lot of information in the market about why people move to DevOps, but did you get a sense of why the less than 50% we’re not moving to DevOps?

Nicole Reineke:

In many cases, it was about the inability to achieve the value that they needed to achieve to make this kind of a time investment. It is a big investment. It is a massive shift in thinking. In the case of large organizations who have infrastructure that is absolutely critical to their organization, they didn’t feel like they had the ability to make this kind of a shift without interrupting their existing business practices. In some cases, it was a we can’t do it right now. It’s just because they couldn’t figure out the value they could add that would make it worth the interruption that may happen to their infrastructure or to their business. That inability to articulate the value was really one of the leading problems.

Another problem that we were able to identify was that they didn’t have the skill set or they didn’t have the knowledge of how to start this kind of an implementation.

Hanna Yehuda:

I think that what we have also seen, which I was actually very surprised from customer perspective, is the fear of change. They actually was the way that they work for years. Think about storage administrators, IT generalist, suddenly they need to go and change everything they need to do from yesterday to tomorrow, which is starting to code. They don’t want to be developers. That’s what they say. But on the same time, if they think about it as a small incremental changes that they need to make, they don’t need to overnight become developers. They can start small by taking some areas where they can improve by automating them. And then later on, continuing to add more and more. But basically, that’s something that we found from customers about the fear of change.

Another aspect is huge organizations, that every hour the business is down is a big loss of money. For them, any kind of change is high risk. High risk means the system is down, losing money and why should I do that? Until they are not getting into the place where they are under the pressure to do it because of competitive, because of workloads, they don’t run in the expectation time and so on. They will not make a change. But, the day will come and they will have to do it.

Bill Pfeifer:

Yeah, I guess it’s a very different way to run your operations and your infrastructure in general. The fear of change would be a pretty powerful factor there. Now, the research primarily based on US customers like 60% or so, but you also had participants from Europe, Asia, South America, pretty global perspective, which was great to see. Did you see any notable differences between regions or is the move toward DevOps happening pretty uniformly around the world, across geographies, across verticals, across company sizes and types?

Nicole Reineke:

We didn’t notice any direct correlation between the physical location of a company’s headquarters and the adoption of DevOps. In many cases, it was vertically driven. Organizations that had a resistance to change, it was fairly consistent across the size of the organization plus the vertical that they were serving. That seemed to be a closer correlation to the adoption of DevOps than the actual geographic location. There were forward-looking companies and forward-adoption companies within most of the geographies that we spoke with. A lot of it had to do with the pressures from competitors as Hanna has mentioned as well as the pressure to perform better within their own organization.

Hanna Yehuda:

Basically, I would just add that we’ve seen that specifically just as an example, that companies that are more tech-oriented, they will move faster to DevOps compared to companies that technology is not a strong thing within the company. The other aspect that we have seen depends on what kind of projects they run. If the project that they have has to be using DevOps to deliver in a real-time solution to the market on the website, then you will see in that area DevOps. While in the same company, in projects that not that critical, you will see them behind and not using the DevOps or partially using DevOps. That’s what we’ve seen with regards to the differences between companies.

Nicole Reineke:

Yeah, I agree completely. There was a vast difference between a company that was running real-time stock trading versus somebody who is doing a backend medical research, which was not… It was time sensitive, but not necessarily as time sensitive as losing millions, if not billions, of dollars in a millisecond.

Hanna Yehuda:

Yeah.

Bill Pfeifer:

Scary thoughts there. Moving that much money that fast. But yeah, I guess you would definitely want to automate that.

Nicole Reineke:

Right.

Bill Pfeifer:

Now, you did this research because you’re obviously very passionate about how and why the industry is moving to DevOps. So now your research is done, do you have a sense of what comes next in the industry? Right? Once we have enough of these SRE unicorns and enough enterprises have moved to DevOps and infrastructure as code that it’s just another operating model, what’s going to be the next big process advancement?

Nicole Reineke:

We are really excited about our next set of research. We’re actually already working on the investigation into the complete disaggregation of application behaviors and functional performance from the infrastructure. One of the end results of that is this thing called NoOps, which is not the elimination of operations as a team or as a skill set, but the disaggregation of underlying operational infrastructure from the utilization of that infrastructure. We’re looking forward to publishing our report on NoOps.

Hanna Yehuda:

Yeah. From a research perspective, if you ask, “What the UX research can bring into this topic?” Think about it, what kind of team do we need to have in the future? What kind of skill set do we need to have for these people to run this kind of solution, of NoOps? That’s the part that I’m excited when I come to this research and ask my questions.

Nicole Reineke:

Yeah. We’re also looking at documenting the process for the adoption of this new technique of running your infrastructure. We have examples of forward-looking customers who’ve actually started to adopt this disaggregation. We’ve seen successful implementation and use cases. We’re looking forward to sharing that.

Bill Pfeifer:

Well, I think I’m completely unqualified to do modern operations. NoOps sounds like just the perfect job for me. I like it. Thanks so much for spending the time with us and giving us a view into your world. We definitely appreciate your insights and look forward to hearing more about what’s going to be coming out of the Office of the CTO’s research teams. We’ll be looking for some great NoOps conversations in the future as well.

For those of you who enjoyed this podcast and want to know more about Nicole and Hanna’s DevOps research work, their white paper has more detail than we covered in our conversation. You can find it attached to this podcast page at www.delltechnologies.com/nexthorizon along with future podcasts and some other great content from the smart folks at our Office of the CTO.

Thank you all for listening to The Next Horizon, a Dell Technologies podcast. We appreciate your time, interest, and attention. I hope you’re as excited as we are about the great innovations that are coming out of our Office of the CTO. Be sure to subscribe to the podcast, either through your favorite podcast app or through the website at www.delltechnologies.com/nexthorizon so you don’t miss any great new content. I look forward to seeing you again for upcoming episodes. I’m Bill Pfeifer and this is The Next Horizon.