Pick a random romantic comedy to stream, and odds are weirdly good that one of the leads is an architect. It’s unlikely they’re a DevOps architect, though. But while the job hasn’t yet captured the public imagination, it’s an essential part of the tech world. Without it, DevOps engineering would devolve into chaos.
Before we get into why that’s the case, let’s define DevOps. Linguistically, it’s a mashup of “development” and “operations.” Functionally, DevOps teams have three core responsibilities, according to Mustafa Khokhawala, senior DevOps architect at Stocktwits. One of them is infrastructure automation, or automating the allocation of hardware like servers and databases. Another is continuous integration — constantly testing new code and merging it into existing software. The last, reliability engineering, involves building smart monitoring features that alert engineers to security and functionality issues.
That may sound simple enough, but especially at an enterprise scale, it requires large teams of DevOps engineers working in sync. A DevOps architect essentially engineers the DevOps engineers, setting clear standards that ensure alignment.
This isn’t always a full-time job. At some companies, DevOps engineers share architecting duties, or the CTO’s office takes on the job. It’s typically more mature companies, striving to scale an already-successful model, that carve out a specific architect role.
However it’s created, though, DevOps architecture is important. Without it, a company could end up with two automated deployment pipelines for a given job. (Or, worse, one-and-a-half.) An enterprise could also find itself with a glitchy code base, as integrating new code is tough when 40 engineers make changes simultaneously.
Architects plan for the future, too, determining which new DevOps technologies a company invests in and how current DevOps best practices can help the company achieve its long-term goals.
Three DevOps architects, including Khokhawala, spoke with Built In about their experiences in the role and shared tips for those who aspire to do what they do.
How did you get your first role in the DevOps architect space?
Dan Knox
Dan KnoxDirector of Engineering at G2
My first role like that would have been at Market6, two jobs ago. We started a new team, a completely new product. My mantra was, “Hey, let's not just do manual pushes. I bet we can get to the point where we can deploy rapidly and deliver more value." That went up to the C-level. We got buy-in, and we deployed so many times. Everybody was happy. That was a pretty interesting project. This was still running on physical hardware, it wasn't even cloud-based.
Even with my first product engineering role, though, DevOps became my concern pretty fast. There’s a DevOps motto: "If something hurts, do it more often.” Because eventually, you’ll automate it. Even in that early role, there were a lot of things where I thought, "Oh, this sucks, let me go write a script so it doesn't." Eventually, if people were doing some bit of work and they couldn't tell me a story of how it was going to be deployed, I’d say, "Okay, let's figure this out. Let's actually map this out and figure it out before you try to just shove the code in manually." Even back then, I could see the value in automating deployment, and I could walk people through getting to a point where they had DevOps just native in everything they did.
Mustufa Khokhawala
Mustufa Khokhawala Senior DevOps Architect at Stocktwits
I started out as an application developer back in the ‘90s, and through the years, I adapted to new technologies, like Java. But five years back, I needed to do something more radical to stay relevant. I had two options. One was a data science specialization, but I decided DevOps would come more naturally to me. I had done a little project management, some testing. I’d worked in siloed companies and seen what happens. I've had at least a taste of all the roles in the software development life cycle.
I already had about 70% of the knowledge I needed for DevOps. I just had to understand the other principles: how do you improve the business delivery, how do you address the cultural aspects of getting the development team to talk to the operations. I did a couple online courses on LinkedIn, and I did small experiments at work to see whether the DevOps theory worked in practice.
I think it also helped me get that first formal DevOps role that when I interviewed, I had to look at case studies and identify where the problem lies. I knew exactly where the problems were. I didn't have experience solving the problems, but unless you understand where the issue is, there's no point in trying to resolve it.
Aleksandr Lalayants
Aleksandr LalayantsDevOps Architect at Verge Health
It was an internal growth thing. After working a couple of places, I ended up at Verge as a web operations engineer and my team and I spent a lot of time fighting fires — reacting to pressing matters at hand. Over time, I started focusing less on how we could react to the next fire fast and more on what we could do to prevent fires. I pitched a monitoring solution. We worked on some notification systems, establishing clearer backup and risk recovery processes, which invariably leads to less fires. That gave us more time for projects that even further decreased the time we spent reacting, and let us spend more time addressing problems proactively. I found myself focusing more on the longer-term, larger picture.
I think it helped that I graduated college with a degree in physics, which directly has very little to do with what I do now, but I learned to solve very large and very complicated systems. I think that's one of the best things I came out of school with: an ability to break down a large block into small components, drill down and identify where we're really constrained — to shine a light into that black box.
For people who want to become DevOps architects, what are key first steps?
Knox: I think it’s becoming a thought leader in your space — in any tech role, not just DevOps. Even when you can influence the people on your team, it’s another step to influence the broader team and the company at large on a technical level. This is really how you get into architecture, though, and where an architect makes the most value: they change or standardize how a company moves technically, and make the processes for delivering code more efficient.
That said, I would not expect somebody to move straight from application architect to DevOps architect. I would consider those different roles requiring different skill sets. Most people work as DevOps engineers before they become DevOps architects, and most people in a DevOps engineer role start in an engineering role first.
Khokhawala: You definitely need to spend at least three to five years as a DevOps engineer, understanding the tooling, the culture, the practices and processes, the cloud business and how it’s taking shape. You have to spend some time in application and systems now, too, because the line between those areas and DevOps is blurry. But you need to go micro first, to get that higher-level picture of what’s going on. If you’ve worked in multiple industries, from health care to finance to shipping, then you also have a better sense of the variations out there.
Lalayants: Any architect position invariably comes back to working with product managers and working with the business component of your company. Oftentimes engineers don't speak business, and the business side does not speak engineering. As a DevOps architect, you have to be able to understand the business side. Everything ultimately revolves around delivering value to clients, delivering results. Without that, nobody's going anywhere.
At the same time, you have to be fluent in the technologies. Learn the tooling, how to automate something, learn a tech base. To be an engineer who personifies the principles of DevOps, you have to be able to read code, and you have to be able to script and understand the hardware side of stuff, even if you were working in the cloud. As far as languages go, Python is popular. Be an engineer first and then develop an understanding of large stacks and their long-term progress.
What are a DevOps architect’s core responsibilities?
Knox: An architect is not necessarily going to be writing code. They're going to be setting up the practice by which work is done. They’re building a pattern so that a company’s complex development process doesn’t become chaos.
This generally looks like an engineer saying, "Hey, we're spinning up this project. We probably need to talk to our architect in order to know how it's going to be deployed, how we're going to set up our local machines in order to function as a team.” Then the architect would come in and say, "Hey, this is how we do it here.”
Khokhawala: In my opinion, a DevOps architect is someone who designs, implements and promotes DevOps practices within the organization. So basically working out or designing the practices, processing and culture necessary for DevOps to work smoothly, being agile and getting things out of the door quickly — that’s my role in a nutshell.
I’m supposed to create and keep a standard that cuts across multiple development or product teams where they have slightly different requirements. The architecture comes from looking at patterns across multiple teams. How can we replicate what works in one place in other places? What works for the organization as a whole, not just for individual teams?
Lalayants: I help structure and mold the long-term direction of the DevOps team as well as being an engineer myself. I would say I spend about 70% of my time doing what a regular engineer would do. I automate, streamline, design, implement, fix problems with developers.
Then I spend about 30% of my time on the architect role, discussing and thinking about our larger and longer-term plans. I focus on the roadmap for how we want to develop our platforms internally, and I push for initiatives that might otherwise get lost due to de-prioritiziation. From an operations standpoint, it’s hard to argue for a feature when it goes out in a year or 16 months. I also read about new technologies, keeping in mind the rest of our stack. There are so many moving pieces that are connected, I have to make sure that a product that could benefit our tool chain doesn't break whatever else we have.
What’s most rewarding about the role?
Knox: To me, the most rewarding part is seeing the time from concept to delivery just drastically drop when you get the right architecture in place. You win for the company; you win for yourself; you enjoy coding because you know that your code’s not going to sit somewhere for three months and then eventually get deployed.
Khokhawala: At Stocktwits, it's the freedom, the flexibility and the calculated risks that I can take on my own without getting 10 levels of approval. That’s what I really like. Maybe six or seven out of 10 times I may succeed, and I may fail on a few occasions, but that's okay. I’m still accountable — if something doesn't work, I get it back working as soon as possible. But I've come from bureaucratic setups, and Stocktwits is a very refreshing break for me.
Lalayants: When I find something that can really benefit us. Part of the reason I do the job is you get to play with some really cool tech, and I'm a nerd. I enjoy that. When I find something and it knocks my socks off, I am like, "Wow, this is awesome.” Or when I present it to the department, and I can just see people’s eyes light up because it will make their jobs easier.
For example, a couple of years ago, I spearheaded the adoption of Kubernetes at Verge, and that was one of my biggest triumphs. I remember when we first demoed it to the group: how you can deploy an application, scale it out seamlessly behind the load balancer, have one of the pods crash, come back right up, and the user doesn’t notice a single thing. It really wowed people.
What’s most difficult about the role?
Khokhawala: One challenge is that I need to progress the organization technologically, make it more and more mature in terms of the DevOps practices, and obviously there are a lot of tools and technologies coming out at a very rapid pace. So catching up with that is a bit of a challenge. I basically try every single thing and then decide what's best for the organization. I need to be a little cautious, but I can't take forever. If it's a three month exercise, I need to finish it in three months, but at the same time I need to do proper due diligence.
Lalayants: The pace at which the landscape changes is difficult. It’s not even about picking the right product — it’s about knowing what's out there. There are so many technologies competing with each other that have a different approach to solving the same problem, or have different design principles behind them. Everything always comes down to how does it work at scale. My team and I, we work really hard to try and keep up with everything. That's probably the biggest challenge, though — the speed.
What resources can people rely on to learn about DevOps news and trends?
Knox: I would definitely say stay up to date on DevOps articles. I know Puppet does some, more or less explaining what everybody's seeing and doing in the field. And then if you want to understand something specific — Why would I ever use Kubernetes? Why is Amazon putting that into their ecosystem? — it's really just kind of researching and following articles around that topic.
Khokhawala: I took a LinkedIn Learning course called DevOps Foundation. It's by two gentlemen named Ernest Mueller and James Wickett who give you a good overview of the DevOps world, but it’s up to you to get more hands-on and dive into each of the areas.
There’s also a weekly newsletter called DevOps Weekly. And there used to be a podcast called DevOps Cafe that is not active right now, but there are a good 50-odd episodes published. It was a talk show, basically, with celebrities from the DevOps world. You can also go to the meetups at DevOps Days. The good news is that at DevOps Days, there's no marketing and no fancy presentations by vendors or sponsors. It's purely engineers talking to each other, discussing their problems, what works, what doesn't work.
Lalayants: I'm a member of Linux Academy. They have a lot of courses out there that are good in terms of organizing and helping you study for certificates, if you want to go that way.
They do a pretty good job of covering from-the-surface stuff. On the flip side, if you want to understand what the DevOps philosophy is all about, there's a book out there called The Phoenix Project, and it's often mentioned in the community. It’s a novel, but it outlines the principles upon which modern DevOps is built. It’s a first step towards really understanding what the point is.
Answers have been condensed and edited. Images courtesy of Shutterstock and interviewees.