Last year, an argument over whether developers should deploy code on Fridays spawned long, thoughtful essays and several funny memes.
The opposition to deploying code on Fridays stems from concern over protecting developers’ personal time. Dave Mangot, who runs Mangoteque, which helps companies improve their DevOps processes, was one of those opposed to Friday deployments. He wrote in a post that holding off from deploying on Fridays is a way to protect work-life balance and to reduce employee stress.
Jeopardizing people’s weekend plans “doesn’t show a lot of respect for your coworkers, or your employees’ work/life balance,” Mangot writes. “They need to take this time to rest and recharge.”
Others disagreed, arguing that deployments should go smoothly at any time if a company’s DevOps processes are set up properly. If companies are afraid of Friday deployments causing unexpected problems and eating into employees’ personal time, the reasoning goes, they should be afraid of deployments on weekdays as well. Good DevOps processes that encourage small, frequent deployments would result in more manageable changes, and also make it easier to fix any bugs that occur.
Small, Frequent Deploys Are Always the Way to Go
Although Mangot is opposed to Friday deployments, he agrees with much of the other side’s views about the benefits of small, incremental deployments.
“This notion of a smaller deploy being safer is partially true,” Mangot told Built In. “There’s no question that if you batch up a bunch of changes, that’s going to be more dangerous than doing a smaller deploy.”
But although companies should develop robust DevOps processes that prevent unexpected problems during deployments, that doesn’t mean problems won’t still occasionally happen, Mangot said.
“Just because it’s small doesn’t mean it isn’t going to have some kind of effect that you did not anticipate,” he said.
“It’s really hard for humans to understand every single component of every one of those systems, because they’re complex.”
Software systems can be complicated, and one of the reasons mistakes happen is because it’s difficult for developers to have a complete understanding of the software systems they are building — especially if they are working as part of an engineering team, he said. Sometimes even good processes won’t prevent mistakes that require a lot of effort to fix.
“One of the main reasons we have outages is that people have an incomplete mental model of the way the system actually works,” Mangot said. “A lot of us are working on complex, distributed systems: we’ve got databases, we’ve got application servers, we have caches.... These are not simple systems. And it’s really hard for humans to understand every single component of every one of those systems, because they’re complex.”
It can take a good deal of time and effort to fix unexpected issues during deployment, not least because developers may have to do some sleuthing to find the source of a problem.
“When something breaks, the reason sometimes it takes a long time to fix is because people will say, ‘I had no idea the system worked that way,’” Mangot said.
Deployment Moratoriums Are a Bad Idea
Avoiding deployments on Fridays is a way to protect employees from unexpected problems that could create extra work on the weekends — but that doesn’t mean companies should ban deployments on Fridays, Mangot said.
“You need to be able to deploy any time you want, day or night, seven days a week,” he said. “Mores are not moratoriums.”
Because unexpected problems can happen at any time, developers need the flexibility to roll out hot fixes without having to jump through unnecessary hoops. But with that said, developers should think twice about rolling out changes that could wait until Monday, especially on Friday afternoons.
“If it’s going to affect the production database, drop a bunch of columns out of the main table, there’s a potential that we could lose 10,000 rows of data for customers, and I’m going to deploy that on Friday — no, save that for Monday when everybody’s here and we can respond,” Mangot said.
Companies Doing Weekend Deploys May Have Bad Processes
Though opinions differ on the wisdom of Friday deployments, targeting weekends for code deployment in order to screen for bugs outside of normal business hours is a bad idea. Although it might be tempting for small startups concerned about attracting users to deploy during off-hours, it’s more important to build out infrastructure that makes it possible to deploy at any time. Mangot said companies can take advantage of techniques such as blue-green deployments, feature flagging and dark launches to ease the roll-out of changes to customers.
Blue-green deployments give development teams access to two production environments, allowing problems to be quickly mitigated by switching to the other environment. Feature flags allow developers to immediately turn off features that were recently introduced to production, and dark launches only release changes to a subsection of customers.
“I want my confidence in what I’m deploying to be based on my procedures and processes.”
“We should have the ability to turn off a feature that’s causing trouble,” Mangot said.
Ultimately, discussions over deployment are about figuring out the best technical and cultural processes that contribute to robust production code and frees employees up to do more important work.
“I want my confidence in what I’m deploying to be based on my procedures and processes,” Mangot said. “I don’t want it to be based on the fact that something’s been in production long enough.”