Imagine you’re a developer working at an e-commerce company during the peak of holiday season. One morning, you push a change to production and then — catastrophe. The company’s website crashes, customers aren’t able to complete their purchases and even orders that were already placed are now in limbo. Engineering teams scramble to find and patch the issue. In the meantime, the company is losing revenue, fast.
Just a few years ago, companies would often rely on code freezes to avoid situations like this. That meant blocking off time during which it is forbidden to push any new code to production. Usually, companies would implement code freezes during periods of high traffic to eliminate any risk of introducing new bugs.
How to Pull Off a Successful Code Freeze
- Use analytics. Try to narrow critical periods down to days or hours.
- Check for alternatives. Does your company need a code freeze or better CI/CD processes?
- Assess the need. Who do the changes impact? Is it mission critical?
- Communicate early. Let stakeholders know about upcoming code freezes early on so they have time to provide input.
Madhukar Kumar, head of growth at developer CRM platform company DevRev, has worked at many companies that used to implement code freezes. They were standard for public-facing software applications where a bug could look bad for the company or cause it to lose out on big profits during peak times.
But all of that is changing fast.
“What we have seen in the last few years — with [continuous integration and continuous deployment] pipelines, with GitHub action triggers — is that even for a production website, we can do multiple deployments in a day on an on-demand basis and the deployments never really bring down the website,” Kumar said.
These days, companies use A/B testing techniques and feature flags to quickly and easily roll out or roll back changes in precise and controlled ways. Considering all these modern methods for managing code rollouts, are code freezes ever still necessary?
Some Code Can’t Be A/B Tested
As lead product support engineer at Saatva, Chris Miller knows from firsthand experience that code freezes are definitely still necessary.
Saatva is a company that sells made-to-order mattresses. Customers come to the Saatva website to browse, choose between different options and colors, and purchase mattresses. Miller said one of the company’s core principles is to use just-in-time manufacturing, where the company manufactures each product as it is ordered instead of holding onto lots of inventory.
“We don’t have a warehouse full of mattresses sitting there waiting for somebody to buy,” Miller said. “We have contracts with manufacturers all over the U.S. When we get an order for a mattress, we send that order to a manufacturer who then produces your product on the fly, transfers it into a delivery hub and then [delivers it].”
That model works great, but it also necessitates a robust technology stack to ensure the company’s just-in-time processes don’t hit any snags, causing significant delays to customers getting their orders. As a result, the company is sensitive to software bugs that could affect its reputation or cost it business.
But having high-profile production code isn’t enough to require code freezes. After all, plenty of companies with highly trafficked websites don’t use code freezes, instead relying on incremental releases and CI/CD methods to roll back any problematic releases.
“But those are generally implemented on the front-end website,” Miller said. “You can say, ‘I want to send 5 percent of the traffic to a specific variation [of the website], either A or B.’”
This is A/B testing, where a small subsection of users is used to test new changes. At that point, developers can decide if the change is working well and then roll it out to the rest of the users, completing the full release of the feature.
“When I buy something, I want to buy the thing — I don’t want it to be kind of like the thing, but not the thing ... It’s either right or it’s wrong, there’s no in between. You can’t test an order.”
Saatva’s developers use techniques like that when pushing changes to the company’s front-end website. However, Saatva also has other technology stacks that are responsible for important back-end operations, like the company’s order management software.
“It actually takes that order, takes the money, makes sure everything’s in there, makes sure customer service representatives and the orders department can see anything that’s going on, and then transfers it to our logistics application,” Miller said.
Then the company’s back-end logistics software takes over, sends the order to the factory, deals with the factory’s response and guides the entire process of the mattress being built to order until it arrives at the customer’s doorstep.
These back-end parts of Saatva’s software can’t use the usual A/B testing, roll out and roll back methods because there’s no practical way to roll out an ordering system to only some users. When it comes to ordering physical products, the order either happens or it doesn’t.
“When I buy something, I want to buy the thing — I don’t want it to be kind of like the thing, but not the thing,” Miller said. “It’s either right or it’s wrong, there’s no in between. You can’t test an order.”
When you order something on Amazon, for example, you want to get the exact item you ordered, he said. There is no room for mistakes because a mistake means the customer didn’t get the item they ordered and now Amazon has to pay to reimburse the customer.
As a result, code releases for software like Saatva’s order management system and logistics system can’t fall back on methods like feature flags and test releases. Features are either fully in production or they’re not. And when the timing feels uncertain, that’s when to turn to code freezes.
Check Whether a Code Freeze Is Necessary
Still, Saatva’s engineering teams don’t implement code freezes all the time. While Miller is responsible for deciding when and whether they occur, it’s not as simple as him making a unilateral decision. He works closely with the analytics team and other stakeholders at the company.
First, the analytics team alerts Miller to upcoming high-traffic periods.
“We revolve a lot around key sales periods,” Miller said. “That sometimes lines up with holidays, sometimes they’re all over the place. You need that data to be able to tell you when is a safe period.”
Saatva has pretty good analytics, he said, so they’re able to predict accurately and precisely when periods of high traffic are going to be. Next, Miller compares these key periods against the production schedule for code coming down the pipeline.
At that point, Miller runs through a checklist to determine whether the situation is appropriate for a code freeze. One important criterion he considers is whether the changes have the potential to hurt revenue if something goes wrong. If not, a code freeze is probably not necessary. He also considers whether the changes are in a part of the application that has been troublesome in the past, whether there is significant testing infrastructure around the changes, and how many departments and people at the company it may affect. The decision is made on a case-by-case basis, Miller said.
“If the only team is my team of all senior architects — all right, I trust you guys a little more than the average team, I think that I have confidence to do it,” he said.
Even if the situation seems like a good candidate for a code freeze after going through the checklist, Miller doesn’t immediately implement one. Instead, he lets stakeholders know that one may be necessary and gathers feedback. Ideally, this is announced well in advance, so business leaders, project managers and developers all have time to provide input.
Sometimes after talking to the developers, Miller will realize that the change is small enough to not affect important functionality or the company’s revenue.
“Then I’ll usually let that slide,” he said.
Bad Code Freezes Can Lead to Low Morale
Miller is careful about the decision-making process because there are tangible downsides to code freezes, sometimes even when they go well. Workloads can get backed up over time, which can create more work in the future once the code freeze is over.
“So it’s like a ripple effect,” Miller said. “If we have to freeze for a week, that means something else has to not be done for a week, and additional time for QA might push that out even further. Depending on how long that is, it just amplifies exponentially.”
Miller said this can have ripple effects outside of the company as well — for example, if the company’s customers are themselves businesses that are waiting on the company’s products.
“It’s like a ripple effect ... If we have to freeze for a week, that means something else has to not be done for a week, and additional time for QA might push that out even further. Depending on how long that is, it just amplifies exponentially.”
Jeremy Steward, an engineer at robotics infrastructure company Tangram Vision, said that in his experience the worst effects of code freezes occur when they are done for the wrong reasons. If a team leader is primarily focused on delivering the maximum number of features in a sprint and forgets to think about the quality of the overall product, their team might hopscotch from feature to feature, writing unfinished code and building up technical debt.
“What happens is this effective code freeze, where you accumulate technical debt and you never go back and have your engineers smooth out the assumptions and fix long-standing bugs,” Steward said.
Ahmed Bashir, an engineer at DevRev, said code freezes also have the potential to hurt developer morale if they happen too often and aren’t handled well.
“Generally speaking, the impact is on developer throughput,” Bashir said. “When developer throughput suffers, developer morale suffers and momentum suffers.”
How to Run Code Freezes Effectively
But there are ways to use code freezes effectively without creating new problems. First, teams should make sure they actually need one. If code changes can be cautiously released using A/B testing and feature flags, teams should probably use that.
If the way the codebase is structured makes code freezes necessary, however, the next step is to invest in good analytics. That way, the window of time where a code freeze is needed can be narrowed down as precisely as possible.
“We have it down to where I can identify almost within hours of what that period of time needs to be. We make sure that it is the smallest amount of time it possibly can be.”
“A lot of people go, ‘I’m going to code freeze for a month or two weeks or a couple of days,” Miller said. “We have it down to where I can identify almost within hours of what that period of time needs to be. We make sure that it is the smallest amount of time it possibly can be.”
A shorter code freeze has less potential impact on developers’ work because there’s not much time for work to get built up.
Miller also stressed the importance of communication with stakeholders about upcoming code freezes and communicating to them early. That means talking to business leaders, project managers, developers and even customers.
“We need to make sure that if they have any opinions or concerns, that they can voice them in advance,” Miller said. “Maybe they know there’s this really big problem over here and we need to get that fixed first.”
It’s an Important Tool, But Use It Carefully
Code freezes serve an important purpose when they are done right. And sometimes the need for a code freeze can be outside of the company’s control, like code freezes to allow time for audits in highly regulated industries.
“Code freezes have been used in areas where explicit approvals are required,” Bashir said. “Technical approvals from partners, federal approvals like from transport authorities or health authorities, areas where you are absolutely vigilant about protecting against leaks or new trademarks, things of that nature.”
Luckily, depending on the type of code being audited, sometimes setting feature flags will be enough to handle those situations.
“Separating features and releases goes a long way to being able to really obviate the need for code freezes,” Bashir said.
So save the big guns for when they are necessary.
