Continuous integration and continuous delivery are commonly grouped together as CI/CD, but each side of the term refers to a distinct part of the DevOps process. While continuous integration (CI) refers to the process of merging developers’ code together into a combined working version, continuous delivery (CD) is how that combined code goes the rest of the way to being ready for release to production, where customers use it.
“Continuous integration is the idea that your change sets are going to be incorporated into the main trunk of development in an automated fashion at regular intervals,” said Marshall Greer, CTO at enterprise social media platform Greenfly. “Continuous delivery is the idea that you’re going to take those changes and immediately roll them out to some common or shared environment, and in some more extreme cases straight out to production.”
WHAT IS CONTINUOUS DELIVERY?
Continuous delivery covers a lot of ground because it encompasses everything that comes after a successful code review. Once code is merged from a feature branch into the main branch, there are other considerations before that code is ready for production — more testing, configurations, database migrations and setting up monitoring. The implementation and automation of all these steps are part of the continuous delivery process, and the way each step is configured depends on the business, the structure of the company’s development team and the nature of its software product.
‘Continuous’ Means Automation
The word “continuous” in both CI and CD refers to the way DevOps processes are increasingly automated, which provides advantages like early bug detection and consistency across development teams.
Eric Johnson, executive vice president of engineering at GitLab, a popular code collaboration and DevOps platform, stressed the importance of automation in continuous delivery.
“You really want to make sure that you’re relying on automation, so that as little as possible — or zero — of what we talked about is done manually,” Johnson said. “That goes for your tests — everything from unit tests to really sophisticated integration tests or UI tests.”
Testing happens in all stages of the development lifecycle, including development, integration and delivery. Developers write and run unit tests along with their code, continuous integration includes tests to check for quality before merges, and still more tests check for functionality and bugs as the code gets ready for release to production.
“You really want to make sure that you’re relying on automation, so that as little as possible — or zero — of what we talked about is done manually.”
“Testing, as a practice, occurs throughout the workflow, but it begins locally on developer machines,” Greer from Greenfly said. “Once those changes are pulled up into the main code repository, the tests are rerun in an automated fashion to make sure there weren’t any defects introduced — maybe a merge went bad or had some unintended side effects.”
Automated testing is a big part of what makes continuous delivery valuable. Instead of a mad scramble to find and fix bugs just before deployment — or worse, having customers discover the bugs — developers get immediate feedback whenever bugs are introduced, allowing them more time to fix changes and have the code ready for deployment when the time comes.
Code changes during integration often necessitates changes in other non-code parts of the project, such as configurations and database migrations. If developers change a form on a website to request more inputs from the user, for example, that change needs to be reflected in the database as a new table or additional columns to existing tables. Because database migrations need to be done for all environments — and sometimes need to be reversed as a result of later code changes — tools that automate migrations make the process much easier.
“Automating your database migrations is really important,” Johnson said. “Having them be both forward and backward compatible so you can roll things back is really important.”
Review Apps Can Save Devs From Spreadsheet Hell
Although technically part of the continuous integration process, review apps — on-demand testing environments for hosting feature branch code that are easy to spin up and spin down — are considered important tools for continuous delivery. Before using review apps, Sean Connolly, a software architect in Boulder, Colorado, was accustomed to waiting around for his turn to test code changes on test environments that everyone wanted to use.
“We used to have these static test environments that were incredibly difficult to work with and maintain,” Connolly said. “You had to go into a spreadsheet, be like, ‘I’m getting test environment No. 3,’ and put your name down. And if it breaks, then you have to go to the DevOps team and they have to get you out of it. Some people just left the spreadsheet, so you’d always have to send them a message on Slack saying; ‘Hey, are you done with QA three? Because I could really use it.’”
Having test environments for feature branches is important to developers, especially front-end developers, because QA, designers and other developers need a place to review changes before those changes are merged into the main codebase.
“It’s something that’s definitely resonating with front-end developers — I know my team can’t live without them. They’re like, ‘If we don’t have this, then we don’t know how to get our job done,’” Connolly said. “We just happen to use it for more than QA. We use it for design review, we use it for code reviews.”
Think Delivery, Not Deployment
CD stands for continuous delivery, but sometimes the abbreviation is misunderstood to mean continuous deployment to the live production environment, Johnson said.
“The two are kind of related, but there are important differences,” he said. “Continuous delivery means that every release is proven to be deployable, but the decision to deploy is actually a business decision. So you want to verify that everything is ready to go — that you can deploy if you choose — but when you deploy is up to a bunch of other sort of non-technical factors.... You may be ready to deploy multiple times a day, but the customer might say, ‘Well, I can only consume one release a week.’”
Johnson said companies may want to avoid frequent deployments to production for a number of reasons. For instance, e-commerce sites may not want to risk introducing bugs during particularly busy shopping seasons, and companies in regulated industries may need external compliance checks before they are allowed to release their code. These companies follow “scheduled” releases, which happen on predetermined dates instead of continuously.
“The decision to deploy is actually a business decision.”
“I used to work for a company where we had a three-week release cycle,” Connolly said. “We talked about the release train, like you’re either on the train or off the train — the release is happening, and you either get your code ready for the release or not.”
The type of release schedule used by a company also depends on their product. It’s easy for websites to continuously deploy to production, for instance, because they can just as easily roll back any errors. That’s not the case for desktop applications or mobile apps, which need to be approved by app stores and downloaded by users.
“Scheduled releases are still common in mobile app development,” Connolly said. “It’s a lot easier to deploy a web application, because the next time you visit the site, you get the latest version. With your mobile apps, the versioning is a little bit more of a heavy choice to bring something to the app store.”
And then, of course, there are also companies that simply haven’t caught up yet.
You Can Have Your Cake and Eat It Too
Companies that deploy infrequently, and that deliver code straight to deployment, may be missing out on a lot of the advantages of continuous delivery.
“If I go back to the beginning of my career, I remember companies deploying big annual releases or quarterly releases,” Johnson said. “But if that’s how often you really deploy, you’re really setting yourselves up for introducing one big change that’s likely to have problems — and troubleshooting a problem when you’ve just deployed this blob of hundreds of features is extremely difficult.”
Large releases encompassing many code changes are difficult to troubleshoot when something goes wrong because there’s simply more code to dig through.
“It’s going to be very difficult to diagnose, and it’s going to slow you down,” Johnson said.
On the other hand, it’s fairly easy to pinpoint the source of an error in a small release with only a few changes.
“If something was working, and you deploy and you broke it, you’re 99 percent certain the problem is in whatever you just deployed,” Johnson said.
But companies don’t have to choose between big and small deployments — they can combine continuous delivery with scheduled deployments to get benefits from both.
“You can do continuous [releases] to an internal environment for internal review by your product managers or stakeholders, and then have that combined with staged rollouts to production,” Greer from Greenfly said.
“The more you can decouple those two, the better off you’re going to be.”
The trick is not to conflate continuous delivery with deployment. Continuous delivery means frequent releases to a branch or staging environment that is production ready, and deployment can happen from there at any time.
“The more you can decouple those two, the better off you’re going to be,” Johnson said. “You can have your cake and eat it too — you can manage complexity from an engineering perspective, by doing these very granular things, but you can still deliver that value at the most frequent interval that your users are willing to consume it.”
But how often does “continuous” mean? According to Johnson, it should be often enough that finding bugs in any individual release is easy.
“You want to shoot for the ideal, where every merge request, every change can be independently verified,” he said. “When there’s a problem, you can say, ‘OK, the problem is likely in this one feature, or in this hundred lines of code.’”
The choice is up to the development team. Friday code deploys can be a touchy subject, but once delivery is separated from deployment, delivering code on Fridays becomes a lot less stressful.
Good Processes Make Everything Easier
Even with all the right CI/CD processes in place, when it comes down to it, actual deployments to live production environments can still be nerve-wracking. Thankfully, deployment techniques can help further mitigate the risk of releasing bugs into production. Johnson said GitLab has tools such as feature flags and canary deployments in their continuous delivery offerings.
Greer said the use of feature flags can trace its origins back to a group of developers who worked at Flickr and Etsy in the mid-2000s.
“The way that they would manage risk is that, in addition to having automated testing in place, they made extensive use of feature flags, or toggles,” Greer said. “The code could actually be rolled out to production right away, but effectively in an inert state where it wasn’t active. In order to activate the code, you would toggle the feature on.”
The same group of developers also experimented with canary deployments, a method of testing out new features on a small subset of users in production. The feature is enabled for all users only if there are no issues or side effects observed in the subset.
“You don’t want to be in a position where your users are telling you something’s wrong.”
“If you have a large user base, and maybe within that user base you have a subgroup whose risk profile is a little bit more forgiving than others — they kind of are comfortable being beta testers,” Greer said. “Then, if you were to enable a particular feature flag for just that set of users, then it’s often referred to as canary testing — as in the canary in a coal mine. If something goes wrong, they will be the only ones that will be impacted by it.”
Johnson said canary deployments can be especially helpful for high-volume applications, where it’s otherwise difficult to test user load on non-production environments.
“Canary allows you to deploy a change to production, but just to a subset of production users — like one-one-thousandth, or one-ten-thousandth of users,” Johnson said. “And then you can get metrics back, just from that cohort that’s experiencing a different version of the application.”
Even if all else fails, projects will always benefit from having proper monitoring set up.
“You don’t want to be in a position where your users are telling you something’s wrong,” Johnson said. “You want to have monitoring in place so you see it immediately from the environment. [If] the environment is telling you what’s wrong, then you’ve got a chance to fix it before your users even have a chance to tell you that there’s a problem.”
