Even the best developers make mistakes, and sometimes those mistakes make it into production.
When that happens, they’re hopefully caught by a developer or an external user and brought to the attention of the development team. At that point, it’s time to roll out a patch.
The main difference between a patch and a new version release is that, while new versions tend to ship with product enhancements and new features, software patches are meant to fix bugs and security vulnerabilities. For products such as web applications, rolling out a patch can be as simple as pushing updated code to the production environment — but that doesn’t mean the patching process is easy.
To help make software patching as stress-free as it can be, we spoke with experienced developers about some best practices you should keep in mind.
Reminders When Patching Software
- Order and priority of patches is important
- Test patches in small batches before pushing to a larger group
- Keep your users in the loop when rolling out fixes
- Remember that patches are a normal part of the development process
Order Matters When It Comes to Patching
A useful tip to keep in mind before releasing a patch is to consider the issues that can come up when rolling out patches internally within a company. The order in which patches are made can be important.
“The biggest struggle for large-scale enterprises is priority of patches, number of machines and isolated networks,” said Chelsea Brown, CEO of Digital Mom Talk.
Brown consults with companies to locate software vulnerabilities in need of patching. She said the priority of patches matter, because different types of vulnerabilities carry different levels of risk. It’s important to understand which issues to tackle first in order to try to minimize the overall risk.
“When you find a problem, a lot of times it’s not just a one-stop fix — sometimes there’s different levels,” Brown said.
“When you find a problem, a lot of times it’s not just a one-stop fix — sometimes there’s different levels.”
Some bugs are not big concerns, and they can be dealt with later. But a high-priority vulnerability — one that grants unauthorized access to a payment system, for example — requires immediate attention.
“Something that doesn’t affect day-to-day operations and isn’t necessarily going to hurt our bottom line or reputation or customers, it’s put on the lower priority end,” Brown said.
Higher-priority issues should be fixed first because leaving them could give attackers time to compromise a system, which can result in significant damage to the company’s reputation and revenue.
It’s important to take into account not only the technical aspect of the vulnerabilities when deciding the priority, but the business side as well. If certain patches are known to take more time and affect other systems, then the decision of when to patch should take that into account.
“As a security person, I think [the patches] are all vital,” Brown said. “But from a business perspective, there are ones that you need to do first — and there are ones that take a little longer. If you’re a company running Windows servers, Windows updates can take literally three to four hours. For a company to have a system down for three to four hours, that’s very critical.”
Be Sure to Test Patches Before Releasing Them
Companies need to understand all the different ways a patch will affect a customer’s system before releasing it. The best way to ensure that is by doing a lot of testing.
Tom DeSot, CIO at Digital Defense, a cybersecurity company that helps clients check for vulnerabilities in their systems, said it’s important to test the changes on a select few machines before pushing them out to a larger group. Within a company, this can be done by rolling out patches to individual machines or certain departments first.
“Typically, you have a lab of computers that are representative of computers on your network used by different divisions with different applications,” DeSot said. “You try to push the updates on those systems, and you’d make sure that it doesn’t cause the system to crash or anything of that nature. Once you validate that the patches are working properly, then you begin pushing them out to the organization at large.”
By approaching it step by step, it’s more likely that issues associated with the patch will get caught early, before they affect a large number of machines. Similarly, companies should test extensively prior to pushing out patches to customers, with a special emphasis on testing that pushes the edge cases of what the application is meant to do.
“They may work in our testing, but we’re testing them for certain types of use,” Brown said. “If a customer gets it and they’re not using it the way that we tested it, a patch could potentially break — and when you send it out to the user, you won’t know that until the user uses it.”
She said it is common for users to contact companies about bugs, only for companies to say that they have never seen that bug before. That kind of situation results from use cases the company hasn’t considered.
Keep Your Users in the Loop
In addition to getting the technical rollout right, Brown said it is also important to notify customers about patches, usually through a written agreement such as a security policy.
“The security policy should have a clause about [how long the company can take to issue] patches, because if a company [has no specified buffer] and a criminal was able to get into it, it opens up the company for potential lawsuits — especially if they suffer a major data breach,” Brown said.
Maya Kaczorowski, a product manager at GitHub, said notifying users of open-source projects can be especially tricky, because it isn’t easy to see who the users of an open-source project are.
“Open-source projects have taken a variety of approaches here, from a security-announcement mailing list to a forum or website,” Kaczorowski said. “Ideally, develop a fix before going public with the vulnerability, but if that’s not yet possible, or if the vulnerability is already public, identify mitigations to help users better protect themselves even without a patch.”
Kaczorowski said it is good practice to have a help file in open-source code repositories that specifies the project’s security reporting and disclosure policy. A critical component of the instructions should be how to reach out to a project’s maintainers about possible vulnerabilities.
For GitHub projects, she suggested using GitHub Security Advisories, which add a project’s vulnerability information to the GitHub Advisory Database that tracks vulnerabilities in GitHub’s open-source projects. The advisory will trigger security alerts and “automatically tell your downstream users about the vulnerability.”
Sometimes, even patches themselves might have bugs. In those cases, once the bug is caught, companies can either roll back the patch or release a new patch that fixes the old one. Brown said the better option is usually to roll out a new patch that either fixes the broken code or renders it “null and void.”
But, most importantly, remember: Patching is a normal and important part of the development process, as long as proper testing is done and customers are informed before patches are rolled out.