Software development is all about forward motion — but, sometimes, going two steps forward means taking one step back. In some cases, commits can actually set your product back, as new code breaks production code that was already thoroughly tested.
“That’s the value of regression testing: Making sure that when you fix bugs and you find issues, they’re not just constantly getting reintroduced into your production environment,” said Crystal Preston-Watson, a quality and accessibility engineer.
Prior to moving to the QA space in 2013, Preston-Watson was a front-end developer: “Usually I was someone introducing bugs into the system,” she said.
WHAT IS REGRESSION TESTING
Regression tests can include a variety of tests, such as API and UI tests, but their shared purpose is to catch regression in the code.
“You’re running tests to make sure that, if you’re adding new features to your platform, that they haven’t changed any other important functionality,” Preston-Watson said. “As you’re adding things to previous features, that the stuff that you’re adding isn’t breaking anything.”
Regression Testing Isn’t Right for Everyone
But not every company needs regression testing. Jeff Sing, a QA manager at Optimizely, which makes testing tools, said a company’s structure and its DevOps processes can affect whether it makes sense. When Facebook was still a startup, he said, it didn’t rely on regression testing because of its fail-fast philosophy.
“They ship so small, so there was nothing super damaging. If I pushed something out, chances are I can flip it back,” Sing said. “A lot of smaller SaaS companies, they’re more agile ... they don’t need this fancy regressing testing framework.”
How much regression testing is needed also depends on the specific cost-benefit trade-offs companies make.
“You’ve got to remember, every industry is a little bit different,” Sing said.
Sing illustrated the difference by comparing two imaginary companies — a software as a service company that processes images, and an insurance company. The SaaS company could get away with less extensive regression testing, because an error is unlikely to have large financial consequences.
“If a picture doesn’t load, your customer’s upset for 15 minutes — move on, right?” Sing said.
For the insurance company, on the other hand, a mistake in the code that calculates insurance rates can result in a lot of lost revenue.
“If someone gets 15 or 20 percent instead of the right percent, how much does that cost you for each customer? So that’s a question your regression test has to be [testing],” Sing said. “If you make that one mistake, and you can’t take the money away from the customer, you can lose millions of dollars.”
Just Because You Can Test Everything Doesn’t Mean You Should
Since the point of regression testing is to make sure none of the existing code broke, it can be tempting to treat the occasion as an opportunity for double-checking everything. But that’s not a good idea.
“I never feel like I know I’ve tested every single thing, because that’s impossible,” Preston-Watson said. “There’s always going to be something where you could be like, ‘But what about this scenario?’ There’s edge cases, there’s corner cases.... If you aimed for 100 percent confidence you would never be done.”
Even running all the existing tests that were written for a project might be overkill. Sing said it’s a misconception that just because automated tests have been written for a project, running those tests every time is useful and cost effective.
“[It’s] super expensive,” Sing said. “It’s time-consuming, because if you have to block your build from building before tests finish running, what if your tests take 12 hours? You can’t deploy until the 12 hours are done.”
Aside from holding up the DevOps process, a large number of tests can also be a liability because the testing regimen itself becomes difficult to maintain.
“Tests get flakey,” Sing said. “[If] I write a test and then I leave, but the feature changes, [then] the test breaks.”
“Rerunning your whole testing process over again, you’re not actually regression testing — you’re just testing twice.”
If login flow tests were written prior to a company changing its authentication token, Sing said, all tests related to and depending on the login will also break.
“Someone has to go in there and fix [the tests],” he said. “And that is really, really expensive. That’s developer hours. You have to be on top of it, and you have to make sure it’s done correctly.”
Preston-Watson said the most important aspect of regression testing isn’t the volume of tests, but what is being tested.
“Rerunning your whole testing process over again, you’re not actually regression testing — you’re just testing twice,” Preston-Watson said. “It is a strategy. You definitely want to make sure that any mission-critical elements that you cannot afford to have go down have regression tests.”
Aside from the mission-critical components of an application, Preston-Watson recommended that tests of new features and known issues that have previously been a concern are included in the list of regression tests. That’s especially true for bugs that have been caught by users.
“If your users are bringing up a bug to you, that’s definitely mission-critical,” Preston-Watson said. “If you have a client who says, ‘Hey, I found this [bug] and it’s impeding my use of this platform,’ yeah, you definitely want to create a test case around that and put that in your regression suite to make sure that doesn’t happen again.”
Don’t Write Tests Before Knowing Why You’re Writing Them
It can be helpful to take a step back and think strategy before you start building your suite of regression tests, Sing said. This is done by taking an audit of the project’s testing landscape, which helps developers understand what code to target tests at.
“Here at Optimizely, we do a lot of research over what features are tied to which code base,” he said. “When I make changes, what downstream can be [affected]?”
Doing audits before regression testing is especially important for companies with larger codebases, because a proportionally smaller amount of code is covered by tests.
“As you get more complex and bigger, you cannot test everything, so you need to have a cost-benefit analysis of what you can test or cannot test,” Sing said.
The hypothetical insurance company he used earlier, for instance, may rank sections of code in terms of which ones would cost the company more money if they break. Tests would definitely be written to catch the most expensive issues, but code on the other end of the list may not be covered.
“We don’t test every single feature,” Sing said about engineers at Optimizely. “There are certain things that we know we cannot break: We cannot block somebody from logging on, we cannot let you log onto someone else’s account — so these are things that we always test. But we may not check that every single image loads, we cannot check that every single button is always there. It’s just really expensive.”
Doing an audit is also helpful for eliminating low-priority code.
“If you don’t understand what will break when you introduce new code, or when you change a configuration, then you’re never going to be building enough tests,” Sing said.
Learn to Do the Right Manual Tests
The types of tests performed during regression testing includes API tests, automated tests and manual tests. Although it may be tempting to only include automated tests as part of regression testing, since they can be run without using QA or developer time, there are many good reasons for testing manually.
“Any company that tells you, ‘We don’t manually test,’ I also cringe,” Sing said. “We manually test for sure, and we do this through exploratory testing.”
The purpose of exploratory testing is to get an overall understanding of the product and look for problems that may have been missed by all the other types of testing. A person manually goes through the application and interacts with it like a customer.
“No robot script framework will beat a human [when it comes to testing],” Sing said. “Did we build this in a way that is delighting our customer? ... Exploratory testing requires high skill and understanding of the product and the end consumer, to catch problems and, more importantly, [to figure out] if this is what our customer wants.”
Code for user interfaces also tends to rely on manual tests more than other types of code. Preston-Watson, whose area of focus is accessibility testing, said there are aspects of accessibility testing that need to be done manually.
“Did we build this in a way that is delighting our customer?”
“With accessibility testing, a lot of that cannot be automated, just because of how assistive technology works with your testing framework,” she said. “You can’t automate VoiceOver, which is the screen reader for iOS and macOS. So with regression testing, for a bug that is found with VoiceOver, I’m going to have to do that manually.”
But she said it is important to automate your regression testing as much as possible.
“Things that you can automate, you should definitely try to do, especially for things that are critical,” Preston-Watson said. “It helps streamline your regression testing process.... Otherwise, it will become this all-encompassing monster of trying to run manual tests.”
The trick is to pick out the parts of user interface testing that can be easily captured with an automated test.
“Some of the more tedious kinds of rote tests are great for automation — there’s really no changing data or input, it’s not wholly unique and dynamic,” Preston-Watson said. “If it’s a basic form, you can automate submitting a form. You can automate the happy path, you can even automate some of the unhappy paths.... There are traditional ways to break a form that you can definitely automate.”
Treat Your Tests Like Sourdough Starter
Although automated tests save time compared to manual ones, automated tests can require substantial maintenance.
“There are some companies that do not have the resources to hire full-time engineers to maintain [automated tests], because it’s expensive to build, maintain and run,” Sing said. “All these things need actual QA or developer support to be able to do it correctly. You can’t just close your eyes, shoot and then pray to God that it still works two years down the line — it needs to be maintained.”
Instead, automated regression tests need to be thought of as additional code that needs maintenance throughout the life of a project.
“It’s part of the product,” Sing said. “If you don’t build this aligned with the product, it basically becomes useless. If you can’t tell me what your tests do, or why you’re running them, or what the value is, it’s a waste of time and money.... Because it’s not about how many tests — it’s about what you’re actually covering, risk analysis wise.”
“You don’t want to have regression tests just to have regression tests.”
Preston-Watson said tests that aren’t kept up to date with the changing codebase can cause problems if the issues they’re supposed to catch are able to slip through the cracks again due to code changes.
“You don’t want to have regression tests just to have regression tests,” she said. “You could actually miss a vital bug or issue in your software because your test is wrong. Especially in vital areas, you want to keep up to date to make sure that they are testing what you need them to be testing for.”
She has been in that situation before herself, where tests she was running were out of date and didn’t actually test anything anymore.
“That’s one of those rookie mistakes,” Preston-Watson said. “Regression testing should be like sourdough starter — you’re constantly checking on it, making sure that it’s good, adding to it, feeding it.”
