As much as we want our software to work immediately as we build, we make mistakes. Sometimes, the architecture doesn’t quite match the instructions we’ve used. That’s why we have to test our software.
Smoke testing, which is also known as build verification testing, confidence testing, or even installation verification, provides us with a method for testing our software. It looks for critical errors to see if the current build is stable. What makes it different from other forms of testing is that smoke testing is non-exhaustive. You can think of it as a minimum viable testing set. You might have some questions, such as:
- What is smoke testing?
- Who is responsible for getting teams to start smoke testing?
- What happens when you don’t implement smoke testing?
Well, stick around. In this article, we’re going to answer all these questions and walk through the journey of Sarah, an engineering manager who is new to the company BobCo, which needs a minimum viable testing strategy to help improve the team’s bug rates.
What Is Smoke Testing?
How Smoke Testing Works
The origins of the term “smoke testing” are debatable. Cem Kaner, James Bach, and Bret Pettichord suggest that it comes from the process of plugging in a board during a hardware test to see if smoke comes out. If it does, the test is over. Another school of thought claims that the name came from the process plumbers use to check pipes for leaks with smoke before filling them with water. Regardless of its origin, however, smoke testing is a crucial component of your building process.
Let’s go through each part of the smoke testing definition, so we’re clear on just what we’re talking about. Nonexhaustive software analysis is a way to look at the things that make our software tick rather than trying to find every bug. This is important because looking at any other bugs without understanding if the core components work is a waste of time. Your tool isn’t driving value if it doesn’t function at all.
Think of it this way: Just seeing if your computer turns on and the operating system starts up when you buy it is the important thing. You don’t need to test every app right away. Your primary intent is to buy a working computer, and you can check that everything else works later.
So, smoke testing should be the lightest testing you do since you only need to see if the most critical aspects of what you built work.
Sarah saw the relevance of this as soon as she walked into BobCo. In the first month, while trying to get to know the team, she realized that they spent a lot of time working on bugs and solving them. Not just that, they were ignoring problems that some people would recognize as critical bugs. The team wasn’t giving priority to the most important ones and was solving whatever came in first instead.
Remember, working on bugs when the critical portions of a program don’t work is a waste of time.
In talking to the engineers on her team, she realized that, although they were doing some testing, it was scattershot. They had no way to know if the software was working or not. They needed to smoke test.
Who Is Responsible for Smoke Testing?
So, who makes smoke testing happen?
The engineering team is responsible for conducting smoke testing. Depending on the size of the team, this responsibility may fall to either the software developers or dedicated testing teams. Ultimately, however, the engineering manager or team leader is accountable for delivering working software.
Smoke testing is the lightest type of testing that you should do. It’s important because critical errors should be visible to the team for all releases. Ensure that, at a minimum, you smoke test all releases to make sure whatever you’re building or what you are adding doesn’t break everything.
Sarah makes sure the team is prepared by doing a post-mortem on the last result. She’s learned that it’s a lot easier to help teams see why testing is important by guiding them to the result, instead of just telling them.
She asks, “What drives this feature’s high bug rate, do you think?”
An engineer, John, speaks up: “We didn’t really test, but we didn’t want to waste time”
Sarah looks at John warmly and responds, “Thank you, John. How much time did it take to get this production ready?”
John answers, “We took an extra two days post-launch for it to work, but we didn’t think it was critical to test as it was a small addition.”
Sarah replies, “Sometimes the smallest pebble can bring down a building if that pebble is in the wrong place in the foundation.”
What Happens When You Don’t Smoke Test?
Instituting a smoke testing policy is important. Engineering management should have a system in place to make sure the engineers are picking the most important bugs to handle first and at the earliest possible point, which is before launch.
When you don’t have a good smoke testing protocol, you’ll notice its absence in your bug rates.
Fixing major bugs can have huge, immediate costs, like the time it takes to fix the problem. It also has costs that are harder to imagine, like loss of customer trust in your software. Remember, once released, you’re pushing that same issue you’re struggling with out to your users. Can you remember the last time something broke for you? It feels pretty rough, right? And it’s not just customers. A team has to smooth that roughness out, too. How much will other employees trust the people that make them do more work?
Sarah instituted a policy for her engineering team: They had to sit with the support team while a crisis is happening. She knew she had to make it real for the team, and that meant helping them see that the time “saved” is not even close to the amount of work that goes into keeping a broken product working.
Smoke testing will help teams understand if a product is working before investing time.
Smoke Helps Avoid the Explosion
As technologists, it’s often best to keep the quote from Alexander Pope top of mind while creating software: “To err is human, and to forgive divine.”
Smoke testing is a way for us to keep that in mind, as we make the things that people need.
