Software development being what it is, perhaps it’s not surprising that most development teams are trending toward increased automation. DevOps has largely automated the processes of integrating, deploying and monitoring code, and development teams use automated testing to catch bugs early and deliver consistently robust products. This ongoing shift is great for efficiency, and, in comparison, it makes manual testing seem a bit cumbersome — what additional value could it possibly add?
Actually, there are still aspects of software testing that are difficult to automate — especially when it comes to user interfaces. Certain aspects of accessibility testing, for example, are best conducted by a person, and final manual reviews can be helpful for catching stray bugs and confirming that software meets user requirements properly. That said, manual tests usually still involve scripts, which describe the series of actions the tester should take and their expected results.
Exploratory testing is a specific type of manual testing. It’s typically performed at the end of software development lifecycles, and it complements the trend toward scripting and automation — yet couldn’t be more different. Instead of giving testers step-by-step scripts to test against, exploratory testing taps into testers’ creative sides, relying on their experience and intuition to guide the testing process and make discoveries that scripts and automated tests have missed.
What Is Exploratory Testing?
“There have been plenty of studies done where exploratory testing by a good tester with experience will expose more bugs than any other form of testing,” said Jay Newlin, software quality assurance manager at Promptworks, a software design and development consultancy.
Newlin said having the right tester is the best way to get the most out of exploratory testing, but there are also plenty of best practices that companies can use to do it well.
It’s OK to Be Choosy When Deciding What to Test
Manual tests take longer to run than automated tests, and exploratory testing’s tester-directed style in particular isn’t the best way to check for known issues.
Jon Quigley, a software testing engineer at Value Transformation whose background is in the automotive industry, said compliance tests, for industries that need to worry about regulation and compliance, are a good example of tests that can and should be automated.
“[Compliance standards] should be so well written that everything can be automated,” Quigley said. “That takes the demand off of the testers to do more of the stuff that only testers can do, because they have imagination.”
“That takes the demand off of the testers to do more of the stuff that only testers can do, because they have imagination.”
Another good example of tests where scripting works best, according to Newlin, is when testing focuses on specific parts and functionalities of software, such as whether a login process is working properly.
“When I’m looking at a very small area of the code, I’m going to use the more scripted style of testing, whether it’s automated or manual,” he said. “The more extended, end-to-end style of testing is where I’m going to engage in exploratory testing.”
For similar reasons, exploratory testing is usually performed by QA testers, not developers. Although developers often write automated unit tests for their own code, time-consuming exploratory tests are better suited for testers who can draw on their manual testing experience — and it also frees developers up to do more coding.
Exploratory Testing Favors Creativity Over Step-By-Step Instructions
Although exploratory testing doesn’t involve scripts, testers do use specific starting points for testing, called charters.
“When done best, exploratory testing uses a test charter, rather than a step-wise [script],” Newlin said. “The charter sets out the ultimate goal for testing.”
Explicit step-by-step instructions constrain testers’ creativity. They can limit the potential problems testers see, whereas goals give testers the freedom to explore. Newlin said that, in his experience, the effect of these different methods on tester behavior can be surprising.
“Exploratory testing opens the creativity and the observation of the human brain.”
“I had to write a script-based test for a client, and handed off the test to a very good tester who was following the steps stepwise,” Newlin said. “Essentially, there was something on the screen blinking in red that was saying, ‘Hey, there’s something wrong.’ But, because nowhere in my 20 steps did I say, ‘Be on the lookout for a thing blinking in red,’ they literally passed my test because they were able to do all 20 of those steps.”
Newlin said the error wasn’t due to the tester’s inability or inexperience, but instead was a result of the mindset that scripted testing instructions create.
“They literally had some kind of mental blinder on,” he said. “And that’s the difference between being so limited in focus on my test steps that I may miss things — whereas exploratory testing opens the creativity and the observation of the human brain, to make sure that they are looking at the whole picture rather than just these 20 things.”
Writing Good Charters Requires Understanding the Product
Creating good charters for exploratory testing also requires an understanding of how the software will be used by customers. Newlin gave the example of a hypothetical shopping app, and outlined potential goals on a charter that focus on how the app would be used in different scenarios. For instance, a parent may use the shopping app on Black Friday to purchase Christmas presents for their children. The tester would then adopt that customer persona and try to use the app accordingly.
Newlin said exploratory testing charters often draw from the same type of personas used to shape the project in its design phase.
“Ultimately, I want to free the tester to follow that story without a specific set of steps.”
“There’s a direct link between the user stories that the design team might lay out,” he said. “I would start with the user stories and the user personas that have come from the design team to start guiding initial charters.... Ultimately, I want to free the tester to follow that story without a specific set of steps.”
It’s also worthwhile to create charters asking testers to use the software in ways that weren’t intended by the developers.
“They’re almost like the negative test cases, testing for unusual occurrences and unusual approaches to using this software,” Newlin said. “A user persona as the hacker who wants to beat on somebody’s API and do some nastiness to back-end services — that’s a prime example of a negative test case.”
Whether testers are able to complete the goals within the charters, regardless of whether they contain positive or negative test cases, determine if the software passes exploratory testing.
“[The testers] say ‘Did it pass or did it fail?’ based on the ultimate goal,” Newlin said. “Overall, how well were they able to complete or not complete that specific journey?”
You Can’t Fix a Bug Until You Can Reproduce It
Apart from charters, exploratory testing has two other important components — a set time limit and a testing diary.
“A time limit is absolutely essential,” Newlin said. “I may be sending a tester into an area where they could end up in almost a mental infinite loop of possibilities — whereas when I give them a time limit, they know how to make sure they’re pacing themselves on options.”
The time limit isn’t only a courtesy for testers to make sure they aren’t wasting their time. When software is used in the wild, customers don’t spend unlimited time trying to figure out how to use it if it’s not intuitive. The time limits in charters should take into account how long a user is reasonably expected to spend on the task — if testers aren’t able to complete the task within the time limit, it could be a red flag.
When testers find bugs during exploratory testing, however, they shouldn’t count the time it takes to document bugs as part of the time limit.
“I want to know what things they went through, the steps that they did, so that we can reproduce any kind of bug in the future.”
“When I have a tester who encounters a bug, I actually do tell them to turn off their timer — I want them to spend as much time as they need,” Newlin said. “[If they have a] 20-minute time limit, and five minutes in they encounter this really cool bug, I actually want them to stop their timer at five minutes and spend the time they need to figure out how to write the bug ticket correctly for what they encountered.”
It’s important for testers to take notes, because developers need to be able to reproduce bugs in order to fix them. Sometimes, a bug only appears when users perform a very specific series of actions — in those cases, a tester’s detailed notes may be the only way to reproduce a bug.
“The tester needs to be able to create some kind of diary,” Newlin said. “If they encounter a problem, I want to know what things they went through, the steps that they did, so that we can reproduce any kind of bug in the future.”
Jon Quigley, from Value Transformation, said that recurring bugs found during exploratory testing may be good candidates to turn into ongoing requirements tests.
“If it becomes a recurring theme, that every time I run this exploratory test I find something, then that exploratory test will likely move into the requirements document,” Quigley said.
Exploratory Testing Gives Testers Room to Be Creative
The effectiveness of exploratory testing also depends on the tester. Anna Pilipchuk, a senior QA engineer at Orangesoft, a mobile app and web development company based in Belarus, said the exploratory tester should have a good understanding of how the software is expected to work.
“You need to know what is the goal of the end user,” Pilipchuk said. “It’s not for new testers who are new to the project.”
It’s also helpful to have some experience with testing, especially with manual testing techniques.
“To guess where defects can be and how to find it, you need experience in finding defects,” Pilipchuk said. “You need to know test design techniques [such as] equivalence classes and border values.”
Quigley said experienced testers also know how to test software like it will be used “in the real world.”
“There’s no substitute for getting in there and knocking around, and having that critical thinking.”
“There’s some things that the person using the product might do in the field, that might cause something strange to happen, that aren’t in the requirements document,” he said. “Like, I accidentally punched all the buttons — what happens then?”
Quigley felt that certain attributes and personalities are particularly suited for being a good exploratory tester.
“In addition to intuition, you need to be able to play these scenarios out — what really might happen in the field — in their head,” he said. “And there’s no substitute for getting in there and knocking around, and having that critical thinking.”
That’s what exploratory testing is good at enabling: exploration. It’s not trying to compete with automated testing’s speed and consistency. Instead, human testers doing exploratory testing are able to find unexpected bugs by drawing on past experience and being creative and intuitive in a way that scripts and step-by-step instructions don’t allow.
Quigley gave the example of a software tester he worked with at a manufacturing facility who was not an engineer or operator at the facility, but who had a knack for uncovering bugs during exploratory testing.
“He found things, and I think he found things because he would ask himself questions — ‘What could an operator do that’s not unheard of? There’s nothing that prohibits this sequence of steps from occurring,’” Quigley said. “It was nothing in the requirements, he just ‘what-if’ed it.”
