Exploratory testing—questioning and learning about the product as you design and execute tests rather than slavishly following predefined scripts—makes sense for many projects. But does it make sense for agile projects? In this column, Johanna Rothman examines how exploratory testing might work on an agile project.
It's easy to see how exploratory testing works on waterfall projects. In fact, it makes a ton of sense on waterfall projects to do at least a little exploratory testing before you decide where to focus your efforts, design your tests, or consider any automation. That's because on a waterfall project testers are much more likely to be crunched for time at the end of the project. (For ideas on how to use exploratory testing on a waterfall project where you don't have enough time see my previous column, "So Many Tests, So Little Time.")
But does exploratory testing make sense on an agile project? Of course. Yet agile exploratory testing doesn't look the same as it does on a waterfall project. There is less of a need for exploratory testing after the product is created and a greater role for exploratory testing as the team defines and develops the product.
Michael Bolton says, "When people talk about exploratory testing, they often mean the process of test execution. But the three pillars of exploratory testing are test design, test execution, and learning." In waterfall projects, the execution is most visible. But on agile projects, the feedback loop between test design, test execution, and learning is quite short—and agile projects require all three pieces.
On agile teams, there is little role for a manual black box tester who waits for the requirements, the design, or the code and finally tests according to a predefined script. That's because there's no waiting for requirements, design, or code. The product owner "defines" a feature or chunk of a feature, but the definition is more like a promise to keep discussing that feature rather than a firm commitment—"This is what the feature will do."
If the developers are using test-driven development, the tests help define the design. Even if the developers aren't using test-driven development, they tend to work in small chunks--no more than a day or so long—which decreases the waiting time for code.
There is still a role for testers who can turn their keen observations into ways to quickly test an area in depth. But those testers are even more valuable if they can develop tests and testing ideas—both manual and automated—before the feature is completely coded.
Roles for Agile Testers
- Testers can help the whole team work in a test-driven way by:
- Assisting with defining requirements in the form of user stories and generating system-level tests from those requirements. Think of this as a questioning, test-driven approach to development.
- Assisting with modeling the requirements (user stories)—by asking questions—and generating system-level tests from these stories.
- Testers provide information about each chunk, as the developer delivers it. Sometimes that information is confirmation that the code does what the user story says, does not have side effects, and is release-able at the end of each iteration. Sometimes that information is extra information about the product and other paths the testing could take.
- Finally, testers can perform system-level regression testing during an iteration. (To be honest, on well-established agile teams, the developers typically run the low-level regression tests, allowing the testers to do what they do best: explore, question, and learn.) If the organization is new to agile, the testers might run regression system-level testing at the end of the project.
Where Does Exploratory Testing Fit During an Agile Project?
Exploratory testing is helpful when working with the product owner and the project team, trying to define user stories. Imagine you're working on a banking system and want to allow between-account transfers. The developers have a scheme to use checksums to verify the From and To account numbers. You may have heard about the Fossbakk case, an example of insufficient checking for input and confirmation failure. If exploratory testers had been involved at the time of user story definition, testers might have asked, "What happens if we put in a longer account number than the system expects?" Or, during the modeling stage (which I certainly would expect on an agile project with a financial transaction system comprised of databases), an exploratory tester could ask questions such as how many ways can we make the transaction "fail?" Testers who ask questions like that all the time and explore the answers to those questions before the code is written may help the developers write better code. And, they'll have the basis for some great, nasty tests to see what the system really does. The questions lead to the test design, which leads to the test execution, which provides learning for everyone on the project.
With these kinds of questions, testers can use exploratory approaches as a first cut to defining tests on chunks of code. Or, testers can use exploratory techniques after verifying their automated tests are working and providing information about the product in its current state.
You can see that each question leads to learning or test design. In the same way, each test design leads to more questions or learning. The three pillars—test design, test execution, and learning—reinforce each other. You don't need to differentiate among the activities; which activity a tester performs is not relevant. What is relevant is that the tester performs all of them, and feeds back information to the rest of the project team.