Mukesh Sharma writes that there are some situations in which exploratory testing does not work. Understanding these limitations is important in devising a holistic test strategy for the team.
Exploratory testing is a popular testing technique used to enhance a product’s test coverage by helping the team better understand the system and find issues when time is of the essence. Additionally, creative exploratory testing helps break the monotony of running regular scripted test cases and empowers testers to learn simultaneously as the tests are being run.
Despite these benefits, there are some situations in which exploratory testing does not work. Understanding these limitations is important in devising a holistic test strategy for the team.
What Is Exploratory Testing?
Exploratory testing is any testing where the tester is learning and designing tests on the fly. The tester then applies that learning to execute further tests and help improve the quality of the testing effort as well as the product’s coverage. Per this definition, exploratory testing can be done with scripted tests, but the learning element is what is most important. In one instance of exploratory testing, the tester is not guided by existing scripts and he uses his skills to explore the product and learn as he tests; he is not constrained by a pre-defined scope. This thought process aligns more with commonly used definitions of exploratory testing, including a very famous one by James Bach in his article “What is Exploratory Testing?” This article also touches upon the idea of exploratory versus scripted testing, which is what is of utmost importance for us here because it helps to decide when exploratory testing will be of value.
So, when do you really say no to exploratory testing?
Say No When Exploratory Tests Replace Scripted Tests
Given the value and return on investment (ROI) of exploratory testing several startups that often work within budget constraints and cannot allocate the required funds to software testing, resort to using this as their main testing technique to determine the readiness of the product to release. While this is certainly a good start and a better solution than not testing the product at all, these startups need to understand that this is not a perennial solution and exploratory testing cannot replace scripted testing. It is important to balance the two testing techniques so the right coverage is obtained within the cost and time constraints. Startups need to say no to exploratory testing—or, rather, say no to exploratory testing alone—once they have reached a few initial milestones. It’s important for all organizations to document and define the test coverage obtained through the overall execution effort in order to convince the product team and other stakeholders that the product is ready to ship. Organizations should have predefined measures and associated metrics that help confirm a product’s conformance with the exit criteria. This helps establish a larger knowledge base about the product and its quality rather than depending on individual testers for the coverage achieved through exploratory testing.
Say No While Dealing with Compliance Testing
In areas where scripted testing is of utmost significance and deviance from it is not acceptable, you must make the tough decision and say no to exploratory testing. This is especially the case with compliance-based testing, in which you must adhere to checklists, government mandates, and specific domain-based testing where obtaining legal certifications is a necessity. For example, in the field of accessibility testing, where Section 508 and other laws such as Web Content Accessibility Guidelines 1.0 and 2.0 are in place in the U.S., the tester has to go with a predefined checklist of tests and even look at using templates such as the Voluntary Product Accessibility Template that make it easier to test the product’s compliance with defined standards.
In fact, accessibility standards are predefined and mandated by individual governments (for example, the use of Disability Discrimination Act in the United Kingdom). Although the larger guidelines may remain the same globally, you must take into account locale specific nuances for which the product is targeted. In such scenarios the reliance on scripted testing is heavy, with almost no room for exploratory testing. Other examples include testing for Sarbanes-Oxley, Health Insurance Portability and Accountability Act, and other such laws and acts, which are highly regulated and require strict adherence to defined guidelines.
While compliance-based testing requires following a scripted flow, one must also keep in mind that not all of accessibility testing is purely script-driven. In fact, a user-driven approach is very valuable in usability and accessibility testing areas in which the tester plays around with the product or even brings in end users in ascertaining the product’s quality. In such cases a combination of ad-hoc, exploratory, and scripted testing can be a very effective approach to adopt. Similarly, in security testing the team may use a resource such as the OWASP Top 10 security vulnerabilities to test the product against. In such cases, exploratory testing comes in handy to enhance test coverage and touch areas that may be hard to perceive while tests are designed upfront. So, there are subtle differences between compliance and checklist-based testing as to where exploratory testing will not work and where it will work in conjunction with scripted tests.
Say No When Testing for Areas That Need Rigor, Close Monitoring, and Initial Base Lining
Closely monitored areas of test execution—such as automated build verification tests, regression tests, and the initial round of performance tests where baselines are established—are instances in which exploratory testing is not going to add much value and will only bring down the team’s productivity and morale. It also will not be able to give much meaningful information about the product’s quality. In such cases, scripted tests with a regular turnaround from the test team are important to cover details such as tests runs, test results, and defects mapped to tests.
Establishing a team-level understanding of exploratory testing’s meaning, when to use the testing technique, and—more importantly—when not to use it goes a long way in building and executing a balanced test strategy. This will help team members stay motivated, keep them excited about their testing tasks, and also help the product benefit from both exploratory and scripted tests, thus empowering the team to sign off on a release of exceptional quality.