Exploratory testing has gained increased recognition as a valid testing methodology. As a test manager or engineer you may be considering which approach to take. However, just as with other aspects of testing, such as automation, its application must be carefully chosen to ensure a successful outcome. In this article we examine the factors you need to consider when making that choice.
Editor's Note: This article was published Feburary 25, 2003.
Exploratory testing has gained increased recognition as a valid testing methodology. As a test manager or engineer, you may be considering whether or not to use exploratory testing on your next project. However, as with other aspects of testing, such as automation, you must choose your method carefully to ensure a successful outcome. In this article we examine the factors you need to consider when making that choice.
In his article "What Is Exploratory Testing?" James Bach defines exploratory testing as "test design and test execution at the same time." In contrast, planned, or scripted, testing has a test planning and design phase, followed by test execution. The products of the test design phase are generally test plans, test specifications, and test cases and scripts. These products are then used during test execution to conduct the formal testing of a system.
In this document we will examine the factors influencing the effectiveness of both approaches, as well as combination and hybrid approaches, to select the most effective and efficient methods within the constraints of time and resources.
Factors to Consider When Choosing a Testing Approach
What are the factors we need to consider when choosing our approach? The main ones are listed below:
- tester domain knowledge
- system complexity
- level of documentation
- timeframes and deadlines
- available resources
- skills required
- acceptable risk levels
Let's examine each of these more closely.
Tester domain knowledge. Simply put, how well does an individual test engineer understand the operation of the system and the business processes that system is supposed to support? If a tester does not understand the system or the business processes, it would be very difficult for him to use, let alone test, the application without the aid of test scripts and cases. Likewise, in user acceptance testing (UAT), where business users are conducting testing, a new interface may require the use of formal scripts or training to orient the users before they can test the system. A simple system such as an information Web site, on the other hand, is well understood by a professional tester, making it a prime candidate for exploratory testing.
System complexity. The complexity of the system can also affect its suitability for exploratory testing. The user needs to understand how to use the system. Where there is a high level of dependency between functions for end-to-end testing (i.e., one process depends on the data created by another), detailed test planning is required. End-to-end testing can be accomplished with exploratory testing; however, the capabilities and skill sets required are typically those of a more experienced test engineer.
Level of documentation. Scripted testing generally flows from business requirements documents and functional specifications. When these documents do not exist or are deficient, it is very difficult to conduct scripted testing in a meaningful way. The shortcuts that would be required, such as scripting from the application as it is built, are accomplished as efficiently using exploratory testing. We will discuss later how some degree of formal planning can be accomplished.
Timeframes and deadlines. The lead-in time to test execution determines whether you can conduct test design before execution. When there is little or no time, you might at least need to start with exploratory testing while documented tests are prepared after the specifications become available.
Available resources. The total number of person-days of effort can determine which approach you should take. Formal test documentation has significant overhead that can be prohibitive where resources and budgets are tight.
Skills required. The skill sets of your team members can affect