I see many test strategy documents in my work with clients. Almost without exception, they are big—fifty or more pages—and stuffed with soporific boilerplate copied from one project to the next. Defect management process, test suspension and resumption criteria, entry and exit criteria, meaningless generic risks ... yawn ... again. One frequently used model even parrots textbook test definitions, from unit to systems integration to load.
I have several problems with test documents like these. In many organizational cultures, mandatory documents become confused with the things they are supposed to represent. A test strategy document is not a test strategy; it’s a document. It might or might not serve to communicate a strategy. The bigger and heavier a document is and the more replicated content it contains, the lower the likelihood it will convey anything important that people can take meaning from. In reality, the central ideas in most genuine test strategies can be expressed effectively in a couple of diagrams or a page or two of clear prose.
There’s a more basic problem. A big test strategy document rarely contains an actual strategy. That leads me to ask if, as testers, we really understand what “strategy” means.
I’ll come back to the documentation issues a little later. For now, let’s acknowledge that you need to communicate your test strategy to project stakeholders in some way and negotiate their agreement.
First principles—What Is a Test Strategy?
A strategy is the overarching direction or design of a campaign, whether that’s a marketing campaign, a football season, or a campaign of war.
A test strategy is the set of big-picture ideas embodying the overarching direction or design of a test effort. It’s the significant values that will inspire, influence and ultimately drive your testing, and the overall decisions you have made about ways and means of delivering on those values.
Note that was “overall” decisions. A strategy is not a detailed plan. It’s the design behind the plan that you, as a skilled tester, have thought about and developed.
Learning about the product and project, you develop ideas about how to test. What approaches are you going to use, and what potential test design do you already have in mind?
It’s unlikely you’ll be able to test everything in every way and to the same level of rigor. An essential strategy decision, therefore, is how to draw the boundaries for coverage and on what basis. (This is probably the most important element for stakeholders to understand.)
For example, will you prioritize coverage and types of testing based on risk? What do you mean by risk, and how will you identify and assess risks that could be mitigated by testing?
Who are your stakeholders? What does product quality mean to them in practical terms? How do they characterize the value they expect to get from the product? Not all stakeholders are equal—in most organizations, the concerns of some are more important than those of others. Organizations and individuals differ in their appetite for risk and their perception of what constitutes risk. These are critical considerations in determining what your test will cover and how.
Overarching facts, principles, and beliefs may drive test priorities or severely constrain the test. For example:
- This product will be used by emergency services. People could die or suffer serious injury as a result of product bugs. Ease of use is critical for some features; others must produce fast and accurate results with no tolerance level for error.
- The product is strategic to the business. Threats to the publicly advertised launch date will be catastrophic to the company’s reputation and share price.
- The product will operate within a strictly regulated environment. Certain types of bugs could result in prosecution of the company’s executives.
- The product is a website that will be hit on a specific date by thousands of users in an event-based marketing campaign. If it crashes or crawls, your company’s name will be mud and it will lose its biggest, most prestigious supplier.