Many people think software testing is just about verifying from a checklist that functions do or do not appear. But what if more testers spent time looking at the product's behavior? Testers working with product owners, a development team, and other stakeholders can compare their understanding of the software and ask the crucial question, “Will this meet our needs?”
I try hard to learn what other people think about testing and how to do it well. In doing so, I’ve heard a variety of answers from gurus telling me what to focus on. If you’ve ever asked for guidance in your career in testing, I suspect you’ll find these ideas familiar:
- Software testing finds bugs
- Software testing verifies conformance to requirements
- Software testing validates functions
There are different versions of these ideas that may be expressed in different ways. Some people focus exclusively on one item. Some will look at two. Sometimes these ideas are presented as the best way (or the “right way”) to deal with questions around testing.
Some organizations embrace one or more of these ideas. They define and direct testing based on their understanding of what these ideas mean. They insist that testing be done a specific way, mandating practices or documents under the belief that controlling practices will ensure maximum effectiveness and the best possible results. Less training time and easier switching between projects are two common reasons to do this.
Frankly, even when these practices work, I find the results unsatisfying. For example, standardizing often consists of detailed scripts. These scripts direct people’s efforts, which often results in actively discouraging questions. The reasons for detailed scripts are often wrapped around concepts that many in the organization have a very shallow understanding of, such as Six Sigma in software development and repeatability of effort.
In Six Sigma, variation is viewed as the cause of error. A shallow understanding of Six Sigma leads to the understanding that varying from assigned steps in a test document will result in “error” in testing, making variation in test runs a cause of deep concern. If the expected results explicitly state one thing, those executing the tests will soon find themselves looking only that thing. As Matt Heusser has often said (and I’ve stolen the line time and again), “At the end of every expected result is another, undocumented statement that says, ‘. . . and nothing else strange happened.’”
The obvious solution is to direct people to look at broader aspects than what is documented as “expected results.” This sets up a conundrum around what is and is not part of what should be looked for.
Many of us would assert that the tester should, out of responsibility and professionalism, track down apparently anomalous behavior and investigate what is going on. Consider the team-reducing variation in a shallow way that has defined steps that take a known period of time to execute. Then add a little time pressure. What do you think happens when they encounter something that does not fit but is not explicitly supposed to be checked for?
The human mind ignores these types of errors—which are often the most important errors, or at least a hint that might lead to the most important error. If you doubt this, then here is an exercise for you: Search for something on Google and look for the banner ads. Your mind has been ignoring these for years. See how large and prominent they are? Funny how you don’t notice them unless you look!
Of course, management can insist that testers be “professional” and investigate off-script issues, but when testers follow that advice, they will exceed the allotted time for the test case. If part of their performance review and resultant pay is tied to those measures, can we really expect them to branch out from the documented steps?
Teams that rely on automated tools that let you simply click and get reports for functional or UI testing are set up for a similar problem. Without careful investigation of the results in the tool and application logs, the software will only report errors in the explicit results. That means the error has to be anticipated in advance in order for the automation code to look for it.
A Different Way
I’ve explored the consequences of these ideas and have tried solutions earlier in my career. I don’t believe they work as broadly as many say they do. Frankly, they fail my “smell test.” Can I suggest a fourth idea about testing based on the things I have seen that actually work?
- Software testing is a systematic evaluation of the behavior of a piece of software, based on some model.
Instead of looking for bugs, what happens if we look at the software’s behavior? If we have a reasonable understanding of the intent of how the software is to be used, can we develop some models around that? One way might be to consider possible logical flows people using the software may use to do what they need to do. Noting what the software does, we can compare that behavior against the expectations of our customers.
These observations can serve as starting points for conversations with the product owners on their needs. The conversations can incorporate the documented requirements, of course, along with the product owners’ expectations and expertise. This means the project team can choose the path they wish to examine next based on its significance and the likelihood of providing information the stakeholders are interested in evaluating.
Instead of a rote checklist, testers working with product owners, a development team, and other stakeholders can compare their understanding of the software and ask the crucial question, “Will this meet our needs?”
Comparing system behavior with the documented requirements means that testers can help initiate and participate in discussions around both the accuracy of the requirements (Do they match the expectations?) and the way those requirements are communicated, thus helping reduce the chance of misunderstanding. This helps the business and requirements analysts do a better job writing requirements and positions us for conversations around how to make requirements better.
By changing what we are looking for, from specific items in a checklist to looking at overall behavior with specific touch points, we change what we do and how we are considered—moving testing from an activity that has to happen to get the software out the door (a cost center to be minimized) to a value-adding activity.
If you have served in this industry for any length of time, you have probably felt offended as a tester. Perhaps someone who had never done testing a day in his life defined a process for you and you did it, knowing the work you were doing was low-value and would take too long. Perhaps you were given a detailed low-variation test plan, were measured on time, and also were told to investigate—a scenario where you can’t win for losing!
If that is the case, it might be time to say something like, “Let’s talk about what I do as a tester.” I know you may be scared or worried about your review. A few testers I know have been fired over this, but consider the alternative: keeping a job you don’t really want to have.
Sometimes, the way to be most effective at your job is to act as if you don’t care about keeping it.
