“How Was This Tested?” Providing Evidence of Your Testing

Many testers, particularly when working off detailed test instructions, have a tendency to minimize the information they record when testing. This can vary from noting what variables or values they start with and what they end with, to simply noting the test “passed.” The challenge comes when problems are found later, possibly after the software is in production. How do we remember what we did, and when? What records do we have to refer to? How do we, as testers, answer the question “How was this tested?”

How Was This Tested?

That is one question every tester will get asked at some point in his or her career. Sometimes it is asked in a manner of “Wow! That is fantastic!” But sometimes, it is asked in a manner of “Why did you not find this problem?”

Of course, the difference between a tester feeling complimented and a tester getting defensive comes down to the tone of voice. Asking, “How was this tested?” can lead to an informative discussion where everyone learns something, or it can set off a confrontation—particularly when the tester already feels frustrated that a bug was missed and got through to production.

So when there are problems found in production, what information are we—testers and other participants and stakeholders in the project—going to want and need about how the software was tested?

Many testers working from a script operate under the belief that if a test “passes,” the only thing that really needs to be recorded is that it passed. If there are problems, then the test is “failed” and at least some level of information is recorded.

The Importance of Artifacts

Many involved in software focus their understanding of testing around test artifacts. The concept of test plans, test cases, and scripts tend to dominate their discussions on testing. What many fail to understand is that these artifacts are not testing. They are models representing how testing can be done.

The most important set of artifacts are the records kept during testing. Test plans and strategy documents describe what we believe testing should look like. Test cases and scripts prepared in advance describe how we believe the scenarios that need to be tested will be approached.

It is the information around the actual testing—the execution of the tests—that is of greatest interest.

Build a Body of Testing Evidence

When working with test teams not certain of the reason for tracking activities and observations, I often use the analogy that when taking notes on testing, testers need to create enough evidence of having tested the software that it would hold up in a court of law.

What do I mean by that?

Consider this: Rigorous testing requires tracking what was done and what was observed. Part of this involves taking adequate notes so you or anyone else can recreate and understand what was done weeks after testing the feature.

All of us are under pressure to get things done quickly. No matter what environment software is being developed in, the actual hands-on testing of the feature or function always seems to be under some form of time pressure. The challenge of keeping rigorous notes and records comes in how long things take—and not letting that note-taking get in the way of finishing testing on time.

Here’s how I try to address that.

Don’t Go It Alone

Have a partner or paired tester working with you, making notes of where and how you navigate, what values you enter, what options you select, even how long it takes between entering data or clicking on the dropdown menu. This partner might notice things you miss—responses or results that you are not paying attention to because you are focused on something else—and these things might be worth investigating later.

The person can also serve as a sounding board to bounce ideas off while working through a scenario. If a behavior is noted as unusual but less important than what you are working through, the partner can help you remember that this could be an additional path to exercise in the next iteration. Another set of eyes on the screen can help you “stay honest” and focus on what needs to be done now.

Record Everything

A common phrase among medical professionals is “If it isn’t written down, it didn’t happen.” Similarly, eyewitness testimony in criminal cases is increasingly coming under critical review because people don’t remember things as accurately as we think we do. Memories are flawed.

Write down everything, as you do it and as it happens. Anything that seems obvious now might not be so obvious in a week or two. Make a note of it.

There are also screen-recording tools to keep track of what the user does and what happens in response to every action: every move, the screen displays, and the messages. This gives you a fairly straightforward way of recording what happened when you tested.

There are other options as well. Tools to facilitate taking notes while testing are available, so one window can have the application under test running and you can have the note-taking tool open in another window. Screen snaps can be copied over into the notes so the tester can show precisely what happened.

Other Forms of Evidence

It’s important not to overlook simple yet easily missed information, such as the build or sprint when the testing was done. The same goes for the version of the database or schema, as those can change. Sometimes when database environments change, there are unexpected consequences that might not show up until later.

Depending on the type of software you are testing, you may have evidence you can identify and capture with minimal work on your part, such as logs: application logs, database logs, system logs—as in, logs on the device you are executing the tests on—and host logs on the system host.

These can contain valuable information about what is being tested. There can also be information to examine that is not readily apparent to an observer. These both have value to the tester, and in the records to be retained as possible evidence around testing.

The Question

Why do we need to keep this information? What is the point?

When unexpected results are found a few iterations or builds after this was tested—or in production—the question about how a given feature was tested will almost certainly arise.

In my experience, the true purpose of keeping this evidence is quite simple: It is a gift your current self is giving to your future self. It might be yourself in a few weeks, in a sprint or two, or months later. It might even be someone else who will make use of your gift.

Whoever it is, when something goes wrong and that person has to search for the reason, he or she—or you—will appreciate the effort you put into explaining how the feature was tested.