This column asks the all-important question, "What isn't there that should be?" The same idea for spotting black holes also applies to spotting "holes in designs and requirements." For example, there are often connections between the quantity of bugs filed against an area and whether the area is thoroughly tested. There can also be holes in what KIND of bugs have been reported. Hendrickson lays out the territory for the search and goes on to suggest how to "look for where there's a lot of nothing."
"Dad, how do they find black holes?" I asked this question many years ago while walking with my father on a clear winter night, admiring the stars. Dad looked down at me, hands stuffed into his pockets, our breath visible in the cold air, and smiled. "Well, kiddo, since they can't see what's not there, they look for where there's a lot of nothing."
The idea stuck: find something by watching for nothing. It's how I look for holes in designs and requirements. If nothing in the design says anything about security, no one is thinking about it yet. If there are no requirements from a given stakeholder, chances are that stakeholder has not been represented in the requirements process.
It's also one way that I monitor the testing effort. If an area has no bugs filed against it, it's an indication that it hasn't been tested. Of course, just because a lot of bugs have been filed about a given area doesn't mean that it is well tested. So I also look at the kinds of bugs that have been filed. Do they get to the heart of the functionality or are they all superficial? Do any bugs involve bad input data, buffer overruns, or long path names? I'm looking for an indication that there's a hole in our testing—kinds of bugs that haven't been filed yet.
Looking for the absence of information isn't easy. You have to know what you expect to see before you can notice that it is missing. That means that you need a mental list of bug categories you might expect to find in the software under test: simultaneous user problems, data corruption bugs, timing issues, etc. Your list depends on what your software does.
You also need to know, in detail, what has already been seen. If you survey the bugs that have been found to date—reading them rather than counting—you can look for patterns in the testing that led to finding the bugs. Just counting isn't enough. What's missing doesn't always fit neatly into a drop down list in the bug tracking system.
As the project progresses, looking for patterns of missing bugs becomes more difficult. The more bugs that have been filed, the more difficult it is to spot the areas with a dearth of bugs. And yet this is the time when it is critical. You're almost done. Soon, someone along Executive Row will start asking why you haven't shipped yet. The next thing you know, the software is released and/or posted to the Web. That's not the time to find out that no one tried searching the catalog in two different Web browsers simultaneously.
One way to look for testing holes is to tabulate bugs on a matrix as they're filed in the bug tracking system. Along one side, divide the software under test into areas. Then list categories of tests that apply to all areas along the top. For example, if you are testing an editing program, the areas along the side might include Draw Tool, Text Tool, Insert Picture, Printing, etc.