Here is the full list of commandments so you can see them in one list as they originally appeared on my Web site :
- Thou shalt pummel thy app with multitudes of input
- Thou shalt covet thy neighbor's apps
- Thou shalt seek thee out the wise oracle
- Thou shalt not worship nonreproducible failures
- Thou shalt honor thy model and automation
- Thou shalt hold thy developers sins against them
- Thou shalt revel in app murder (celebrate the BSOD!)
- Thou shalt keep holy the Sabbath (release)
- Thou shalt covet thy developer's source code
You can click at the end of this column to read part 1, commandments 1 and 2, from last month. And now, here are my interpretations of numbers 3-6.
3. Thou Shalt Seek Thee Out the Wise Oracle
We all know that there are at least two parts to testing. First we apply, then we check. When we apply inputs we are testing whether the software did what it was supposed to do with those inputs. Without the ability to verify that this is indeed fact, testing is much less effective.
Testers call this the "oracle problem" in reference to the wise oracle that knows all the answers. Of course, the answer we are interested in is "did the app do what it was supposed to do when I applied some test?" This requires our oracle to intimately understand what the application is supposed to do given any specific combination of inputs and environmental conditions. Automating the oracle is very hard, but a worthwhile pursuit, not only as a valuable testing tool but also as an intellectual pursuit. Forcing yourself to think like such an oracle can often be more productive than anything else that you might choose to do, whether or not you ultimately succeed in automating it.
4. Thou Shalt Not Worship Irreproducible Failures
We've all been here, haven't we? You see a bug, usually a good bug, then it won't reproduce. The better the bug, the worse you feel about it. I have seen many a good tester waste hours and even days trying to reproduce a bug that they saw only once.
The effort to reproduce such a bug is often valiant but without the proper tools, the effort can be a waste of time. But the problem I see is that the time is wasted anyhow, without the tester even realizing it. I had a tester spend an entire day trying to remember the reproduction steps of a crashing bug, with no success. I would have preferred that the particular tester spend his time in better ways than that. I understand the frustration as well as any tester but the pursuit of such a bug is often time not well spent.
The moral of this commandment is twofold. First, try your best to be ever alert and remember (or record) the sequences of actions you are taking against the software. Remember also the application's response. Second, consider using debugger-class tools that can track your actions and the state of the software. This takes much guesswork out of reproducing bugs and prevents otherwise good testers form breaking this commandment.
5. Thou Shalt Honor Thy Model and Automation
Commandment one was about the importance of random testing-emphasis on random. This commandment is about intelligent random testing-emphasis on intelligent. When intelligence meets automation, the result is called model-based testing. Get used to the term because it is the automation technology of the future.
Software models such as objects, black boxes, or structure diagrams help us to understand software. Testing models help us understand testing. A testing