Random Testing and "App Compat"

[article]
The Ten Commandments of Software Testing, Part 1
Summary:

Thoughts of testing tasks keeping you up at night? Looking for some helpful tips? Well then, you've come to the right place. James Whittaker's Software Testing Ten Commandments may be just what the doctor ordered. Curious? Take a peek at this week's column to find out more!

In 1996, I posted my version of the Ten Commandments of software testing on a Web site. I wrote these commandments in the middle of the night when I couldn't sleep. I was thinking about what things I could do to try to repro an elusive bug. As most testers will attest, sleep can be hard to come by in such a circumstance! Somehow in my slightly warped tester brain, these "things" took the form of commandments. It was a very spiritual moment.

I took the time to write the commandments down and then promptly went into a very deep sleep that made me late for work the next day. I was determined not to make my sleepless night-and late arrival-completely without benefit and decided to post the commandments to my Web site (they can still be found on http://www.howtobreaksoftware.com/). I wondered if anyone would notice.

Well, I can now say for certain that, indeed, they noticed. It took a while, but I started getting mail about them. At first, the mail trickled in. Then, months later, it began pouring in, at the rate of two or three a month. It steadily increased until I would be surprised if a week would go by without getting at least one note about my commandments.

The mail was neither complimentary nor derogatory. In typical testing fashion, it was inquisitive. Could I please interpret one of the commandments? Would I settle a bet between two colleagues about commandment number four (except I thought "settle" meant "pay" and I told them "no")? And the most frequently asked question: "Why are there only nine commandments?"

Well, after hundreds of private e-conversations (some of them months-long in duration) with individual testers around the globe, I have finally decided to come clean, publicly, and get these commandments "off my chest" once and for all. But it will take three of these columns to accomplish the task. This column explains the first two. The second will explain numbers three through six, then seven through nine (and the lack of the tenth) in the third column. I hope you enjoy them, as I have enjoyed them over the years.

First, I'll just list the commandments so you can see them as original visitors to my old Web site saw them, listed but not explained:

1. Thou shalt pummel thine app with multitudes of input
2. Thou shalt covet thy neighbor's apps
3. Thou shalt seek thee out the wise oracle
4. Thou shalt not worship nonreproducible failures
5. Thou shalt honor thy model and automation
6. Thou shalt hold thy developers' sins against them
7. Thou shalt revel in app murder (celebrate the BSOD!)
8. Thou shalt keep holy the Sabbath (release)
9. Thou shalt covet thy developers' source code

And now, here are my interpretations of numbers one and two. I just hope I remember all the things I included in my earlier interpretations! Any of my old correspondents are welcome to chime in at the end of this column.

1. Thou Shalt Pummel Thine App with Multitudes of Input
One of the first things that any tester learns is that the input domain of almost any nontrivial software application is infinite. Not only are there lots of individual inputs, but inputs can be combined and sequenced in so many different combinations that it is impossible to apply them all. One of the second things testers learn is that the trick is to apply the right set of inputs so that infinity doesn't have to be faced head-on.

Well, of course I agree with this approach. My

About the author

James Whittaker's picture James Whittaker

James A. Whittaker is is a technology executive with a career that spans academia, start-ups, and industry. He was an early thought leader in model-based testing where his Ph.D. dissertation became a standard reference on the subject. While a professor at the Florida Institute of Technology, James founded the world's largest academic software testing research center and helped make testing a degree track for undergraduates. He wrote How to Break Software, How to Break Software Security (with Hugh Thompson), and How to Break Web Software (with Mike Andrews). While at Microsoft, James transformed many of his testing ideas into tools and techniques for developers and testers, and wrote the book Exploratory Software Testing. For the past three years he worked for Google as an engineering director where he co-wrote How Google Tests Software (with Jason Arbon and Jeff Carollo). He's currently a development manager at Microsoft where he is busy re-inventing the web.

StickyMinds is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!

Upcoming Events

Nov 09
Nov 09
Apr 13
May 03