Things that Find Bugs in the Night: Massive Automated Regression Testing


Nighttime is the right time to run those extra tests you've always dreamed of. In this column, Harry Robinson explains how you can wake up to 40,000 additional tests a day at a cost that won't give you nightmares.

Testing Around the Clock
It's 10 PM. Do you know what your test system is doing? Probably nothing. And the same will be true at 11 PM, at midnight, and through the night. That's a shame. Wouldn't it be nice if you could pick up a few million extra tests in the off-hours? Well, you can, and for minimal cost and effort. Imagine you are testing a GUI where it takes a second to feed in an input and evaluate the result. Even at that conservative rate, a single machine could run through more than forty thousand tests in twelve hours! That's pretty respectable. If you are testing an API, you could churn through a billion tests without breaking much of a sweat. That's better than respectable! A few minutes of thinking and programming before leaving the office could pick up several hours of additional testing for free. And the results are well worth the effort. I love to hear a developer gasp "How on earth did you ever find this bug?"

Of course, it doesn't do much good to keep running the exact same tests over and over. Your machine must have a way to generate new inputs every time. Here are some ideas:

  • Use a random number generator to choose numbers within equivalence classes.
  • Feed the application random strings by mapping random numbers to a character set.
  • Generate every three-letter combination of printable ASCII characters. That's 884,736 strings!
  • Endlessly press all available buttons. (To find memory leaks, press every button except "Exit".)

Oddly enough, providing fuel for round-the-clock testing requires creativity more than it requires programming ability. For instance, to generate random strings, I once tweaked an existing password generator program from the Internet. Within 10 minutes, I had generated millions of input strings that looked like "Mh7-ZyQr?9d3W".

As another example, I tested a system that processed natural language, so I needed to provide almost—sensible sounding phrases to the parser. I downloaded etexts such as Moby Dick and The Hacker's Dictionary from the Internet. This allowed me to test the application with evocative phrases such as "Come forth from behind your cotton bags!" and "Barfulation! Who wrote this, Quux?" respectively.

If you want to increase the effectiveness of these generated tests, you can go several routes:

  • Have your developers add assertions to their code. Assertions check the internal state of the software and make it possible to immediately detect more subtle bugs like data corruption and stack overflows.
  • Vary probabilities in your randomness. For instance, if your test program is exercising a dialog and has to choose between clicking buttons and typing characters into fields, favoring one over the other can dramatically change the behavior of your tests.
  • Use heuristic oracles to determine if results are reasonable. For instance, if you are testing a mortgage calculator, check that the calculated payment is never negative.

About the author

StickyMinds is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.