Wreaking Havoc With Simple Tests

[article]
Summary:
Do the simplest thing that could possibly wreak havoc-that's how Danny Faught approaches testing. He finds this simple idea especially helpful when doing reliability and robustness testing. And as he'll tell you, Danny's one for making big plans for all the evil things he can do to an application, but he's most efficient when he starts out small and incrementally builds his own tools. In this week's column, he gives several examples of how he's used small programs to cause big problems, and then gives tips on how you can use this approach.

According to Martin Fowler, one of the mantras of the agile world is "Do the simplest thing that could possibly work." For my testing, I like to do the simplest thing that could possibly wreak havoc. Here are some stories that illustrate how I've applied this idea.

Flip a Bit
I was testing an embedded operating system on a handheld device. I surmised that the flash memory on the device could fail intermittently and cause file corruption. So I decided to simulate corruption in an executable file in order to test the system's robustness. I wrote a simple Perl script named "bitflip" that picks a single random binary bit in any file and flips it to the opposite value. The operating system I was testing couldn't run Perl, but it did support networking . So I ran the script on another machine to corrupt a copy of one of the standard utilities that comes with the embedded operating system. I made the corrupted file accessible over the network and ran it on the device under test. Usually the corrupted program would either crash, as we would expect it to, or run as if nothing had changed. But about 10 percent of the time, the entire operating system would crash.

The operating system vendor couldn't fix the crashes, and eventually the project chose a different operating system for the device. So the simple bitflip program, along with several other factors, managed to wreak a lot of havoc with this system. I could have spent a lot of time designing a more systematic test, but in this case the randomized approach was sufficient to prove the point.

Monkey Around
On another project, I was testing an early alpha version of a Windows application. It would occasionally crash in many different ways that weren't easily reproducible. I wanted to automate a test that could reproduce at least a few of the crashes. The simplest thing to do in this case was to write a "monkey test" that would randomly click on the main application window to test the application's reliability.

Again I turned to Perl, using the Win32::GuiTest module. I learned how to locate the application on the screen and click a random point in the window. After running the script for a few minutes, I found a handful of issues in the test with which I would have to deal. Because I allowed the monkey script to click and drag the application's border, it dragged the window under the task bar. Later clicks would activate items on the task bar, which was outside the scope of what I wanted it to do, so I changed my geometry calculations to keep the clicks off of the application's border, the title bar, and the resize widget in the lower right corner. I could test the effects of moving and resizing the window later.

I also observed that the monkey tended to open modal dialogs in the application that had to be dealt with before the main window would accept input again. The monkey would click futilely in the main window for a long time, only dismissing the dialog by accident if the dialog happened to overlap with the main window on the screen. I changed the script to detect these dialogs and dismiss them as soon as they appeared. A future enhancement could do something more interesting with the dialogs.

As I struggled with learning how to use the Win32::GuiTest module and with making my test script robust, I realized that this approach wasn't exactly simple--it's now 355 lines of code. However, it was

Tags: 

About the author

Danny R. Faught's picture Danny R. Faught

Danny R. Faught is the proprietor of Tejas Software Consulting, an independent consulting practice focusing on helping clients manage the quality of the software they produce.

StickyMinds is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!

Upcoming Events

Oct 12
Oct 15
Nov 09
Nov 09