Many of the testers reading this article use white box testing techniques based upon the structure of the code. Others conduct primarily black box testing based upon some external specification such as the requirements. And of course, some of them do both. In this short article I will focus on the value of requirements based testing.
For the last twelve years, I have been the co-owner of a restaurant in Tampa, Florida. A couple of years ago I had a meeting with a group of local restaurateurs and we discussed (among more important topics, like how to sell more beer and wine) the software we used to run our restaurants. Here's a little anecdote told to me by one of my fellow restaurateurs.
"We used to have a clunky old application: slow, awkward, really just plain ugly, and full of bugs. When we got new hardware we also "upgraded" to new state-of-the-art software. We fell in love immediately with its look and feel: it was slick with touch screens, icons, and other nifty features.
We quickly learned though, that the builders had obviously never worked in a restaurant-maybe they'd never even eaten in one. One particularly annoying problem was that after a customer's check was closed out it couldn't be reopened. So, when the customer yelled out, "Oh, just give me one more beer after all," we had to delete the old ticket, create a new ticket, and re-enter all the old data plus the new order. (If you simply opened a second check the customer would have to pay two separate checks.) Meanwhile the customer is getting very thirsty and consequently unhappy.
When we complained to the vendor, they assured us that the code did exactly what it was designed to do. Finally, after complaints from other users, the vendor issued an "enhancement" that allowed an existing ticket to be modified. Yea!!! Unfortunately though, when we ran our daily and weekly reports, we eventually discovered that the original ticket and the modified ticket were both added into our sales summaries, which played havoc with our management metrics and caused us to overpay our state sales taxes.
The vendor of this system had created a very attractive product that appeared to work seamlessly. Unfortunately the product was built upon a flawed concept. The developers and testers didn't seem to understand the restaurant business. Even though the code did what it was designed to do, the customer was unable to use the software to do some of the most basic functions needed to effectively run a restaurant. As important as it is to employ techniques that excel in finding bugs (think orthogonal arrays, state transition diagrams, structural testing, etc.) it is also important to ensure that the systems we build satisfy the user's actual requirements.
Notice that I didn't say requirements specifications. Please note the distinction drawn between a requirement and a requirement specification. The requirement is a need of the user; the requirement specification is our imperfect attempt at documenting this need.
A complaint that many test professionals make is that it is difficult, if not impossible, to test a requirements specification. This is because by their very nature, requirements specifications are inaccurate and constantly changing. They change because the users change, the business changes, some requirements turn our to be infeasible or too expensive, the users may not know exactly what they want, or we maybe we just got 'em wrong in the first place. Very few, if any, organizations will ever successfully create a complete, accurate requirements specification prior to the commencement of design and coding.
But just because we can't document the requirements perfectly doesn't change the fact that all systems have requirements. These requirements may or may not exist in a requirements specification, but they do exist, otherwise why are we building the system in the first place?
In my view, the key is to have a good starting point-a reasonable view