Bob Grady describes the three possible goals of any software project: 1) meeting a specific time to market, 2) some feature set that would make the customers happy, and 3) keeping defects low. According to Grady, there is only one primary goal for a given project, and the other goals are managed within the constraints of that goal.

It makes sense to me that different customers have different project goals, depending on where the product is in the product lifetime. Table 1 (below) describes the intersection of where your market is with your possible project goals. If you’re just starting out with a product, you have technology enthusiasts as your primary customers. They want the software fast. The software has to do something, and not be too shabby, but the enthusiasts don’t need a lot of product, it just has to do something well.

Once you’ve increased your market to include the visionaries, you have slightly different concerns. Although the priorities are frequently similar, your customers are early adopters. They have a specific problem, and they want your software to fix that problem. Unfortunately, it seems as if each early adopter has his own problem, and that the problems are sufficiently different that you may be able to use a core product, but you have several one-off solutions to address each of your early adopters’ problems. The Early Adopters drive the demand for quick releases with new desired features. If your competitors release a product with a few more features, and then you release a product with those features plus a few more, you’ve seen the product leapfrog game, an example of Early Adopters in action.

Table 1: Project priorities based on where you are in the product’s lifecycle

Once you hit the Mainstream, the pragmatists take over. Pragmatists want your software to work. They care about upgrade availability, but not as much about installing upgrades or new versions as you do. (Installing upgrades costs them money and time, and they have perfectly good software to use until they choose to upgrade.) In fact, once your software has reached the late majority, even though it’s still in the mainstream, your customers don’t want releases from you any more often than they absolutely have to take them. The skeptics may or may not buy your software, but even if they buy it, nothing says they’ll actually use the software.

Let’s take word processing software as an example. Back in the ‘80s, there were several successful and competing word processing software packages. However, once PCs were available for the work and home markets, people wanted to use word processing software that was easier than using a markup language under Unix. Many companies, including Wang, Xerox, Corel, Microsoft, and Lotus, made word processing software. During the late ‘80s and early ‘90s, each of these companies released numerous versions of their products. In my opinion, the word processing wars were won with Microsoft’s release of Word 5.1. At the time, it had fewer obvious defects than its competitors, and it made its release date ahead of the competition. After Word 5.1, the other companies still released their products, but Microsoft became what Moore calls the market "gorilla." Now, with the most recent versions of Office, Word is firmly in the late majority. Those of us who use Word don’t want to upgrade any more than necessary. When we upgrade, we discover new problems from the ones we’re used to, and most of us prefer to stick with our known problems.

When you decide what’s important to your project, define where your product is in its lifecycle. Are you selling to Early Adopters or the Late Majority? Is it some combination of markets? Use those customer definitions to determine your project goals. Even when you define what’s most important for your customers, your company will still have corporate goals it needs to accomplish with a particular release.

Many project managers (PMs) talk about the tradeoffs between cost, schedule, and quality, but that’s too simplistic for what PMs actually do. In reality, PMs trade off among the things the customers value (time to market, the feature set, and the defect levels) and the things the company values (cost to market, the number of people working on the project, and the working environment of the project). Release criteria help make these tradeoffs obvious.

As testers, and especially as test leads or test managers, we’re more frequently aware of potential risks than other people—that’s our job. You can’t change the corporate needs that drive the release. After all, if you need revenue this quarter, you still need revenue. However, you can help illuminate the tradeoff decisions by discussing potential risks with the project manager. You can use the tradeoffs between the six project attributes (time to market, feature set, defect levels, cost to market, people, working environment) to best employ the test people and best use the test time to the project’s advantage.

How do you ferret out this information? I interview the various project stakeholders: the project sponsor or senior management, marketing, support, training, whoever has a stake in the project’s "doneness."

Use Context-Free Questions
I use context-free questions [Gause and Weinberg] as my primary interviewing technique for discovering what success means. These are some context-free questions:

1. What does success look like?
2. Why are these results desirable?
3. What is the solution worth to you?
4. What problems does this system solve?
5. What problems could this system create?

You’ll notice that the questions also don’t ask "how." Sometimes, when we ask how, we start designing things. You don’t want to design the project or the product; you want information about the product, to know how to evaluate the product’s state of completion throughout the project.

Once you’ve collected information about the product and the project, you can start to draft release criteria. As a nice side benefit, you’ll learn where to put your scarce testing resources and time.

1. Moore, Geoffrey. Crossing the Chasm. Harperbusiness, New York: 1995.
2. Grady, Robert. Practical Software Metrics for Project Management and Process Improvement. Prentice Hall, New York: 1992.
3. If you are interested in this article you would also benefit from the article "Release Criteria: Is this Software Done?" STQE magazine, volume 4, issue 2 (March/April 2002).