Avery is a project manager at a company with a virtual lock on the market. Because the customers are captive, each customer wants a beta copy immediately, so they can start using the software. Concerned that a beta with many defects would make their customers angry, Avery decided it would be more cost-effective and less risky for the developers to find and fix the majority of the defects before they started system test.

Two projects, two completely different approaches toward finding and fixing defects. We all have different attitudes toward fixing defects, specifically which ones to fix and when. The choice to fix or not to fix depends upon many factors: the kind of product you’re developing; the risks associated with shipping known or unknown defects; your development processes; and what it will cost you to fix the defect when you do fix it.

One of the least understood factors is the actual cost of fixing a defect. This cost feeds back into your choice of development lifecycle and development process and can help you decide how much risk you’re willing to take to ship or not ship the product. However, many people don’t actually know what it costs their organizations to fix a defect. If you’re not sure either, here is an estimation technique to measure that cost:

In system test, when people are 100 percent dedicated to finding and fixing defects, count the number of fixes. You know how many people (developers, testers, and anyone else) worked on the project, and you know the duration of system test. This allows you to calculate the cost to fix a defect during this phase of the project. Here’s how you can find the average cost to find and fix a defect:

Note that the number of defects you find isn’t enough information—it’s the number of defects you fix. Detecting defects is only the first step. Locating the failure, deciding how to fix it, developer testing (a.k.a. unit testing) the fix, system testing the fix, and looking for other defects this fix caused is why the fix value is what’s important. Let’s look at some examples. In these examples, I assume a person-day is $500.

Dan’s project is uncovering large numbers of defects now that they’re in system test, and even though most of the defects are quick to fix, some take very long. Avery’s project is uncovering very few defects in system test, but because there’s such a long time between the time each defect is uncovered, it appears to take a long time to fix a defect. Using the calculation above, here are Dan and Avery’s data for system test:

As long as you’re looking at the whole picture of the project, this estimation metric is good for system test fix cost. But, look at how high Avery’s cost to fix is. In reality, Avery’s project met its beta release date (20 work days in system test), at a very low risk of disappointing the customers. Dan’s test time took two months (40 work days), and even though the team has fixed 125 defects, they have over 300 defects not fixed. Avery’s cost-to-fix is high, because his team worked hard to prevent defects before system test. In fact, using the estimation technique above, Avery’s cost to fix a defect is highly inflated in system test, because they found and fixed most of the defects beforehand. Because Avery’s project detected and fixed most of the defects before system test, the estimation technique is not valid. Instead, Avery’s project can calculate the actual cost of finding and fixing their defects. Avery uses his average of 8 hours system-test time to find and fix a defect. Here’s the table with Avery’s more realistic estimate of system test cost:

Using the updated data, here’s a clearer picture of the cost to Dan and Avery’s projects to fix a defect:

Avery has high system test costs, because his project spends more time looking for defects than fixing defects However, Avery’s total defect fix cost for Avery’s larger project is lower than Dan’s smaller project. And Avery’s post-release fix cost is substantially lower.

Each project will have its own cost to fix a defect, because the cost depends on the activities undertaken in the project and when you start tracking defects, as well as cost to fix. Use your fix cost to decide how you want to proceed with this project or the next one. If your cost is too high, and you’re not yet in system test, you could try some defect detection and prevention techniques. Just make sure that if everyone is associated with finding and fixing defects that you don’t only count the fix time, that you count the detection time also.

If your find-and-fix cost is high in system test, what’s the risk of releasing early? Avery might have used his find-and-fix cost of $3333 to chose to end system test early and release early, knowing that his post-release cost would rise. Only Avery and his management could estimate the risk of releasing early.

Use the pre-release fix costs to see if you and your staff are being cost-effective in your pre-release activities. I’ve found that each organization has a typical post-release cost, not necessarily tied to the project. So I use the post-release cost to help define release criteria. This helps you manage the risk of too many defects to fix after the release.

Knowing how much it costs you to find and fix a defect allows you to ask questions about how you’re finding, fixing, and verifying defects. Yet another way to build a system with the appropriate quality.