Developing a usable and consumable test-metric-reporting system is a challenge for all testing organizations. This article describes a system employable by small and large organizations and all test efforts. By using existing tools, test teams can show current progress and predict future test efforts.
Test professionals know that metrics are an important project deliverable. Developing a system to create, track, and report test results is a different story. The "whys" of test metrics have been covered before, so I'll instead show you how to produce the data to address those "whys." The data includes:
- Quantifying the test team's contribution to the project
- Showing a project's testing progress (Pass %)
- Developing product quality measures (Executed/Pass or Fail)
- Gathering historical data to predict future progress
There are, of course, limitations to any measurement system:
- For large projects, it is difficult to execute every test for every new iteration.
- Test quality will affect measurement quality.
- Varying levels of specificity may create false positives or negatives.
- Limitations of the test tools (measurement devices) can affect the accuracy of metrics.
You don't have to spend a lot of money to start a test metrics program; a spreadsheet-based system will do. There are benefits to commercial test management tools that should drive your decision. Even so, many of these tools use industry-standard databases that you can read from directly to create reports that meet your organization's needs.
What Data Do I Start With?
It's easy to get carried away with tracking, but there are only a few measurements needed to start a rich test metrics program. Start with the basics and determine which are most valuable and can be used to derive other metrics.
Make sure everyone in your organization uses the same terminology. Terms such as "% Complete" to describe the "Pass %" are inaccurate unless the definitions are clear. For this article, let's use these definitions :
|Test||An instruction to perform a particular action with a clear purpose, assumed setup parameters, and expected results|
|Test Execution||Performing a test or test group|
|Test Status||The results of a test when executed|
Let's keep to a few simple and well-understood measurements:
|Pass||This test passed during this test execution.|
|Fail||This test failed during this test execution.|
|Not Executed||This test was not executed during this test execution.|
These measurements are the most basic we can track when executing a test. They can be tallied along with our total number of tests to form percentages.
Figure 1: Set of tests and its execution status
We have determined the total number of tests, their results, and the Pass % , Fail % , and Not Executed % using these simple formulas:
|Total Tests||A count of all tests to be measured|
|Total by Test Status||The count of tests by their status|
Percentage of tests in Pass status
Percentage of tests in Fail status
|Not Executed %||Percentage of tests not executed
The Daily News--Summarizing What Testing Accomplished Today
We've uncovered some useful information. At the time that this report was generated, the Pass % was 55% and 27% of the tests hadn't been executed. There should be several issues in the defect management system to record the failures.
Anyone involved can get a quick idea of the current testing status. Combining this with previous project reports, the project team can see what happened yesterday and forecast what may happen tomorrow.
By managing the reporting and raw data separately, we can add more tests and still get the data we need. Let's use an example where "Setup" is the functional area being tested (see Figure 2).
This allows the report to show information for specific areas of the project and provide more granular information. Now let's manipulate the data a bit to glean