Is Code Coverage a Silver Bullet?

I vividly remember a project we were executing where release after release, the client would define a code coverage number for us to achieve as part of the test exit criteria. While code coverage is a good number to look at in terms of testing reach achieved in a testing cycle, is it foolproof? Is this metric a silver bullet for understanding the team’s coverage and vouching for testing scope before sign-off?

This is a question I have been closely looking at over the years. Recently, I read a blog post about how code coverage is measured at Google across the varied programming languages they use and how they test it for both their daily runs and the individual commits they do. It is very exciting to see how organizations are now embracing code coverage runs early on in their test cycles as opposed to just a “nice to check” run at the end.

How Code Coverage Is Measured

Many of us are familiar with the idea of looking at the percentage of lines of code as executed by tests. This is usually only new code created by the project and does not include code libraries, functions, and operating system calls that might each contain branch conditions. We might also be talking about just module or function coverage (are all the functions called at least once?), statement coverage, or branch coverage.

In my experience, statement coverage is the most common and easiest to obtain measure for code coverage. It is relatively easy to get from automated unit and integration tests, but lately we’ve seen teams tend to be open to running all tests, even manual exploration on instrumented builds. When the test cycle is done, the tools generate a list of what code was covered—and, perhaps more importantly, what was not.

As testers, we understand that certain sections of the code may be very difficult to reach during the course of an execution cycle. A combination of static code analysis and code coverage results can point out areas of the code base that might need additional testing. Some tools can also indicate if the path is dead and impossible to enter, or pin down paths that are just hard to reach. In specific domains—such as the aviation industry, where evaluations are very stringent—code coverage is a very important metric to showcase test coverage in getting certifications, which gives us some confidence in the product’s quality.

The Downside of Code Coverage

Like with many testing tools, there can be problems with over-reliance on code coverage numbers. Code coverage alone just shows us that the code was executed; it does not show us that the application is fit for its purpose. In other words, 100 percent coverage does not actually verify requirements in any way.

For example, unless we test requirements to see if they really map to end-user expectations, how do we know if the system under development is what will be accepted in the marketplace? If we do not look at live issues from the past or try to understand end-user pain points, how can we validate them in the implemented system? If we do not take the time to understand competition in the marketplace, how do we know whether our product will have an edge or at least be able to survive when released?

All of these questions are not going to be exposed by a code coverage run. If the tester were to solely look at what he has covered from his code coverage results and work on improving them, he may miss the other angles discussed above, which are growing in importance in today’s markets.

A Team Effort

The best code coverage effort is a collective effort. Implementing the tools generally requires developers and testers working together. An upside of this is that both sides understand where the other is coming from a little better. The test management team has to look at the code coverage results of the entire team to understand what the gaps are and identify areas that need additional focus to map with the product’s quality expectations.

Consequently, we cannot discount certain fringe benefits code coverage gives a tester. For example, code coverage runs provide good discussion grounds for developers and testers to collaborate on. Result analysis after code coverage runs gives testers the opportunity to understand system internals and code functioning in greater detail rather than purely looking at it as a black box. It promotes system understanding, both at a grassroots level as well as a module or intermodule functioning level, giving testers newer perspectives to think about and try in their testing scenarios. And it promotes the tester’s debugging and troubleshooting skills when trying to understand what the gaps are and how to increase test coverage in such gap areas. All of these benefits are undoubtedly valuable.

Silver Bullet? Not Yet

The main point the team needs to understand is that code coverage is not and will not become the silver bullet to solving their coverage issues. It is, however, one of the core steps to resolving these issues. The test team has to use it in conjunction with other approaches, including requirements traceability, requirements verification, defect analysis, and end-user feedback reports. to be able to achieve the desired level of test coverage.