100 Percent Unit Test Coverage Is Not Enough

[article]
Summary:
Many people equate 100 percent unit test coverage with high code quality, but that is not enough. Code coverage tools only measure whether the tests execute the code; they make no judgment on the effectiveness of the tests. Testers should review unit tests, even if they have high coverage levels, and either help improve the tests or supplement them with extra tests where necessary.

Why do we need testers when our code is 100 percent covered by unit tests run by developers? Why should testers review unit tests when creating their tests? Because 100 percent unit test code coverage is not enough. 

Code coverage tools trace the execution of your code and provide metrics about that execution. One of the most common measures is statement coverage. Statement coverage gives a percentage of the statements executed over the total number of statements in the program. Many organizations set goals for unit test coverage, with a common target being 80 percent statement coverage.

Developers pride themselves on getting to 100 percent unit test coverage, and people on the project teams associate this with having high-quality code. But even though the tests execute every line of code and we then call the code “fully tested,” this can be misleading.

One hundred percent unit test coverage does not mean we had good tests, or even that the tests are complete. The tests could be missing important data and only testing with data that succeeds, failing to test data that causes failures. One hundred percent unit test coverage doesn’t say anything about missing code, missing error handling, or missing requirements.

Tests also might not actually check the functionality of the code. Merely executing the code without checking its functionality still counts in the coverage metrics.

The following examples illustrate what can go wrong with relying on unit test coverage alone. The code is Python, and tests are written with the unittest module. Hopefully these examples inspire testers to review unit tests, improve these tests, or supplement them with functional tests to fill in the gaps—even if test coverage is 100 percent.

Missing Test Cases

Consider the following function (only_correct_data), which takes several parameters, performs some math, and returns the result. It would only take one unit test to achieve 100 percent code coverage, because this function only has one line of code:

def only_correct_data (a, b, c) :

    return (a / (b - c))

def test_only_correct_data(self):
   #only tests with data that leads to correct results

    self.assertEqual( only_correct_data(1,2,3) , -1)
    self.assertEqual( only_correct_data(2,3,1) , 1)
    self.assertEqual( only_correct_data(0,2,3) , 0)

The test coverage is 100 percent, with three test cases that cover a negative result, a positive result, and a zero result. What is missing is the test case that would create a divide by zero error. Using the call only_correct_data(2, 2, 2) would result in the divide by zero exception. Finding missing tests cases for the developer is a great way to improve the unit tests.

Another benefit of reviewing these unit tests and making sure they are thorough is that the tester can concentrate on the overall system quality, the user experience, and acceptability to the customer. Knowing that the math is correct and verified by unit tests allows the tester to concentrate on these other factors related to quality.

User Comments

7 comments
new
Nikhil Bhandari's picture

Good article John Ruberto. Thanks for sharing your thoughts.

 

There few more tips that can be useful in this context

  • Do mutation testing, ensure you have validated your tests
  • Validate your assertions, many a times assertions are missing in un Unit tests
  • Along with Unit Testing, have your Integration and Functional testing measure code coverage
  • Use Mocks, best way to cover all your negative scenarios
  • Have a seperate set of unit tests for UI and measure code coverage

 

 

July 11, 2017 - 12:58am
new
John Ruberto's picture

Thanks Nikhil!  Great insignts.  

July 17, 2017 - 6:36pm
new
Gene Gotimer's picture

I really enjoy mutation testing for myself when checking for missing tests and asserts. I use pitest for Java, but I've heard good things about Ninja Turtles for .Net.

July 11, 2017 - 2:40pm
new
John Ruberto's picture

Thanks Gene, I agree test the tests...

July 17, 2017 - 6:37pm
new
Kathy Iberle's picture

John, do you find that using TDD causes people to  write those missing unit tests right away?   I notice that most of these cases involve failing to handle possible input values, which I would expect to be fairly obvious in TDD.  I don't think TDD would trigger everything - for instance, failing the handle an exception thrown by an API call internally probably wouldn't occur to the developer right off - but seems TDD would help make the tests more thorough from the start.  What is your experience? . Kathy Iberle

July 17, 2017 - 5:51pm
new
John Ruberto's picture

Great question.  I definitely see a strong correlation between coverage level & those that use TDD, but I don't think I have a good data set to show that TDD leads to better tests (or not).  I have to believe that the "missing requirements" examples might be better covered with TDD (and for sure BDD).  Also, the focus on creating a failing test that eventually passes must put some good thinking behind the assertions - leading to better outcomes than my example here.  I believe, but don't have the data....

July 17, 2017 - 6:36pm
new
Kathy Iberle's picture

The old "defensive programming" practices required the programmer to pay attention to the parameters & purpose of the module.  I would think that TDD would do the same, and requires creation of the unit tests too, which is a bonus.

July 25, 2017 - 8:21pm

About the author

StickyMinds is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.