Why do we need testers when our code is 100 percent covered by unit tests run by developers? Why should testers review unit tests when creating their tests? Because 100 percent unit test code coverage is not enough.
Code coverage tools trace the execution of your code and provide metrics about that execution. One of the most common measures is statement coverage. Statement coverage gives a percentage of the statements executed over the total number of statements in the program. Many organizations set goals for unit test coverage, with a common target being 80 percent statement coverage.
Developers pride themselves on getting to 100 percent unit test coverage, and people on the project teams associate this with having high-quality code. But even though the tests execute every line of code and we then call the code “fully tested,” this can be misleading.
One hundred percent unit test coverage does not mean we had good tests, or even that the tests are complete. The tests could be missing important data and only testing with data that succeeds, failing to test data that causes failures. One hundred percent unit test coverage doesn’t say anything about missing code, missing error handling, or missing requirements.
Tests also might not actually check the functionality of the code. Merely executing the code without checking its functionality still counts in the coverage metrics.
The following examples illustrate what can go wrong with relying on unit test coverage alone. The code is Python, and tests are written with the unittest module. Hopefully these examples inspire testers to review unit tests, improve these tests, or supplement them with functional tests to fill in the gaps—even if test coverage is 100 percent.
Missing Test Cases
Consider the following function (only_correct_data), which takes several parameters, performs some math, and returns the result. It would only take one unit test to achieve 100 percent code coverage, because this function only has one line of code:
def only_correct_data (a, b, c) :
return (a / (b - c))
#only tests with data that leads to correct results
self.assertEqual( only_correct_data(1,2,3) , -1)
self.assertEqual( only_correct_data(2,3,1) , 1)
self.assertEqual( only_correct_data(0,2,3) , 0)
The test coverage is 100 percent, with three test cases that cover a negative result, a positive result, and a zero result. What is missing is the test case that would create a divide by zero error. Using the call only_correct_data(2, 2, 2) would result in the divide by zero exception. Finding missing tests cases for the developer is a great way to improve the unit tests.
Another benefit of reviewing these unit tests and making sure they are thorough is that the tester can concentrate on the overall system quality, the user experience, and acceptability to the customer. Knowing that the math is correct and verified by unit tests allows the tester to concentrate on these other factors related to quality.