Years ago, I worked on a project where management imposed a strict requirement of having 100% code coverage for every project under development.
This initiative had noble intentions. It was during the time when unit testing wasn’t as prevalent as it is today. Few people in the organization practiced it, and even fewer did unit testing consistently.
A group of developers had gone to a conference where many talks were devoted to unit testing. After returning, they decided to put their new knowledge into practice. Upper management supported them, and the great conversion to better programming techniques began.
Internal presentations were given. New tools were installed. And, more importantly, a new company-wide rule was imposed: all development teams had to focus on writing tests exclusively until they reached the 100% code coverage mark. After they reached this goal, any code check-in that lowered this metric had to be rejected by the build systems.
As you might guess, this didn’t play out well.
Crushed by this severe limitation, developers started to seek ways to game the system. Naturally, many of them came to the same realization: if you wrap all tests with try/catch blocks and don’t introduce any assertions in them, those tests are guaranteed to pass.
People started to mindlessly create tests for the sake of meeting the mandatory 100% coverage requirement.
Needless to say, those tests didn’t add any value to the projects. Moreover, they damaged the project because of all the effort and time they steered away from productive activities, and because of the upkeep costs required to maintain the tests moving forward.
Eventually, the requirement was lowered to 90%, then to 80%, and then retracted altogether (for the better!).
This company fell victim to the unit testing mistake #5:
"Targeting a specific test coverage number."
It’s a mistake because coverage metrics are a good negative indicator but a bad positive one.
If a metric shows that there’s too little coverage in your code base — say, only 10% — that’s a good indication that you are not testing enough.
But the reverse isn’t true: even 100% coverage isn’t a guarantee that you have a good-quality test suite. A test suite that provides high coverage can still be of poor quality.
This is especially true when the company imposes a high coverage number as a requirement. Such a policy leads to low-quality tests. In the most extreme cases, like in the story above, the company may end up with assertion-free testing, in which unit tests don’t provide any protection against bugs.
The best way to view a coverage metric is as an indicator, not a goal in and of itself.
Think of a patient in a hospital. Their high temperature might indicate a fever and is a helpful observation. But the hospital shouldn’t make the proper temperature of this patient a goal to target by any means necessary.
Otherwise, the hospital might end up with the quick and "efficient" solution of installing an air conditioner next to the patient and regulating their temperature by adjusting the amount of cold air flowing onto their skin.
Of course, this approach doesn’t make any sense.
Likewise, targeting a specific coverage number creates a perverse incentive that goes against the goal of unit testing. Instead of focusing on testing the things that matter, people start to seek ways to attain this artificial target.
Proper unit testing is difficult enough already. Imposing a mandatory coverage number only distracts developers from being mindful about what they test, and makes proper unit testing even harder to achieve.
Remember: It’s good to have a high level of coverage in core parts of your system. It’s bad to make this high level a requirement. The difference is subtle but critical.
For more on the pillars of a good unit test and how to refactor your tests to being more valuable, see by book:
(Use the nwsentr40 code during checkout on the Manning book page to get a 40% discount.)
This book will help you look at many of your tests in a new light and see which of them contribute to the project and which must be refactored or eliminated altogether.
Vladimir Khorikov