Yesterday, we discussed the unit testing mistake #2 (ignoring production code), let’s now talk about the mistake #3:
"Writing too many tests."
I know, you probably expected the opposite, something like "Writing too few tests". And how is it even possible to write too many tests? Don’t we want as many tests as possible?
Not at all.
People often assume that tests have no cost of ownership. This isn’t the case. Remember:
Code is a liability, not an asset.
Tests are code, too. The more tests you have, the more you extend the surface area for potential bugs in those tests, and the higher their upkeep cost.
That’s right, unit tests, just like any other code, are also vulnerable to bugs and require maintenance.
It’s always better to solve problems with as little code as possible. A small number of highly valuable tests will do a much better job protecting the project from bugs than a large number of mediocre tests.
But how to determine the value of your tests?
One of the most important factors is the test’s accuracy, which can be represented as the signal-to-noise ratio:
![[Enable Images]](https://khorikov.org/images/2020/2020-03-06-signal-to-noise.png)
A test is accurate insofar as it generates a strong signal (is capable of finding bugs) with as little noise (false alarms) as possible.
Notice that both properties (signal and noise) are critically important. There’s no use for a test that isn’t capable of finding any bugs, even if it doesn’t raise false alarms.
Similarly, the test’s accuracy goes to zero when it generates a lot of noise, even if it’s capable of finding all the bugs in code. These findings are simply lost in the sea of irrelevant information.
The issue with writing too many low-quality tests is that they all contribute to the noise component more so than to the signal one. This is called brittle tests.
It usually goes like this. You start writing tests (lots of them) until you reach some high coverage number, something around 90 or even 100%. At first, everything looks great. You are glad those tests verify your code and protect you from bugs.
Then, this day comes, you want to clean up a little bit. Maybe rearrange some classes, reshuffle responsibilities so that they make more sense. You know, usual refactoring. But it turns out, you now have to fix a million tests.
Have you been there? If so, you fell victim to the unit testing mistake #3 — "Writing too many tests."
Don’t worry, you aren’t alone. A lot of projects that struggle with unit testing fall into the extreme ends of the test coverage spectrum: they either don’t have unit tests at all, or they have too many of them. Nowadays, the latter is even more prevalent than the former.
For more information about how to determine the quality of a test, and also how to maximize the signal and minimize the noise, see my book:
(Use the nwsentr40 code during checkout on the Manning book page to get a 40% discount.)
Tomorrow: You’ll see what contributes to the noise component of the signal-to-noise ratio the most — giving special privileges to tests.
Vladimir Khorikov