I wanted to share this short tweet thread with you:

Graham makes a very on-point observation here.

Unit testing is hard. As hard as writing code in general. Moreover, the quality of the test code matters as much as the production code. Trying to outsource unit testing to code generation tools implies that you don’t consider tests important, which is a wrong mindset altogether.

Think about it this way. Would you use tooling to generate code for your domain model?

No, that’s an awful idea. But why, exactly?

There are (or were in the past at least) tools that allow you to generate code out of a UML model. All you have to do is put together a couple of boxes representing your domain classes, click Generate, and voilà! The domain model is ready.

So, why not?

Because UML is a very crude way to represent a domain model. You can’t capture all the details of your application’s domain knowledge with a UML diagram. And if you try to do so, you will expend so much effort that it would be easier to just write the code directly and skip the UML stage altogether.

UML is great for a high-level description of the system, but it can’t replace manual code-writing. In fact, code-writing itself is part of the discovery process where you uncover the application’s domain knowledge and see how it all works together.

The situation with tests is similar. Just as it doesn’t make sense to auto-generate code for your domain model, it doesn’t make sense to do so for unit tests covering that domain model.

The tooling can’t understand what constitutes observable behavior for your production code. It also can’t give your tests meaningful names. All it can do is pick up implementation details and test those.

And that leads to horrible tests. I’ve shared this image multiple times already (including in my book), but it’s never a bad idea to repeat important concepts:

This is test accuracy, one of the most important factors of a good unit test. You can view this metric as the signal-to-noise ratio.

A test is accurate insofar as it generates a strong signal (is capable of finding bugs) with as little noise (false alarms) as possible.

Notice that both properties (signal and noise) are critically important. There’s no use for a test that isn’t capable of finding any bugs, even if it doesn’t raise false alarms.

Similarly, the test’s accuracy goes to zero when it generates a lot of noise, even if it’s capable of finding all the bugs in code. These findings are simply lost in the sea of irrelevant information.

The issue with auto-generated tests is that they generate a huge amount of noise. Almost any refactoring of the underlying code makes such tests fail, even if the refactoring didn’t introduce any bugs. You will be drowned in all the false positives such tests produce.

So, don’t cut corners when writing tests. Treat the test code as a first-class citizen. Don’t outsource unit testing to scaffolding or code generation tooling.

--Vlad

https://enterprisecraftsmanship.com