If the mayor of a peaceful town with 100 residents were to hire a squad of 80 police officers, the police officers would have very little to do and the taxpayers might raise their eyebrows. If the crime rate doesn't justify the size of the squad, then it seems reasonable to reduce the size of the squad.
I think of a unit test as being like a police officer whose job is to ensure the consistent behaviour of a citizen. I want unit tests to earn their keep by preventing
crimes unwanted changes in behaviour. Just as police officers need wages, food and lodgings, so unit tests cost us time and money in their construction, maintenance and execution time.
In my organisation, we require 80% unit test coverage on the changed code in each pull request. I'd guess that the cost of building the tests is roughly the same as the cost of building the code.
Our pipeline enforces zero test failures before merging, but while wrestling the code into shape, refactoring, modifying things and adding new functionality:
- We often see tests fail because they're brittle (e.g. when a constructor acquires a new argument). Brittle test failure is IMO a problem with the policing being ill-informed, not the citizens being wayward.
- Sometimes they fail because the System Under Test has been changed because the requirement changed, but the unit test hasn't been changed. Again, this is a problem with the policing, not the citizens.
- Very rarely do we see them fail for what I'd call "helpful" reasons. To me, a "helpful" reason would be when the behaviour of the System Under Test has been inadvertently changed. This is the kind of crime that justifies the police presence.
I'm thinking we could justifiably reduce the
taxpayer burden required coverage. Some errors might be caught a bit later as a result (during testing instead of before the pull request), but perhaps the cost saved in not writing unit tests could offest the delay in catching the bugs.
Is there something wrong with looking at unit tests this way? Do other organisations require less than 70 - 80% coverage, and if so, why? Is a brittle test failure actually something I should be thankful for?