Can you have too much unit test coverage?

Question

Google tells me that the ideal unit test coverage is 70 - 80%

(although Google's source for that doesn't look especially credible). This old post suggests that in fact it depends on what gives you confidence to release. Which is right?

If the mayor of a peaceful town with 100 residents were to hire a squad of 80 police officers, the police officers would have very little to do and the taxpayers might raise their eyebrows. If the crime rate doesn't justify the size of the squad, then it seems reasonable to reduce the size of the squad.

I think of a unit test as being like a police officer whose job is to ensure the consistent behaviour of a citizen. I want unit tests to earn their keep by preventing crimes unwanted changes in behaviour. Just as police officers need wages, food and lodgings, so unit tests cost us time and money in their construction, maintenance and execution time.

In my organisation, we require 80% unit test coverage on the changed code in each pull request. I'd guess that the cost of building the tests is roughly the same as the cost of building the code.

Our pipeline enforces zero test failures before merging, but while wrestling the code into shape, refactoring, modifying things and adding new functionality:

• We often see tests fail because they're brittle (e.g. when a constructor acquires a new argument). Brittle test failure is IMO a problem with the policing being ill-informed, not the citizens being wayward.
• Sometimes they fail because the System Under Test has been changed because the requirement changed, but the unit test hasn't been changed. Again, this is a problem with the policing, not the citizens.
• Very rarely do we see them fail for what I'd call "helpful" reasons. To me, a "helpful" reason would be when the behaviour of the System Under Test has been inadvertently changed. This is the kind of crime that justifies the police presence.

I'm thinking we could justifiably reduce the taxpayer burden required coverage. Some errors might be caught a bit later as a result (during testing instead of before the pull request), but perhaps the cost saved in not writing unit tests could offest the delay in catching the bugs.

Is there something wrong with looking at unit tests this way? Do other organisations require less than 70 - 80% coverage, and if so, why? Is a brittle test failure actually something I should be thankful for?

Answer 1

Tests are useful even when they pass.

Tests provide an example of how to use the code. I've learned how more than one codebase really works by looking at its tests. Reading code is hard. Simple examples of how it works are really valuable. So please don't think a test suite is only adding value when it shows a failure.

However, yes it is entirely possible to choke a code base with brittle tests so that refactoring actually becomes harder. A test is supposed to enable refactoring. Refactoring means changing the structure of code without changing it's behavior. A test that breaks upon a correct refactoring is failing to stick to testing behavior and is instead testing structure. So it's not really "too much testing" that kills you so much as "testing the wrong thing".

Too much of testing the wrong thing will wear you down and exhaust you because now to change anything you have to fix not just the code but all of it's tests.

Even if you carefully avoid ever writing brittle structurally focused tests and instead faithfully stick to behavioral only tests you still can't get to 100% code coverage. Why?

Because not all code is behavioral. Some is strictly structural.

Michael Feathers called this code fascia after that white connective tissue that holds sections of an orange together. I've talking about it before.

It's hard to test this code with anything but brittle tests. Thankfully that code is simple. It doesn't need tests. It doesn't have interesting behavior. There's little to gain from testing here. Testing here is more trouble than it's worth.

Keep your test's focused on the juicy behavioral bits and you'll find when a test fails you actually care.

Answer 2

Code metrics are typically only useful if developers are testing the right thing in the first place, and if the code under-test is written in a way which makes testing easy. If tests are difficult to write and brittle, that indicates issues elsewhere -- generally with the structure of the code and/or the tests themselves;

Only someone who is familiar with the overall code structure, system architecture and its requirements can know whether the number chosen by the organisation is useful or whether it needs recalibrating.

In reality the trigger for the issues you mention may be that developers are wrongly focused on the coverage number rather than writing code and tests in the right way or writing useful tests, in which case it would be impossible to assess whether the coverage number is appropriate since this would need to be addressed first.

As per Kent Beck's TDD book, easily-testable code is code where each individual behaviour can be tested in isolation. Indeed, a behaviour is what Kent Beck originally intended as the target of Unit Testing.

That is to say, a unit is a behaviour and not a class nor a function nor anything else that somebody might point to as a single "thing" in the structure of some code;

Developers who have read the book (and many more who haven't read it) frequently carry this misconception afterwards, which seems to have propagated far more widely than Kent Beck's original intention -- probably because his original intentions are far more nuanced, whereas it's very easy for people to over-simplify the advice into 'One unit is one class', Unfortunately an example of where over-simplification ends up being misleading and harmful.

While Kent Beck does not explicitly state it, a unit may frequently comprise several classes/methods depending on the structure of the code, since code structure does not play a role in defining what a unit is, as per his tweet from 2019:

Tests should be coupled to the behavior of code and decoupled from the structure of code. Seeing tests that fail on both counts.

• https://twitter.com/kentbeck/status/1182714083230904320

Also see this related thread: What kind of code would Kent Beck avoid unit testing?

---

As far as effort is concerned; if an organisation has decided on a number then its usefulness depends a great deal on developers having properly understood Kent Beck's advice on what constitutes a useful unit test in the first place.

Metrics cannot tell you anything at all about the validity or usefulness of tests, that's what code reviews are for, and more importantly education/mentoring for developers to learn more effective ways of writing tests and structuring the code to make those tests provide value.

The amount of time used for writing tests is really up to the business -- if stakeholders have decided they want to invest 50% of developer time (or more) in writing tests, then that's the value stakeholders place on the quality of software being produced by developers.

Any change in the expected test coverage should not be associated with developers spending less time on testing, indeed the justification for changing the required unit test coverage should be based on developers continuing to spend the same amount of time writing tests but either writing different kinds of tests (for example, integration tests, UI tests, end-to-end system tests) or writing the unit tests in a different way.

If there's reason to suspect the current unit test coverage isn't providing a good return on investment, then it's a discussion to have with stakeholders. If better testing/quality can be achieved in the same amount of time by a change in the way developers write tests or the kinds of tests developers are writing, then that's a valuable discussion to have with the whole development team and stakeholders.

Answer 3

We often see tests fail because they're brittle (e.g. when a constructor acquires a new argument). Brittle test failure is IMO a problem with the policing being ill-informed, not the citizens being wayward.

That isn't an example of a brittle test. That is an example of a test that flagged you when you made a change to your code. If some other part of your code relies on the original set up arguments, it is now broken, and might be about to bring your system down hard.

Sometimes they fail because the System Under Test has been changed because the requirement changed, but the unit test hasn't been changed. Again, this is a problem with the policing, not the citizens.

If a requirement changed, you have to construct new test to reflect the new requirements. If you want backwards compatibility both sets of test need to pass.

Having existing tests that flag you when you make a change does a good job of checking for unintended consequences.

Answer 4

The team that I work on practices TDD and we don't have any code coverage goals at all. Instead we focus on things like how easy it is to add new features, how often we find defects in production and how quickly we're able to fix them - these are much better indicators of software health than some arbitrary coverage numbers.

The problem with enforcing specific code coverage requirements is that it encourages devs to write (potentially brittle) "filler" tests after the fact to make up for missing coverage, rather than thinking about the requirements and coming up with useful tests upfront. You can technically achieve 100% coverage by exercising all of your code but without asserting on anything useful.

Then there's also the fact that some units (specifically those interacting with external services) cannot be fully tested locally, and high unit test coverage in these cases doesn't necessarily mean that the software actually works. An integration test is much more useful here, and once again, I'd focus on actually testing the requirements rather than aiming for some specific coverage percentage.

In my organisation, we require 80% unit test coverage on the changed code in each pull request. I'd guess that the cost of building the tests is roughly the same as the cost of building the code.

If you follow TDD, you naturally get the "right" amount of tests without having to measure coverage. And with sufficient practice, it doesn't really take any extra time since you've just pulled forward the design aspects of "building the code" and cut down the time required in the implementation step.

I'm thinking we could justifiably reduce the required coverage. Some errors might be caught a bit later as a result

These two statements aren't necessarily correlated. Focus less on coverage numbers and more on writing useful tests, and you'll still be able to catch most errors while hopefully spending less time than you do today.

Answer 5

One problem with the "just cover as good as you can" is that you never know what you are missing. One point of unit testing is the fact that you can push a button and get a reliable result.

If I were to add a new feature, and in the end there is 70% coverage, did I do well? Hard to know. Maybe it was 75% before? Then I probably did a bad job. Or did I?

I do agree that some parts of code are so hard to cover with unit tests, that it's not worth it. For me personally it's more around the last 10 percent instead of 20-30%, but I think we are on the same page: the cost is rising and at some point the cost is too high.

However, what I prefer is to have those areas I did not cover clearly marked and then excluded from my code coverage results. This way, I know when I implement something, where I am missing coverage and I don't have to wade through 20-30% of the code every time to make sure I cannot cover it because it's not worth the cost.

So technically, the coverage tool will say 100%, and it is our common understanding in the team, that means 100% of the code, that we did not explicitely rule out. Once you see it this way, there is no reason to ever be under 100%. Because if you are, it's not a conscious, informed decision, but just an oversight.

And once you have to make explicit decisions to rule something out, instead of just saying "ah, that's too hard, leave it", you will find a lot of dead code, that you cannot test, because it can never actually be called the way you build your app. A prime example is multiple layers of error recovery of errors that can actually never happen because another layer already took care of it. Or decision branches that can never happen, because another layer already filtered that data. The best result of really thinking about whether this must be excluded from coverage (instead of just saying "ah, good enough for now") is taking dead code out. Less maintenance, less code review, less testing. Code not in the program is the cheapest to have.

So I am a proponent of 100% coverage with common sense exceptions defined by the team and clearly visible in the code.

Answer 6

The desired level of quality depends on the nature of your project:

• Are you doing a Spike? 0% may even be OK
• What it is the impact in case calculating the wrong results?
• Are you handling money?
• Can people die in case of an error? (Cars, aircrafts, rockets, medical equipment)

And it depends if the effort would be better done in some other task like updating a vulnerable lib or improving the UX.

That said, safety and quality is something very different to coverage. What you comment is that your tests are not providing a lot of value. It happened to me at first. Testing is a whole ability that you need to practice and get better at.

Check a few things:

• Are you doing a test class by each main class? This may lead to a lot of dummy test with no value. Sometimes you can test the whole use case just mocking repositories.

• Are you mocking too much. Sometimes using the real classes is less brittle and much more simple

• Are you doing TDD? This typical leads to simpler code and theoretically 100% of useful test. Making them fail first will mean that you make sure that they detect errors. Coding them beforehand means that they are not coupled to the implementation (Warning. Read TDD takes time to learn)

• Consider using utilities like EqualsVerifier for testing repeated code. It is fast to code and use and will provide more coverage with real value.

• Why is there code not tested? Is it really not used? Can it be removed?

• Refactor test.

  • If the test broke... Was it a bad test? Why?
  • If you needed to change the test without a change of behaviour? Was it coupled with the implementation? Can you fix it?
  • If a single changes breaks a lot of things. It is there test duplication? Can you remove it? Are you testing everything in every test?

• If you end with some bean with untested getter and setters or the else case in the enum... do not worry. You know that they are not problematic.

Answer 7

I think the fundamental answer to my question lies here (or more correctly in Kent Beck's book, which I don't actually have, so could be mis-quoting here).

Testing is a bet. The bet pays off when your expectations are violated [my emphasis]... So, if you could, you would only write those tests that pay off. Since you can't know which tests would pay off (if you did, then you would already know and you wouldn't be learning anything), you write tests that might pay off. As you test, you reflect on which kinds of tests tend to pay off and which don't, and you write more of the ones that do pay off, and fewer of the ones that don't.

Unsurprisingly, Kent Beck's metaphor of gambling is much better than mine about police officers. I think what's been missing at my organisation is the feedback cycle which allows us to say "hey, bets like these are just not paying off". It's like being forced to just play the slots instead of capitalising on our skills at poker.

I found another really good answer from Martin Fowler in this video, where he says that with too little coverage you lack confidence that your change won't break anything; and with too much coverage you find you're spending more time changing tests than code.

Answer 8

First of all, coverage should be viewed as an outcome rather than a destination. If you are writing good unit tests (or better still, using TDD), your coverage should be high as a matter of course.

If you focus purely on coverage, then the temptation is to simply add integration tests that take data in, pass it through the layers and then test whatever comes out. High coverage here really means very little.

It is also possible to get good code coverage with bad data or even the wrong requirements. Having tests pass with spurious data and failing with good data, is of course useless - as is testing the wrong thing.

So why use code coverage at all? Well, testing all executions paths is a very hard problem for all but simple code. As Wikipedia puts it:

Testing will not catch every error in the program, because it cannot evaluate every execution path in any but the most trivial programs. This problem is a superset of the halting problem, which is undecidable.

While it isn't really possible to have too much coverage, you can of course write too many tests - or rather, too many tests testing the same thing. As any good tester will tell you, testing only ever covers a representative subset of values - usually at given points e.g. zero, one, many, silly, oops.

Take a program that adds a list of numbers as a case in point. You'd want to know what happens to the empty list, a list with one item, a list with a few items, a list with a massive amount of items and a list containing some duff items.

Clearly having tests that added one item, then two items then three etc is overkill here (for unit tests anyway). If you really must test all possible permutations, then this is where slower integration tests can be useful. These would normally be run at a quiet time such as overnight or at weekends.

A final point is that there may be code you don't want to test directly - such as code that writes to the file system, talks to a service or accesses a database. Trying to test such code is likely to build in a dependency (which could be on a developer machine, on a build server or elsewhere). This would make the test brittle. It is better just to mock out these calls and test them another way. For this reason, I'm immediately suspicious if test coverage appears to be too high. This can mean hidden dependencies and poor use of integration tests.

Answer 9

You ask the same question a number of different ways

1. Can you have too much unit test coverage?

   Yes. Anything over 100% is obviously too much.

2. Can I offset the cost of writing tests against the benefits and allow some acceptable level of bugs?

   Sure. If you have an acceptable level of bugs. and If writing unit tests is somehow burdensome? But I think most of the time there is no excuse not to have 100% unit test coverage.

3. Brittle tests are bad!

   Fix em

4. Requirements change!

   Update the test to reflect the new requirements.

5. Do other orgs require less than 70% coverage?

   Yes. Some orgs don't test at all.

Stop trying to make up excuses for not testing! its 2022 ffs! we have have this argument, just write your god dam tests!

Answer 10

I would like to approach the answer from a different angle.

We agreed on the exact % to fulfil is arbitrary. It will depend on the organization and which value provides more "confidence".

It's more important to test and guarantee that code "does what is supposed to do" rather than testing "how it's done". If that's achieved with only a 60%. Why would it be wrong?

So, why my company or customers imposes me an arbitrary coverage ratio?

Because the reliability/quality/confidence of any source code decay disproportionately when source code changes. Every new condition or new LoC has the potential to cause a lot of new execution paths that we should be aware of. Or at least to be aware of the most significant ones. So, setting a coverage ratio is a way to say "We don't want the confidence in the code to decrease with every change. We want it to be always within a threshold".¹

As for the question. I don't know if there's such a thing as enough coverage. I know that the coverage ratio doesn't always reflect if the source code does what it's supposed to. I know tho, that there could be too many tests just for the sake of the coverage. Tests whose only purpose is reaching that magic number. In my experience, every code has a threshold. But beyond that threshold, the efforts involved have very few benefits in return. They don't worth it. That's the case of tests focused on testing POJOS, DTOs, etc. Or tests focused on testing functions isolated for the sake of the coverage.

Coverage is not an end. It's not even a mean to an end. It's just an "objective" metric interpreted "subjectively".

That said, it should not be used as an argument to don't do tests! Ever! You can propose doing fewer tests when you can prove that those few are enough to provide the so-called confidence. It's then when you can back, with results, a different % or do fewer tests.

---

^{1: Whether that 80% is generated by good tests or not is a different subject.}