Is there such a thing as having too many unit tests?

Question

I have been tasked with writing unit tests for an existing application. After finishing my first file, I have 717 lines of test code for 419 lines of original code.

Is this ratio going to become unmanageable as we increase our code coverage?

My understanding of unit testing was to test each method in the class to ensure that every method worked as expected. However, in the pull request my tech lead noted that I should focus on higher level testing. He suggested testing 4-5 use cases that are most commonly used with the class in question, rather than exhaustively testing each function.

I trust my tech lead's comment. He has more experience than I do, and he has better instincts when it comes to designing software. But how does a multi-person team write tests for such an ambiguous standard; that is, how do I know my peers and I share the same idea for "most common use cases"?

To me, 100% unit test coverage is a lofty goal, but even if we only reached 50%, we would know that 100% of that 50% was covered. Otherwise, writing tests for part of each file leaves a lot of room to cheat.

Answer 1

Yes, with 100% coverage you will write some tests you don't need. Unfortunately, the only reliable way to determine which tests you don't need is to write all of them, then wait 10 years or so to see which ones never failed.

Maintaining a lot of tests is not usually problematic. Many teams have automated integration and system tests on top of 100% unit test coverage.

However, you are not in a test maintenance phase, you are playing catch up. It's a lot better to have 100% of your classes at 50% test coverage than 50% of your classes at 100% test coverage, and your lead seems to be trying to get you to allocate your time accordingly. After you have that baseline, then the next step is usually pushing for 100% in files that are changed going forward.

Answer 2

If you have worked on large code bases created using Test Driven Development, you would already know there can be such a thing as too many unit tests. In some cases, most of the development effort consists of updating low-quality tests that would be best implemented as invariant, precondition, and postcondition checks in the relevant classes, at run-time (i.e. testing as a side effect of a higher level test).

Another problem is the creation of poor quality designs, using cargo-cult driven design techniques, that result in a proliferation of things to test (more classes, interfaces, etc). In this case the burden may seem to be updating the testing code, but the true problem is poor quality design.

Answer 3

Answers to your questions

Is there such a thing as having too many unit tests?

Sure... You could, for example, have multiple tests which seem to be different at first glance but really test the same thing (logically depend on the same lines of "interesting" application code under test).

Or you could test internals of your code that never surface outwards (i.e., are not part of any kind of interface contract), where one could argue about whether that makes sense, at all. For example the exact wording of internal log messages or whatever.

I have been tasked with writing unit tests for an existing application. After finishing my first file, I have 717 lines of test code for 419 lines of original code.

That strikes me as quite normal. Your tests spend a lot of lines-of-code on setup and teardown on top of the actual tests. The ratio may improve, or may not. I myself am quite test heavy, and often invest more l-o-c and time on the tests than the actual code.

Is this ratio going to become unmanageable as we increase our code coverage?

The ratio does not factor in so much. There are other qualities of tests that tend to make them unmanageable. If you regularly have to refactor a whole bunch of tests when doing rather simple changes in your code, you should take a good look at the reasons. And those are not how many lines you have, but how you approach the coding of the tests.

My understanding of unit testing was to test each method in the class to ensure that every method worked as expected.

That is correct for "unit" tests in the strict sense. Here, "unit" being something like a method or a class. The point of "unit" testing is to only test one specific unit of code, not the whole system. Ideally you would remove the whole rest of the system (using doubles or whatnot).

However, in the pull request my tech lead noted that I should focus on higher level testing.

Then you fell into the trap of assuming people actually meant unit tests when they said unit tests. I have met many programmers who say "unit test" but mean something quite different.

He suggested testing 4-5 use cases that are most commonly used with the class in question, rather than exhaustively testing each function.

Sure, just concentrating on the top 80% of important code reduces load as well... I appreciate that you think highly of your boss, but this does not strike me as the optimum choice.

To me, 100% unit test coverage is a lofty goal, but even if we only reached 50%, we would know that 100% of that 50% was covered.

I do not know what "unit test coverage" is. I assume you mean "code coverage", i.e. that after running the test suite, every line of code (=100%) has been executed at least once.

This is a nice ballpark metric, but by far not the best standard one could shoot for. Just executing code lines is not the whole picture; this does not account for different paths through complicated, nested branches, for example. It is more of a metric that points its finger at pieces of code that are tested too little (obviously, if a class as 10% or 5% code coverage, then something is wrong); on the other hand a 100% coverage won't tell you whether you have tested enough or if you have tested correctly.

Integration testing

It annoys me substantially when people are constantly talking about unit testing today, by default. In my opinion (and experience), unit testing is great for libraries/APIs; in more business oriented areas (where we talk about uses cases like in the question at hand), they are not necessarily the best option.

For general application code and in the average business (where earning money, hitting deadlines and fulfilling customer satisfaction is important, and you mainly want to avoid bugs that are either directly in the user's face, or which could lead to real disasters - we are not talking NASA rocket launches here), integration or feature tests are much more useful.

Those go hand in hand with Behaviour Driven Development or Feature Driven Development; those do not work with (strict) unit tests, by definition.

To keep it short(ish), an integration/feature test exercises the whole application stack. In a web-based application, it would act like a browser clicking through the application (and no, obviously it does not have to be that simplistic, there are very powerful frameworks out there to do that - check out http://cucumber.io for an example).

Oh, to answer your last questions: you get your whole team to have a high test coverage by making sure that a new feature is only programmed after its feature test has been implemented and failed. And yes, that means every feature. This guarantees you a 100% (positive) feature coverage. It by definition guarantees that a feature of your application will never "go away". It does not guarantee a 100% code coverage (for example, unless you actively program negative features, you will not be exercising your error handling / exception handling).

It does not guarantee you a bug-free application; of course you will want to write feature tests for obvious or very dangerous buggy situations, wrong user input, hacking (for example, surrounding session management, security and such) etc.; but even only programming the positive tests has a tremendous benefit and is quite feasible with modern, powerful frameworks.

Feature/integration tests obviously have their own can of worms (e.g., performance; redundant testing of 3rd party frameworks; since you usually do not use doubles they also tend to be harder to write, in my experience...), but I'd take a 100% positive-feature-tested application over a 100% code-coverage-unit-tested application (not library!) any day.

Answer 4

Yes, it is possible to have too many unit tests. If you have 100% coverage with unit tests and no integration tests for example, you have a clear issue.

Some scenarios:

1. You over-engineer your tests to a specific implementation. Then you have to throw away unit tests when you refactor, not to say when you change implementation (a very frequent pain point when performing performance optimizations).

   A good balance between unit tests and integration tests reduce this problem without losing significant coverage.

2. You could have reasonable coverage for every commit with 20% of the tests you have, leaving the remaining 80% for integration or at least separate test passes; the major negative effects you see in this scenario are slow changes as you have to wait a large time for tests to execute.

3. You modify too much the code to allow you to test it; for example, I have seen a lot of abuse of IoC on components that will never require to be modified or at least it is costly and low priority to generalize them, but people invest a lot of time generalizing and refactoring them to allow unit testing them.

I particularly agree with the suggestion of getting 50% coverage on 100% of the files, instead of 100% coverage on 50% of the files; focus your initial efforts on the most common positive cases, and the most dangerous negative cases, don't invest too much on error handling and unusual paths, not because they are not important but because you have a limited time and an infinite testing universe, so you need to prioritize on any case.

Answer 5

Keep in mind that each test has a cost as well as a benefit. Drawbacks include:

• a test has to be written;
• a test takes (typically a very small amount of) time to run;
• a test must be maintained with the code - tests must change when APIs they are testing change;
• you may have to change your design in order to write a test (although these changes are usually for the better).

If the costs outweigh the benefits, a test is better off not written. For example, if functionality is hard to test, the API changes often, correctness is relatively unimportant, and the chance of the test finding a defect is low, you're probably better off not writing it.

As for your particular ratio of tests to code, if the code is sufficiently logic dense then that ratio can be warranted. However, it's probably not worth maintaining such a high ratio throughout a typical application.

Answer 6

Yes, there is such a thing as too many unit tests.

While testing is good every unit test is:

• A potential maintenance burden that is tightly coupled to the API

• Time that could be spent on something else

• A slice of time in the Unit Test suite

• May be adding no real value because it is in effect a duplicate of some other test having minuscule chance that some other test will pass and this test will fail.

It's wise to aim for 100% code coverage but that far from means a suite of tests each of which independently provides 100% code coverage on some specified entry point (function/method/call etc.).

Though given how difficult it can to achieve good coverage and drive bugs out the truth is probably that there is such a thing as 'the wrong unit tests' as much as 'too many unit tests'.

Pragmatics for most code indicates:

1. Make sure you have 100% coverage of entry-points (everything gets tested somehow) and aim to be close to 100% code coverage of 'non-errors' paths.

2. Test any relevant min/max values or sizes

3. Test anything you think is a funny special case particularly 'odd' values.

4. When you find a bug add a unit test that would have revealed that bug and think about whether any similar cases should be added.

For more complex algorithms consider also:

1. Doing some bulk testing of more cases.
2. Comparing result to a 'brute-force' implementation and checking the invariants.
3. Using some method of producing random test cases and checking against brute-force and post-conditions including invariants.

For example check a sorting algorithm with some randomized input and validating the data is sorted at the end by scanning it.

I'd say your tech lead is proposing 'minimal bare ass' testing. I'm offering 'highest value quality testing' and there's a spectrum in between.

Maybe your senior knows the component you're building will be embedded in some larger piece and unit tested more thoroughly when integrated.

The key lesson is to add tests when bugs are found. Which leads me to my best lesson about developing unit tests:

Focus on units not sub-units. If you're building a unit out of sub-units write very basic tests for the sub-units until they're plausible and achieve better coverage by testing sub-units through their controlling units.

So if you're writing a compiler and need to write a symbol table (say). Get the symbol table up and running with a basic test and then work on (say) the declaration parser that fills the table. Only add further tests to the symbol table 'stand-alone' unit if you find bugs in it. Otherwise increase coverage by unit tests on the declaration parser and later the whole compiler.

That gets best bang for buck (one test of the whole is testing multiple components) and leaves more capacity for re-design and refinement because only the 'outer' interface is used in tests which tends to be more stable.

Coupled with debug code testing pre-conditions, post-conditions including invariants at all levels you get maximum test coverage from minimal test implementation.

Answer 7

My understanding of unit testing was to test each method in the class to ensure that every method worked as expected.

This understanding is wrong.

Unit tests verify the behavior of the unit under test.

In that sense a unit is not necessarily "a method in a class". I like the definition of a unit by Roy Osherove in The Art of Unit Testing:

A unit is all of the production code that has the same reason to change.

Based on this, a unit test should verify every desired behavior of your code. Where the "desire" is more or less taken from the requirements.

---

However, in the pull request my tech lead noted that I should focus on higher level testing.

He is right, but in a different way than he thinks.

From your question I understand that you are the "dedicated tester" in that project.

The big misunderstanding is that he expect you to write unit tests (in contrast to "test using a unit testing framework"). Writing ynit tests is the responsibility of the developers, not the testers (in an ideal world, I know...). On the other hand you tagged this question with TDD, which implies exactly this.

Your job as the tester is to write (or execute manually) module and/or application tests. And this kind of tests should mainly verify that all the units work together smoothly. That means you have to select your test cases so that each unit is executed at least once. And that check is that is runs. The actual result is less important since it is subject to change with future requirements.

To stress the dump automobile analogy once more: How many tests are done with a car at the end of the assembly line? Exactly one: it must drive to the parking lot by itself...

The point here is:

We need to be aware of that difference between "unit tests" and "test automated using a unit testing framework".

---

To me, 100% unit test coverage is a lofty goal, but even if we only reached 50%, we would know that 100% of that 50% was covered.

Unit tests are a safety net. They give you confidence to refactor your code to reduce technical debt or add new behavior without fearing to break already implemented behavior.

You don't need 100% code coverage.

But you need 100% behavior coverage. (Yes, code coverage and behavior coverage correlate somehow, but they are not identical for the sake of it.)

If you have less than 100% behavior coverage, a successful run of your test suite means nothing since you could have changed some of the untested behavior. And you will get noticed by your client the day after your release went online...

---

Conclusion

Few tests are better than no test. No doubt!

But there is no such thing like having too much unit tests.

This is because each unit test verifies a single expectation about the codes behavior. And you cannot write more unit tests than you have expectations on your code. And a hole in your safety harness is a chance for an unwanted change to harm the production system.

Answer 8

Firstly, it's not necessarily a problem to have more lines of test than of production code. Test code is (or should be) linear and easy to comprehend - its necessary complexity is very, very low, whether or not the production code is. If the complexity of the tests starts to approach that of the production code, then you likely do have a problem.

Yes, it is possible to have too many unit tests - a simple thought experiment shows that you can continue adding tests that don't provide additional value, and that all those added tests can inhibit at least some refactorings.

The advice to test only the most common cases is flawed, in my opinion. These may act as smoke tests to save system-test time, but the really valuable tests catch cases that are hard to exercise in the whole system. For example, controlled error-injection of memory-allocation failures can be used to exercise recovery paths that may otherwise be of completely unknown quality. Or pass zero as a value you know will be used as a divisor (or a negative number that will be square-rooted), and make sure you don't get an unhandled exception.

The next most valuable tests are those that exercise the extreme limits or boundary points. For example, a function that accepts (1-based) months of the year should be tested with 0, 1, 12 and 13, so you know that the valid-invalid transitions are in the right place. It's over-testing to also use 2..11 for these tests.

You're in a difficult position, in that you have to write tests for existing code. It's easier to identify the edge cases as you are writing (or about to write) the code.

Answer 9

Absolutely, yes. I used to be an SDET for a large software company. Our small team had to maintain test code that used to be handled by a much larger team. On top of that, our product had some dependencies that were constantly introducing breaking changes, which mean constant test maintenance for us. We didn't have the option to increase the team size, so we had to throw away thousands of the less valuable tests when they failed. Otherwise, we'd never be able to keep up with the defects.

Before you dismiss this as a mere management problem, consider that many projects in the real world suffer from reduced staffing as they approach legacy status. Sometimes it even starts happening right after the first release.

Answer 10

Having more lines of test code than product code is not necessarily a problem, assuming you're refactoring your test code to eliminate copy-paste.

What's a problem is having tests that are mirrors of your implementation, with no business meaning -- for example, tests loaded with mocks and stubs and only asserting that a method calls some other method.

A great quote in the "why most unit testing is waste" paper is that unit tests should have a "broad, formal, independent oracle of correctness, and ... ascribable business value"

Answer 11

The quality of the tests is much more important than the quantity. One good test is worth any number of bad tests. (in fact a bad test has a negative value). Aiming for coverage carries the risk of lots of tests that don't really do anything useful (classic example are those tests that just test that a programming entity exists). I've even encountered tests that continue to pass once I deleted the code they were testing! (problem was excessive mocking). I don't see any reason for having a limit on the number of tests you write although not everything is as necessary to test as other things. Imperative code tends to need tests more than declarative code at the unit level because the latter doesn't have too many 'moving parts'. e.g., if you have a function foo() that just calls another function bar(), a test that checks that bar is called when foo is called seems somewhat unnecessary.

Answer 12

One thing I didn't see mentioned is that your tests need to be quick and easy for any developer to run at any time.

You don't want to have to check in to source control and wait an hour or more (depending on the size of your code base) before the tests complete to see whether your change broke something - you want to be able to do that on your own machine before you check in to source control (or at least, before you push your changes). Ideally, you should be able to run your tests with a single script or button push.

And when you run those tests locally, you want them to run fast - on the order of seconds. Any slower, and you'll be tempted to not run them enough or at all.

So, having so many tests that running them all takes minutes, or having a few overly complex tests, could be a problem.

Answer 13

To me, 100% unit test coverage is a lofty goal, but even if we only reached 50%, we would know that 100% of that 50% was covered.

It depends on what metric you are using for your test coverage

1. Statement Coverage - is each executable code statement being executed?
2. Branch Coverage - is each code branch being executed?
3. Decision Coverage - is each decision (true/false) being executed [note: very similar to branch coverage, in practical terms]
4. Condition Coverage - is each condition being executed [for simple boolean cases, conditions may be the same as decisions, but more complex decisions may have multiple conditions]

Various combinations of these also exist, depending on your situation, leading to Modified Condition/Decision Coverage (aka MC/DC, although it should be M-C/D-C).

So please be more specific when you talk about test coverage.

But irrespective of the metric used, anything less than 100% coverage means you have untested code... which means vulnerabilities or design errors may exist.