137

I have been tasked with writing unit tests for an existing application. After finishing my first file, I have 717 lines of test code for 419 lines of original code.

Is this ratio going to become unmanageable as we increase our code coverage?

My understanding of unit testing was to test each method in the class to ensure that every method worked as expected. However, in the pull request my tech lead noted that I should focus on higher level testing. He suggested testing 4-5 use cases that are most commonly used with the class in question, rather than exhaustively testing each function.

I trust my tech lead's comment. He has more experience than I do, and he has better instincts when it comes to designing software. But how does a multi-person team write tests for such an ambiguous standard; that is, how do I know my peers and I share the same idea for "most common use cases"?

To me, 100% unit test coverage is a lofty goal, but even if we only reached 50%, we would know that 100% of that 50% was covered. Otherwise, writing tests for part of each file leaves a lot of room to cheat.

  • 145
    It depends. Are you writing a game of tic-tac-toe, or are you writing code to manage a nuclear reactor? – Bryan Oakley May 3 '17 at 17:57
  • 11
    With sufficiently many unit tests, you could detect exotic hardware implementation issues like the Pentium FDIV bug or correlations in cryptographic primitives, so it seems like there's not a hard limit past which no further unit tests could be useful. Just a practical limit on when it's too costly. – Nat May 3 '17 at 18:45
  • 5
    Testing at higher levels will provide you with better perspective of the real coverage. By real coverage I mean the one is more likely to happen during the regular usage of the system. That's the kind of coverage you want to achive first. In the 50% that last to reach could have YAGNI or dead code that once removed will contribute to increase the overall coverage too. – Laiv May 3 '17 at 19:42
  • 5
    If you get too much tests (which you don't seem to have at the moment) the most likely issue is that the code you are testing does too much. So single responsibility is not adhered. When code is well split up the testing also won't create much burden. If the classes do a lot, have lots of side effects etc. it will become a nightmare. – Luc Franken May 4 '17 at 7:40
  • 12
    The sqlite testing document is a fun read: sqlite.org/testing.html. Quote: "the SQLite library consists of approximately 122.9 KSLOC of C code. By comparison, the project has 745 times as much test code and test scripts - 91596.1 KSLOC." – user60561 May 4 '17 at 15:13

11 Answers 11

179

Yes, with 100% coverage you will write some tests you don't need. Unfortunately, the only reliable way to determine which tests you don't need is to write all of them, then wait 10 years or so to see which ones never failed.

Maintaining a lot of tests is not usually problematic. Many teams have automated integration and system tests on top of 100% unit test coverage.

However, you are not in a test maintenance phase, you are playing catch up. It's a lot better to have 100% of your classes at 50% test coverage than 50% of your classes at 100% test coverage, and your lead seems to be trying to get you to allocate your time accordingly. After you have that baseline, then the next step is usually pushing for 100% in files that are changed going forward.

  • 11
    Thanks for your answer. It helped put my question in perspective and addressed the real problem -- my attitude! +1 – user2954463 May 3 '17 at 19:27
  • 43
    @astra Your attitude isn't that bad. It's good to question why. To answer your other question excellent question: "How do I know my peers and I share the same idea for "most common use cases"? You get them to look at your tests. Look at theirs. Talk about them. You'll learn a lot and maybe they will too. Code reviewing tests is seldom a waste of time. Though I tend to do mine at a terminal rather than a conference room – candied_orange May 3 '17 at 21:37
  • 18
    A test never failing in 10 years doesn't even guarantee it being unnecessary, it could end up failing in year 11. – Pharap May 4 '17 at 13:30
  • 24
    Pragmatically, you could take the opposite approach. Write the tests that you think cover the common cases. But then, every time you encounter a failure, write a test to cover that area. – stannius May 4 '17 at 17:49
  • 10
    @Pharap My only issue with this answer is that there is the implicit assumption that a test can only add value when it fails. A good unit test also provides a great form of living documentation. It also added value when you wrote the test, by forcing you to think about reusability/composability/encapulation. Untested code in my experience tends to be inflexible monolithic beasts. – ArTs May 5 '17 at 1:51
66

If you have worked on large code bases created using Test Driven Development, you would already know there can be such a thing as too many unit tests. In some cases, most of the development effort consists of updating low-quality tests that would be best implemented as invariant, precondition, and postcondition checks in the relevant classes, at run-time (i.e. testing as a side effect of a higher level test).

Another problem is the creation of poor quality designs, using cargo-cult driven design techniques, that result in a proliferation of things to test (more classes, interfaces, etc). In this case the burden may seem to be updating the testing code, but the true problem is poor quality design.

  • 16
    Upvoted for pointing out pre-conditions, post-conditions and invariants should be treated as unit-testing. That way every use is a unit-test when that debug code is active. – Persixty May 3 '17 at 20:35
  • This is a great answer and aligns perfectly with my experience. – Tony Ennis May 5 '17 at 12:41
  • And one more problem: If you have gated checkins (you really should!) having large amounts of low quality, possibly even long running tests will slow everything down without providing any real benefits. And then obviously the fun fact where you change one thing in a class and hundreds of tests fail. – Voo May 5 '17 at 20:07
  • 3
    This is a much better answer than the accepted one! "In some cases, most of the development effort consists of updating low-quality tests" - I have experienced this and it sucks. More than having no tests at all, in some respects. – Benjamin Hodgson May 5 '17 at 23:15
36

Answers to your questions

Is there such a thing as having too many unit tests?

Sure... You could, for example, have multiple tests which seem to be different at first glance but really test the same thing (logically depend on the same lines of "interesting" application code under test).

Or you could test internals of your code that never surface outwards (i.e., are not part of any kind of interface contract), where one could argue about whether that makes sense, at all. For example the exact wording of internal log messages or whatever.

I have been tasked with writing unit tests for an existing application. After finishing my first file, I have 717 lines of test code for 419 lines of original code.

That strikes me as quite normal. Your tests spend a lot of lines-of-code on setup and teardown on top of the actual tests. The ratio may improve, or may not. I myself am quite test heavy, and often invest more l-o-c and time on the tests than the actual code.

Is this ratio going to become unmanageable as we increase our code coverage?

The ratio does not factor in so much. There are other qualities of tests that tend to make them unmanageable. If you regularly have to refactor a whole bunch of tests when doing rather simple changes in your code, you should take a good look at the reasons. And those are not how many lines you have, but how you approach the coding of the tests.

My understanding of unit testing was to test each method in the class to ensure that every method worked as expected.

That is correct for "unit" tests in the strict sense. Here, "unit" being something like a method or a class. The point of "unit" testing is to only test one specific unit of code, not the whole system. Ideally you would remove the whole rest of the system (using doubles or whatnot).

However, in the pull request my tech lead noted that I should focus on higher level testing.

Then you fell into the trap of assuming people actually meant unit tests when they said unit tests. I have met many programmers who say "unit test" but mean something quite different.

He suggested testing 4-5 use cases that are most commonly used with the class in question, rather than exhaustively testing each function.

Sure, just concentrating on the top 80% of important code reduces load as well... I appreciate that you think highly of your boss, but this does not strike me as the optimum choice.

To me, 100% unit test coverage is a lofty goal, but even if we only reached 50%, we would know that 100% of that 50% was covered.

I do not know what "unit test coverage" is. I assume you mean "code coverage", i.e. that after running the test suite, every line of code (=100%) has been executed at least once.

This is a nice ballpark metric, but by far not the best standard one could shoot for. Just executing code lines is not the whole picture; this does not account for different paths through complicated, nested branches, for example. It is more of a metric that points its finger at pieces of code that are tested too little (obviously, if a class as 10% or 5% code coverage, then something is wrong); on the other hand a 100% coverage won't tell you whether you have tested enough or if you have tested correctly.

Integration testing

It annoys me substantially when people are constantly talking about unit testing today, by default. In my opinion (and experience), unit testing is great for libraries/APIs; in more business oriented areas (where we talk about uses cases like in the question at hand), they are not necessarily the best option.

For general application code and in the average business (where earning money, hitting deadlines and fulfilling customer satisfaction is important, and you mainly want to avoid bugs that are either directly in the user's face, or which could lead to real disasters - we are not talking NASA rocket launches here), integration or feature tests are much more useful.

Those go hand in hand with Behaviour Driven Development or Feature Driven Development; those do not work with (strict) unit tests, by definition.

To keep it short(ish), an integration/feature test exercises the whole application stack. In a web-based application, it would act like a browser clicking through the application (and no, obviously it does not have to be that simplistic, there are very powerful frameworks out there to do that - check out http://cucumber.io for an example).

Oh, to answer your last questions: you get your whole team to have a high test coverage by making sure that a new feature is only programmed after its feature test has been implemented and failed. And yes, that means every feature. This guarantees you a 100% (positive) feature coverage. It by definition guarantees that a feature of your application will never "go away". It does not guarantee a 100% code coverage (for example, unless you actively program negative features, you will not be exercising your error handling / exception handling).

It does not guarantee you a bug-free application; of course you will want to write feature tests for obvious or very dangerous buggy situations, wrong user input, hacking (for example, surrounding session management, security and such) etc.; but even only programming the positive tests has a tremendous benefit and is quite feasible with modern, powerful frameworks.

Feature/integration tests obviously have their own can of worms (e.g., performance; redundant testing of 3rd party frameworks; since you usually do not use doubles they also tend to be harder to write, in my experience...), but I'd take a 100% positive-feature-tested application over a 100% code-coverage-unit-tested application (not library!) any day.

  • 1
    Integration tests are great but are no replacement for unit tests, also not for business applications. There are multiple problems with them: a) they by definition take a long time to run (it also means incremental tests are pretty much useless), b) they make it incredibly hard to pinpoint the actual problem (oh 50 integration tests just failed, what change caused that?) and c) they cover the same code paths repeatedly. – Voo May 5 '17 at 20:18
  • 1
    a) is a problem because it makes running the tests on gated check-ins cumbersome and makes it less likely that programmers will run the tests repeatedly while developing, which coupled with b) decreases efficiency and the ability to quickly diagnose bugs. c) means that changing one little thing can easily cause dozens or hundreds (been there) of your integration tests to fail, which means you'll spend a loot of time fixing them. This also means that integration tests mostly test only happy paths, because writing these tests targeted is cumbersome or impossible. – Voo May 5 '17 at 20:20
  • 1
    @Voo, all of what you wrote is true, and as far as I can tell I already mentioned all problems you noted in the answer... – AnoE May 7 '17 at 19:59
  • If you agree with that summary I really don't see how you can come to the conclusion that you'd prefer integration tests to unit tests. Comprehensive integration test suites of large programs take hours or even days to run, they're great to have but are nigh useless during actual development. And your acceptance tests (which everyone is doing, right?) will catch many of the same issues that integration tests would find that would get missed by unit tests - the opposite is not true though. – Voo May 8 '17 at 20:14
24

Yes, it is possible to have too many unit tests. If you have 100% coverage with unit tests and no integration tests for example, you have a clear issue.

Some scenarios:

  1. You over-engineer your tests to a specific implementation. Then you have to throw away unit tests when you refactor, not to say when you change implementation (a very frequent pain point when performing performance optimizations).

    A good balance between unit tests and integration tests reduce this problem without losing significant coverage.

  2. You could have reasonable coverage for every commit with 20% of the tests you have, leaving the remaining 80% for integration or at least separate test passes; the major negative effects you see in this scenario are slow changes as you have to wait a large time for tests to execute.

  3. You modify too much the code to allow you to test it; for example, I have seen a lot of abuse of IoC on components that will never require to be modified or at least it is costly and low priority to generalize them, but people invest a lot of time generalizing and refactoring them to allow unit testing them.

I particularly agree with the suggestion of getting 50% coverage on 100% of the files, instead of 100% coverage on 50% of the files; focus your initial efforts on the most common positive cases, and the most dangerous negative cases, don't invest too much on error handling and unusual paths, not because they are not important but because you have a limited time and an infinite testing universe, so you need to prioritize on any case.

  • 2
    Which is not a problem with the unit tests, but with the organisation for having its priorities wrong by demanding a specific number for unit test coverage without spending the resources to create and execute proper tests at other levels. – jwenting May 4 '17 at 11:11
  • 2
    Agree strongly on #3 and would also extend it to manually passing in instances of lower-level classes to higher-level classes. If a high-level-thing relies on some low-level-thing to get something done, that's ok. If it hides the details of that from callers, I'd call that good design. But if you then make low-level-thing part of high-level-thing's interface and make callers pass it in because it makes your tests pretty, now the tail is wagging the dog. (If low-level-thing is reused in a lot of places, and changes a lot, that changes things. In my experience that hasn't been typical.) – johncip May 4 '17 at 19:41
  • I love your description @johncip, definitely that is a frequent example of how a nice class gets horrible by adding a bunch of unnecessary required parameters to the constructor... – Bruno Guardia May 5 '17 at 16:20
19

Keep in mind that each test has a cost as well as a benefit. Drawbacks include:

  • a test has to be written;
  • a test takes (typically a very small amount of) time to run;
  • a test must be maintained with the code - tests must change when APIs they are testing change;
  • you may have to change your design in order to write a test (although these changes are usually for the better).

If the costs outweigh the benefits, a test is better off not written. For example, if functionality is hard to test, the API changes often, correctness is relatively unimportant, and the chance of the test finding a defect is low, you're probably better off not writing it.

As for your particular ratio of tests to code, if the code is sufficiently logic dense then that ratio can be warranted. However, it's probably not worth maintaining such a high ratio throughout a typical application.

12

Yes, there is such a thing as too many unit tests.

While testing is good every unit test is:

  • A potential maintenance burden that is tightly coupled to the API

  • Time that could be spent on something else

  • A slice of time in the Unit Test suite
  • May be adding no real value because it is in effect a duplicate of some other test having minuscule chance that some other test will pass and this test will fail.

It's wise to aim for 100% code coverage but that far from means a suite of tests each of which independently provides 100% code coverage on some specified entry point (function/method/call etc.).

Though given how difficult it can to achieve good coverage and drive bugs out the truth is probably that there is such a thing as 'the wrong unit tests' as much as 'too many unit tests'.

Pragmatics for most code indicates:

  1. Make sure you have 100% coverage of entry-points (everything gets tested somehow) and aim to be close to 100% code coverage of 'non-errors' paths.

  2. Test any relevant min/max values or sizes

  3. Test anything you think is a funny special case particularly 'odd' values.

  4. When you find a bug add a unit test that would have revealed that bug and think about whether any similar cases should be added.

For more complex algorithms consider also:

  1. Doing some bulk testing of more cases.
  2. Comparing result to a 'brute-force' implementation and checking the invariants.
  3. Using some method of producing random test cases and checking against brute-force and post-conditions including invariants.

For example check a sorting algorithm with some randomized input and validating the data is sorted at the end by scanning it.

I'd say your tech lead is proposing 'minimal bare ass' testing. I'm offering 'highest value quality testing' and there's a spectrum in between.

Maybe your senior knows the component you're building will be embedded in some larger piece and unit tested more thoroughly when integrated.

The key lesson is to add tests when bugs are found. Which leads me my best lesson about developing unit tests:

Focus on units not sub-units. If you're building a unit out of sub-units write very basic tests for the sub-units until they're plausible and achieve better coverage by testing sub-units through their controlling units.

So if you're writing a compiler and need to write a symbol table (say). Get the symbol table up and running with a basic test and then work on (say) the declaration parser that fills the table. Only add further tests to the symbol table 'stand-alone' unit if you find bugs in it. Otherwise increase coverage by unit tests on the declaration parser and later the whole compiler.

That gets best bang for buck (one test of the whole is testing multiple components) and leaves more capacity for re-design and refinement because only the 'outer' interface is used in tests which tends to be more stable.

Coupled with debug code testing pre-conditions, post-conditions including invariants at all levels you get maximum test coverage from minimal test implementation.

  • 4
    I wouldn't say that 100% coverage is pragmatic. 100% coverage is an extremely high standard. – Bryan Oakley May 3 '17 at 17:58
  • Unfortunately even the randomized method can miss errors. There is no substitute for proofs, even if informal. – Frank Hileman May 3 '17 at 18:07
  • @BryanOakley Point taken. That's an overstatement. But it is more important to get close to it than people give credit. "I tested the easy path it's all good" is always going to cause problems later. – Persixty May 3 '17 at 18:11
  • @FrankHileman The question wasn't "Is unit testing a good substitute for designing software carefully, static checking logic and proving algorithms" then the answer is 'no'. Neither method will produce high-quality software on their own. – Persixty May 3 '17 at 18:19
3

Firstly, it's not necessarily a problem to have more lines of test than of production code. Test code is (or should be) linear and easy to comprehend - its necessary complexity is very, very low, whether or not the production code is. If the complexity of the tests starts to approach that of the production code, then you likely do have a problem.

Yes, it is possible to have too many unit tests - a simple thought experiment shows that you can continue adding tests that don't provide additional value, and that all those added tests can inhibit at least some refactorings.

The advice to test only the most common cases is flawed, in my opinion. These may act as smoke tests to save system-test time, but the really valuable tests catch cases that are hard to exercise in the whole system. For example, controlled error-injection of memory-allocation failures can be used to exercise recovery paths that may otherwise be of completely unknown quality. Or pass zero as a value you know will be used as a divisor (or a negative number that will be square-rooted), and make sure you don't get an unhandled exception.

The next most valuable tests are those that exercise the extreme limits or boundary points. For example, a function that accepts (1-based) months of the year should be tested with 0, 1, 12 and 13, so you know that the valid-invalid transitions are in the right place. It's over-testing to also use 2..11 for these tests.

You're in a difficult position, in that you have to write tests for existing code. It's easier to identify the edge cases as you are writing (or about to write) the code.

3

My understanding of unit testing was to test each method in the class to ensure that every method worked as expected.

This understanding is wrong.

Unit tests verify the behavior of the unit under test.

In that sense a unit is not necessarily "a method in a class". I like the definition of a unit by Roy Osherove in The Art of Unit Testing:

A unit is all of the production code that has the same reason to change.

Based on this, a unit test should verify every desired behavior of your code. Where the "desire" is more or less taken from the requirements.


However, in the pull request my tech lead noted that I should focus on higher level testing.

He is right, but in a different way than he thinks.

From your question I understand that you are the "dedicated tester" in that project.

The big misunderstanding is that he expect you to write unit tests (in contrast to "test using a unit testing framework"). Writing ynit tests is the responsibility of the developers, not the testers (in an ideal world, I know...). On the other hand you tagged this question with TDD, which implies exactly this.

Your job as the tester is to write (or execute manually) module and/or application tests. And this kind of tests should mainly verify that all the units work together smoothly. That means you have to select your test cases so that each unit is executed at least once. And that check is that is runs. The actual result is less important since it is subject to change with future requirements.

To stress the dump automobile analogy once more: How many tests are done with a car at the end of the assembly line? Exactly one: it must drive to the parking lot by itself...

The point here is:

We need to be aware of that difference between "unit tests" and "test automated using a unit testing framework".


To me, 100% unit test coverage is a lofty goal, but even if we only reached 50%, we would know that 100% of that 50% was covered.

Unit tests are a safety net. They give you confidence to refactor your code to reduce technical debt or add new behavior without fearing to break already implemented behavior.

You don't need 100% code coverage.

But you need 100% behavior coverage. (Yes, code coverage and behavior coverage correlate somehow, but they are not identical for the sake of it.)

If you have less than 100% behavior coverage, a successful run of your test suite means nothing since you could have changed some of the untested behavior. And you will get noticed by your client the day after your release went online...


Conclusion

Few tests are better than no test. No doubt!

But there is no such thing like having too much unit tests.

This is because each unit test verifies a single expectation about the codes behavior. And you cannot write more unit tests than you have expectations on your code. And a hole in your safety harness is a chance for an unwanted change to harm the production system.

2

Absolutely, yes. I used to be an SDET for a large software company. Our small team had to maintain test code that used to be handled by a much larger team. On top of that, our product had some dependencies that were constantly introducing breaking changes, which mean constant test maintenance for us. We didn't have the option to increase the team size, so we had to throw away thousands of the less valuable tests when they failed. Otherwise, we'd never be able to keep up with the defects.

Before you dismiss this as a mere management problem, consider that many projects in the real world suffer from reduced staffing as they approach legacy status. Sometimes it even starts happening right after the first release.

  • 4
    "On top of that, our product had some dependencies that were constantly introducing breaking changes, which mean constant test maintenance for us." - Those tests you say require maintenance sound like the valuable ones if your dependencies constantly break. – CodeMonkey May 4 '17 at 7:50
  • 2
    That's not a problem with the tests, but with the organisation. – jwenting May 4 '17 at 11:12
  • 2
    @CodeMonkey The dependencies didn't break. They were being updated in ways that required changes to our product. Yes, the tests were valuable, but not nearly as valuable as others. Automated tests are most valuable when the equivalent manual test is hard. – mrog May 5 '17 at 15:39
  • 2
    @jwenting Yes, it's an organizational problem, not a code problem. But that doesn't change the fact that there were too many tests. A failing test that can't be investigated is useless, regardless of the cause. – mrog May 5 '17 at 15:41
  • What is an "SDET"? – Peter Mortensen May 5 '17 at 16:34
1

Having more lines of test code than product code is not necessarily a problem, assuming you're refactoring your test code to eliminate copy-paste.

What's a problem is having tests that are mirrors of your implementation, with no business meaning -- for example, tests loaded with mocks and stubs and only asserting that a method calls some other method.

A great quote in the "why most unit testing is waste" paper is that unit tests should have a "broad, formal, independent oracle of correctness, and ... ascribable business value"

0

One thing I didn't see mentioned is that your tests need to be quick and easy for any developer to run at any time.

You don't want to have to check in to source control and wait an hour or more (depending on the size of your code base) before the tests complete to see whether your change broke something - you want to be able to do that on your own machine before you check in to source control (or at least, before you push your changes). Ideally, you should be able to run your tests with a single script or button push.

And when you run those tests locally, you want them to run fast - on the order of seconds. Any slower, and you'll be tempted to not run them enough or at all.

So, having so many tests that running them all takes minutes, or having a few overly complex tests, could be a problem.

protected by gnat May 4 '17 at 17:51

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.