35

I was discussing unit/integration testing with a colleague, and he made an interesting case against writing unit tests. I'm a big unit test (JUnit primarily) proponent, but am interested to hear others' takes, as he made some interesting points.

To sum up his points:

  • When major code changes occur (new set of POJOs, major application refactoring, etc.), unit tests tend to be commented out rather than reworked.
  • Time is better spent on integration tests covering use cases, which make the smaller-scoped tests less/not-at-all important.

Thoughts on this? I'm still pro-unit test (as I consistently see it producing improved code), although integration tests sound at least as valuable.

  • 9
    You said it yourself: writing unit tests produces better code, because it forces you to write code that is testable. Unit-testable code has a lot of very important properties that improve its overall quality. – Robert Harvey Dec 12 '15 at 7:17
  • 1
    @Robert Harvey - Agreed - I'd be interested to see a list of these properties. – Jeff Levine Dec 12 '15 at 12:20
  • 18
    As for the fist point - you are basically saying a drawback of unit tests is that they don't work when you don't use them. You can say the same thing about any other practice. Also, your use of the passive voice is glossing over the fact that someone is actively out-commenting the unit tests. So it seems you don't have buy-in from the developers on the team, which is a different problem. – JacquesB Dec 12 '15 at 16:22
  • 1
  • 2
62

I tend to side with your friend because all too often, unit tests are testing the wrong things.

Unit tests are not inherently bad. But they often test the implementation details rather than the input/output flow. You end up with completely pointless tests when this happens. My own rule is that a good unit test tells you that you just broke something; a bad unit test merely tells you that you just changed something.

An example off the top of my head is one test that got tucked into WordPress a few years back. The functionality being tested revolved around filters that called one another, and the tests were verifying that callbacks would then get called in the correct order. But instead of (a) running the chain to verify that callbacks get called in the expected order, the tests focused on (b) reading some internal state that arguably shouldn't have been exposed to begin with. Change the internals and (b) turns red; whereas (a) only turns red if changes to the internals break the expected result while doing so. (b) was clearly a pointless test in my view.

If you have a class that exposes a few methods to the outside world, the correct thing to test in my view are the latter methods only. If you test the internal logic as well, you may end up exposing the internal logic to the outside world, using convoluted testing methods, or with a litany of unit tests that invariably break whenever you want to change anything.

With all that said, I'd be surprised if your friend is as critical about unit tests per se as you seem to suggest. Rather I'd gather he's pragmatic. That is, he observed that the unit tests that get written are mostly pointless in practice. Quoting: "unit tests tend to be commented out rather than reworked". To me there's an implicit message in there - if they tend to need reworking it is because they tend to suck. Assuming so, the second proposition follows: developers would waste less time writing code that is harder to get wrong - i.e. integration tests.

As such it's not about one being better or worse. It's just that one is a lot easier to get wrong, and indeed very often wrong in practice.

  • 13
    This, this, a hundred times this. This is a great thing about test driven development. Writing tests first will help you to write tests that are not testing details of your implementation but the intended behaviour you were trying to implement. – wirrbel Dec 12 '15 at 14:03
  • 9
    I think fragile unit tests aren't a inherent trait of unit testing, but instead of misunderstanding what unit testing is there for. What we need here is better education for developers. If you unit test implementation details, you are doing it wrong. If you make private methods public "in order to test them", you are doing it wrong. Always test the public interface & behavior, which should change less often than implementation details! – Andres F. Dec 12 '15 at 20:09
  • 2
    @DocBrown: Indeed not. What I meant is, it's so often done wrong that OP's colleague has it correct in most cases - or at least in all cases I've ever run into in unit-test obsessed companies. – Denis de Bernardy Dec 12 '15 at 21:05
  • 3
    @DenisdeBernardy: my point is: all what you wrote is surely correct with lots of wisdom from practice, but I am missing a conclusion what that means to the OPs case with his colleague, in the context of the suggestion of "integration tests as the better alternative". – Doc Brown Dec 13 '15 at 5:56
  • 5
    I heavily agree with Doc Brown. Essentially, your position seems to be that unit tests are good, but when they're badly written, they become a hassle. Thus, you don't agree with OP's colleague at all, merely that you should make sure you actually write unit tests before asserting their usefulness. I think your answer is very confusing since the first sentence states you agree with OP's colleague when it doesn't seem to be the case. It looks more like a rant about badly written unit tests (which is a problem in practice, I concede), not a case against unit tests in general. – Vincent Savard Dec 16 '15 at 15:18
38

When major code changes occur, unit tests tend to be commented out rather than reworked.

With an undisciplined bunch of cowboy coders who think all tests getting "green" is fulfilled when you comment all existing tests out: of course. Reworking unit tests takes the same amount of discipline as writing them at the first hand, so what you have to work on here is your team's "definition of done".

Time is better spent on integration tests covering use cases which make the smaller-scoped tests less/not-at-all important.

That is neither completely wrong nor completely right. The effort for writing useful integration tests depends very much on what kind of software you are writing. If you have a software where integration tests can almost as easily be written as unit tests, they run still fast and give you a good coverage of your code, then your colleague has a point. But for lots of systems I have seen these three points cannot easily be fulfilled by integration tests, that's why you should strive to have also unit tests.

  • This is very much how I felt as we were discussing it. Assuming full and relevant integration tests can be written for a use case, it seems as if the unit test would be a moot point. (Although now that I write this, I'm thinking of unexpected future cases - such as our backend being turned into a web service for another application). – Jeff Levine Dec 12 '15 at 12:23
  • +1 For "you have to work on your definition of done". Well said! – Andres F. Dec 12 '15 at 20:07
14

I don't know that I accept your colleague's arguments.

  1. If unit tests are commented out, it's because someone's being lazy. That's not the unit tests' fault. And if you start using lazy as an excuse, you can argue against almost any practice that requires a little effort.
  2. If you're doing it right, unit tests don't have to take a long time. They're not stopping you from doing more important work. If we all agree that fixing broken code is more important than anything else, a unit test is pretty damn important. It's an easy way to test post conditions. Without it, how do you know you've met the post condition guarantees? And if you haven't, your code is broken.

There are other, more compelling arguments against unit tests. Well, not exactly against them. Start with DHH's Test-induced design damage. He's a fun writer, so do read it. What I took from it:

If you make big design changes to accommodate your unit tests, they may hinder other goals in your project. If possible, ask an impartial person which approach is easier to follow. If your highly-testable code is hard to understand, you might lose all of the benefits you gained from unit testing. The cognitive overhead of too much abstraction slows you down and makes leaky mistakes easier. Don't discount it.

If you want to get nuanced and philosophical, there are many great resources:

  • +1 for that article, might be a better answer to the OPs case than all what we can write here- – Doc Brown Dec 12 '15 at 7:12
  • +1 for point number 1 about laziness being an excuse. I can't stress enough how frustrating that is. I've been there. Recently, actually. – Greg Burghardt Dec 12 '15 at 11:24
  • 3
    Reason 1 is not the unit tests' fault, but it's still a reason against unit tests. – user253751 Dec 12 '15 at 11:47
  • After reading DHH's article, make sure you look up the videos where he discussed the topic with Kent Beck and Martin Fowler (they're on youtube) - a lot of people seem to get a more extreme version of his point than he intended, and he refines it quite a bit in these discussions. – Jules Dec 21 '15 at 16:00
6

When major code changes occur (new set of POJOs, major application refactoring, etc.), unit tests tend to be commented out rather than reworked.

Don't do that. If you find someone who does that, murder them and make sure they do not reproduce, as their children might inherit that defective gene. The key as I see it from watching CSI television shows is to scatter a lot of misleading DNA samples everywhere. This way you pollute the crime scene with so much noise that, given the costly and time-consuming nature of DNA testing, the probability that they'll pin it down to you is astronomically low.

4

unit tests tend to be commented out rather than reworked.

At which point the code review process says "go fix the unit tests" and this isn't a problem any more :-) If your code review process isn't catching this kind of major flaw in submitted code, then that's the problem.

3

When major code changes occur (new set of POJOs, major application refactoring, etc.), unit tests tend to be commented out rather than reworked.

I always try to keep refactoring and change of functionality separate. When I need to do both, I usually commit the refactoring first.

When refactoring code without changing functionality existing unit tests are supposed to help ensure that refactoring does not accidentally break functionality. So for such a commit I would consider disabling or removing unit tests to be a major warning sign. Any developer doing it should be told not to do so when the code is being reviewed.

It is possible that changes which do not change functionality still cause unit tests to fail due to flawed unit tests. If you understand the code you are changing then the cause of such unit test failures is usually immediately obvious and easy to fix.

For example if a function takes three arguments a unit test covering the interaction between the first two arguments for the function might not have taken care to provide a valid value for the third argument. This flaw in the unit test may be exposed by a refactoring of the tested code, but is easy to fix if you understand what the code is supposed to do and what the unit test is testing.

When changing existing functionality it will usually be necessary to also change some unit tests. In this case unit tests help ensure that your code changes the functionality as intended and doesn't have unintended side effects.

When fixing bugs or adding new functionality, one usually need to add more unit tests. For those it can be helpful to commit unit tests first and commit the bug fix or new functionality later. That makes it easier to verify that the new unit tests did not pass with the older code but do pass with the newer code. This approach is not entirely without drawbacks though, so there also exist arguments in favor of committing both new unit tests and code updates simultaneously.

Time is better spent on integration tests covering use cases, which make the smaller-scoped tests less/not-at-all important.

There is some element of truth to this. If you can get coverage of the lower layers of the software stack with tests targeting the higher layers of the software stack, your tests may be more helpful when refactoring code.

I don't think you'll find an agreement on the exact distinction between a unit test and an integration test though. And I wouldn't worry if you have a test case which one developer call a unit test and another call an integration test, as long as they can agree that it is a useful test case.

3

Integration tests are made to check whether the system behaves as it's supposed to, which always has business value. Unit tests don't always do that. Unit tests could be testing dead code, for example.

There's a philosophy of testing that says that any test that doesn't give you information is a waste. When a piece of code is not being worked on, its unit tests are always (or nearly always) going to be green, giving you little or no information.

Every testing method will be incomplete. Even if you get 100% coverage by some metric, you still aren't going through nearly every possible set of input data. So, don't think unit tests are magic.

I've often seen the claim that having unit tests improves your code. In my opinion, the improvements that unit tests bring to code are in forcing people who aren't used to it to break their code into small, well-factored pieces. If you get used to designing your code in small, well-factored pieces normally, you may find that you don't need unit tests to force you to do that anymore.

Depending on how heavily you unit test and how big your program is, you may end up with an enormous amount of unit tests. If so, big changes become harder. If a big change causes a lot of failed unit tests, you can refuse to make the change, do a lot of work to fix a lot of tests, or throw the tests away. None of the choices are ideal.

Unit tests are a tool in the toolbox. They're there to serve you, not the other way around. Make sure to get out of them at least as much as you put in.

3

Unit test are definitely not the silver bullet some proponents claim them to be. But in general they have a very positive cost/benefit ratio. They will not make your code "better", maybe more modular. "better" has so many more factors than what "testability" implies. But they will assure it works in all the cases and usages you thought about, and will eliminate most of your bugs (probably the more trivial ones).

The greatest benefit of unit tests though is they make your code more flexible, and resilient to change. You can refactor or add features, and be pretty confident you didn't break anything. This alone makes them worthwhile for most projects.

Referring to the points made by your friend:

  1. If adding POJOs breaks your unit tests then they are probably very badly written. POJOs are usually the language you are using to talk about you domain, and the problems you are solving, changing them obviously means your entire business logic, and probably presentation code must change. You can't expect what's assuring those parts of your program works not to change...

  2. Complaining that Unit tests are bad, because people comment them out, is like complaining seat belts are bad because people don't wear them. If one comments out testing code and breaks something, he should end up in a very serious and unpleasant conversation with his boss...

  3. Integration tests are far superior to unit tests. no question. especially if you are testing a product. But writing good, comprehensive integration tests, that will give you the same value, coverage and assurance of correctness that unit test give you, is incredibly hard.

2

I can only think of one argument against unit tests, and it's pretty weak.

Unit tests show the majority of their worth when you refactor or change code at a later stage. You see a huge reduction in the time required to add features and ensure you haven't broken anything.

However: There is a (small) cost in time of writing tests in the first place though.

Therefore: If a project is short enough and not requiring ongoing maintenance/updates it is possible that the cost of writing tests outweighs their benefits.

  • +1 for attempting to earnestly answer the question that was asked. – RubberDuck Dec 12 '15 at 11:07
  • 2
    The cost of unit test is definitively not small. Good unit tests usually take more time to write than what they test. Most of the time the benefits exceed the cost. – AK_ Dec 12 '15 at 17:08
  • 1
    in theory only - when you refactor, you end up with a huge cost updating the unit tests too as they're nearly always far too granular. If you were talking about having a good integration test suite, then you'd be right. – gbjbaanb Dec 12 '15 at 17:53
  • If a project is short enough and not requiring ongoing maintenance/updates - most of the legacy systems have began as short project without tests - and end up with hiring more developers for maintaining growed system without tests – Fabio Feb 3 '18 at 8:22

protected by gnat Dec 13 '15 at 13:14

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.