Is it reasonable to not write unit tests because they tend to get commented out later or because integration tests are more valuable?

Question

I was discussing unit/integration testing with a colleague, and he made an interesting case against writing unit tests. I'm a big unit test (JUnit primarily) proponent, but am interested to hear others' takes, as he made some interesting points.

To sum up his points:

• When major code changes occur (new set of POJOs, major application refactoring, etc.), unit tests tend to be commented out rather than reworked.
• Time is better spent on integration tests covering use cases, which make the smaller-scoped tests less/not-at-all important.

Thoughts on this? I'm still pro-unit test (as I consistently see it producing improved code), although integration tests sound at least as valuable.

Answer 1

I tend to side with your friend because all too often, unit tests are testing the wrong things.

Unit tests are not inherently bad. But they often test the implementation details rather than the input/output flow. You end up with completely pointless tests when this happens. My own rule is that a good unit test tells you that you just broke something; a bad unit test merely tells you that you just changed something.

An example off the top of my head is one test that got tucked into WordPress a few years back. The functionality being tested revolved around filters that called one another, and the tests were verifying that callbacks would then get called in the correct order. But instead of (a) running the chain to verify that callbacks get called in the expected order, the tests focused on (b) reading some internal state that arguably shouldn't have been exposed to begin with. Change the internals and (b) turns red; whereas (a) only turns red if changes to the internals break the expected result while doing so. (b) was clearly a pointless test in my view.

If you have a class that exposes a few methods to the outside world, the correct thing to test in my view are the latter methods only. If you test the internal logic as well, you may end up exposing the internal logic to the outside world, using convoluted testing methods, or with a litany of unit tests that invariably break whenever you want to change anything.

With all that said, I'd be surprised if your friend is as critical about unit tests per se as you seem to suggest. Rather I'd gather he's pragmatic. That is, he observed that the unit tests that get written are mostly pointless in practice. Quoting: "unit tests tend to be commented out rather than reworked". To me there's an implicit message in there - if they tend to need reworking it is because they tend to suck. Assuming so, the second proposition follows: developers would waste less time writing code that is harder to get wrong - i.e. integration tests.

As such it's not about one being better or worse. It's just that one is a lot easier to get wrong, and indeed very often wrong in practice.

Answer 2

When major code changes occur, unit tests tend to be commented out rather than reworked.

With an undisciplined bunch of cowboy coders who think all tests getting "green" is fulfilled when you comment all existing tests out: of course. Reworking unit tests takes the same amount of discipline as writing them at the first hand, so what you have to work on here is your team's "definition of done".

Time is better spent on integration tests covering use cases which make the smaller-scoped tests less/not-at-all important.

That is neither completely wrong nor completely right. The effort for writing useful integration tests depends very much on what kind of software you are writing. If you have a software where integration tests can almost as easily be written as unit tests, they run still fast and give you a good coverage of your code, then your colleague has a point. But for lots of systems I have seen these three points cannot easily be fulfilled by integration tests, that's why you should strive to have also unit tests.

Answer 3

I don't know that I accept your colleague's arguments.

1. If unit tests are commented out, it's because someone's being lazy. That's not the unit tests' fault. And if you start using lazy as an excuse, you can argue against almost any practice that requires a little effort.
2. If you're doing it right, unit tests don't have to take a long time. They're not stopping you from doing more important work. If we all agree that fixing broken code is more important than anything else, a unit test is pretty damn important. It's an easy way to test post conditions. Without it, how do you know you've met the post condition guarantees? And if you haven't, your code is broken.

There are other, more compelling arguments against unit tests. Well, not exactly against them. Start with DHH's Test-induced design damage. He's a fun writer, so do read it. What I took from it:

If you make big design changes to accommodate your unit tests, they may hinder other goals in your project. If possible, ask an impartial person which approach is easier to follow. If your highly-testable code is hard to understand, you might lose all of the benefits you gained from unit testing. The cognitive overhead of too much abstraction slows you down and makes leaky mistakes easier. Don't discount it.

If you want to get nuanced and philosophical, there are many great resources:

• Ian Bicking's Testing considered harmful (Testing isn't enough. Tests don't make code beautiful.)
• Dan Manges's Increasing Code Coverage May Be Harmful (Using code coverage as a metric doesn't tell you if you're doing anything useful.)
• Then there's Any good arguments for not writing unit tests? [closed] from Stack Overflow. (Overconfidence, the cost is not recognized, doesn't find the insidious bugs, not a replacement for beautiful code [kinda like Bicking], etc...)

Answer 4

When major code changes occur (new set of POJOs, major application refactoring, etc.), unit tests tend to be commented out rather than reworked.

Don't do that. If you find someone who does that, murder them and make sure they do not reproduce, as their children might inherit that defective gene. The key as I see it from watching CSI television shows is to scatter a lot of misleading DNA samples everywhere. This way you pollute the crime scene with so much noise that, given the costly and time-consuming nature of DNA testing, the probability that they'll pin it down to you is astronomically low.

Answer 5

unit tests tend to be commented out rather than reworked.

At which point the code review process says "go fix the unit tests" and this isn't a problem any more :-) If your code review process isn't catching this kind of major flaw in submitted code, then that's the problem.

Answer 6

When major code changes occur (new set of POJOs, major application refactoring, etc.), unit tests tend to be commented out rather than reworked.

I always try to keep refactoring and change of functionality separate. When I need to do both, I usually commit the refactoring first.

When refactoring code without changing functionality existing unit tests are supposed to help ensure that refactoring does not accidentally break functionality. So for such a commit I would consider disabling or removing unit tests to be a major warning sign. Any developer doing it should be told not to do so when the code is being reviewed.

It is possible that changes which do not change functionality still cause unit tests to fail due to flawed unit tests. If you understand the code you are changing then the cause of such unit test failures is usually immediately obvious and easy to fix.

For example if a function takes three arguments a unit test covering the interaction between the first two arguments for the function might not have taken care to provide a valid value for the third argument. This flaw in the unit test may be exposed by a refactoring of the tested code, but is easy to fix if you understand what the code is supposed to do and what the unit test is testing.

When changing existing functionality it will usually be necessary to also change some unit tests. In this case unit tests help ensure that your code changes the functionality as intended and doesn't have unintended side effects.

When fixing bugs or adding new functionality, one usually need to add more unit tests. For those it can be helpful to commit unit tests first and commit the bug fix or new functionality later. That makes it easier to verify that the new unit tests did not pass with the older code but do pass with the newer code. This approach is not entirely without drawbacks though, so there also exist arguments in favor of committing both new unit tests and code updates simultaneously.

Time is better spent on integration tests covering use cases, which make the smaller-scoped tests less/not-at-all important.

There is some element of truth to this. If you can get coverage of the lower layers of the software stack with tests targeting the higher layers of the software stack, your tests may be more helpful when refactoring code.

I don't think you'll find an agreement on the exact distinction between a unit test and an integration test though. And I wouldn't worry if you have a test case which one developer call a unit test and another call an integration test, as long as they can agree that it is a useful test case.

Answer 7

Integration tests are made to check whether the system behaves as it's supposed to, which always has business value. Unit tests don't always do that. Unit tests could be testing dead code, for example.

There's a philosophy of testing that says that any test that doesn't give you information is a waste. When a piece of code is not being worked on, its unit tests are always (or nearly always) going to be green, giving you little or no information.

Every testing method will be incomplete. Even if you get 100% coverage by some metric, you still aren't going through nearly every possible set of input data. So, don't think unit tests are magic.

I've often seen the claim that having unit tests improves your code. In my opinion, the improvements that unit tests bring to code are in forcing people who aren't used to it to break their code into small, well-factored pieces. If you get used to designing your code in small, well-factored pieces normally, you may find that you don't need unit tests to force you to do that anymore.

Depending on how heavily you unit test and how big your program is, you may end up with an enormous amount of unit tests. If so, big changes become harder. If a big change causes a lot of failed unit tests, you can refuse to make the change, do a lot of work to fix a lot of tests, or throw the tests away. None of the choices are ideal.

Unit tests are a tool in the toolbox. They're there to serve you, not the other way around. Make sure to get out of them at least as much as you put in.

Answer 8

Unit test are definitely not the silver bullet some proponents claim them to be. But in general they have a very positive cost/benefit ratio. They will not make your code "better", maybe more modular. "better" has so many more factors than what "testability" implies. But they will assure it works in all the cases and usages you thought about, and will eliminate most of your bugs (probably the more trivial ones).

The greatest benefit of unit tests though is they make your code more flexible, and resilient to change. You can refactor or add features, and be pretty confident you didn't break anything. This alone makes them worthwhile for most projects.

Referring to the points made by your friend:

1. If adding POJOs breaks your unit tests then they are probably very badly written. POJOs are usually the language you are using to talk about you domain, and the problems you are solving, changing them obviously means your entire business logic, and probably presentation code must change. You can't expect what's assuring those parts of your program works not to change...

2. Complaining that Unit tests are bad, because people comment them out, is like complaining seat belts are bad because people don't wear them. If one comments out testing code and breaks something, he should end up in a very serious and unpleasant conversation with his boss...

3. Integration tests are far superior to unit tests. no question. especially if you are testing a product. But writing good, comprehensive integration tests, that will give you the same value, coverage and assurance of correctness that unit test give you, is incredibly hard.

Answer 9

I can only think of one argument against unit tests, and it's pretty weak.

Unit tests show the majority of their worth when you refactor or change code at a later stage. You see a huge reduction in the time required to add features and ensure you haven't broken anything.

However: There is a (small) cost in time of writing tests in the first place though.

Therefore: If a project is short enough and not requiring ongoing maintenance/updates it is possible that the cost of writing tests outweighs their benefits.