29

We have new (quite big) project starting, that we planned to develop using TDD.

The idea of TDD failed (many business and non-business reasons), but right now we have a conversation - should we write unit tests anyway, or not. My friend says that there is no (or close to zero) sense in writing unit tests without TDD, we should focus only on integration tests. I believe the opposite, that there is still some sense in writing plain unit tests, just to make code more futureproof. What do you think?

Added: I think that this is not a duplicate of >>this question<< - I understand difference between UT and TDD. My question is not about differences, but about sense of writing Unit Tests without TDD.

6
  • 22
    I'm curious what reasoning your friend has for such an absurd stance...
    – Telastyn
    Aug 8, 2014 at 13:05
  • 11
    I'd wager that the vast majority of projects with some unit tests are not using TDD.
    – Casey
    Aug 8, 2014 at 19:46
  • 2
    What will be your integration levels? What will be your units? How often will you be refactoring below each test level? How fast will your integration tests be to run? How easy will they be to write? How many combinatorial cases do different parts of your code generate? etc... If you don't know the answers to these, then maybe it's too early to make firm decisions. Sometimes TDD is great. Sometimes the benefits are less clear. Sometimes Unit tests are essential. Sometimes a decent set of integration tests buys you almost as much, and are much more flexible... Keep your options open. Aug 8, 2014 at 20:45
  • 2
    As some practical advice from experience, don't test everything if you're not doing TDD. Make a determination about what kinds of tests are valuable. I've found that unit tests on pure input/output methods are extraordinarily valuable, while integration tests at a very high level of the application (e.g., actually sending web requests on a web application) are also extraordinarily valuable. Watch out for mid-tier integration tests and unit tests that require a lot of mock set up. Also watch this video: youtube.com/watch?v=R9FOchgTtLM.
    – jpmc26
    Aug 9, 2014 at 16:19
  • Your update doesn't make sense with regards to the question you asked. If you understand the difference between TDD and Unit Tests, then what's holding you back from writing unit tests. Voting to leave your question closed although I could see an argument for closing as "unclear on what you're asking" instead of duplicate.
    – user53019
    Aug 11, 2014 at 11:04

6 Answers 6

52

TDD is used mainly (1) to ensure coverage, (2) and to drive maintainable, understandable, testable design. If you don't use TDD, you don't get guaranteed code coverage. But that in no way means that you should abandon that goal and blithely live on with 0% coverage.

Regression tests were invented for a reason. The reason is that in the long run, they save you more time in prevented errors than they take in additional effort to write. This has been proven over and over again. Therefore, unless you are seriously convinced that your organization is much, much better at software engineering than all the gurus who recommend regression testing (or if you plan on going down very soon so that there is no long run for you), yes, you should absolutely have unit tests, for exactly the reason that applies to virtually every other organization in the world: because they catch errors earlier than integration tests do, and that will save you money. Not writing them is like passing up free money just lying around in the street.

15
  • 13
    "If you don't use TDD, you don't get guaranteed code coverage.": I do not think so. You can develop for two days, and for the next two days you write the tests. The important point is that you do not consider a feature finished until you have the wanted code coverage.
    – Giorgio
    Aug 8, 2014 at 13:14
  • 5
    @DougM - In an ideal world maybe...
    – Telastyn
    Aug 8, 2014 at 14:53
  • 7
    Sadly TDD goes hand-in-hand with mocking and until people stop doing that all it proves is that your test runs faster. TDD is dead. Long live testing.
    – Deleted
    Aug 8, 2014 at 15:00
  • 17
    TDD does not guarantee code coverage. That's a dangerous assumption. You can code against tests, pass those tests, but still have edge cases. Aug 8, 2014 at 17:20
  • 4
    @MickyDuncan I'm not quite sure I entirely understand your concern. Mocking is a perfectly valid technique used to isolate one component from the others so that tests of that component's behaviour can be performed independently. Yes, taken to extremes it can lead to over-engineered software, but so can any development technique if used inappropriately. Besides, as DHH states in the article you cite, the idea of only ever using full system tests is just as bad, if not actually worse. It's important to use judgment to decide what the best way of testing any particular feature is.
    – Jules
    Aug 9, 2014 at 3:53
22

I have a relevant anecdote from something that's going on right now for me. I'm on a project that does not use TDD. Our QA folks are moving us in that direction, but we're a small outfit and it has been a long, drawn-out process.

Anyways, I was recently using a third-party library to do a specific task. There was an issue regarding the use of that library, so it's been put on me to essentially write a version of that same library on my own. In total, it ended up being about 5,000 lines of executable code and about 2 months of my time. I know lines of code is a poor metric, but for this answer I feel it's a decent indicator of magnitude.

There was one particular data structure I needed which would allow me to keep track of an arbitrary number of bits. Since the project is in Java, I chose Java's BitSet and modified it a bit(I needed the ability to track the leading 0s as well, which Java's BitSet doesn't do for some reason.....). After reaching ~93% coverage I started writing some tests that would actually stress the system I had written. I needed to benchmark certain aspects of the functionality to ensure they would be fast enough for my end requirements. Unsurprisingly, one of the functions I had overridden from the BitSet interface was absurdly slow when dealing with large bit sets(hundreds of millions of bits in this case). Other overridden functions relied on this one function, so it was a huge bottle neck.

What I ended up doing was going to the drawing board, and figuring out a way to manipulate the underlying structure of BitSet, which is a long[]. I designed the algorithm, asked colleagues for their input, and then I set about writing the code. Then, I ran the unit tests. Some of them broke, and the ones that did pointed me exactly to where I needed to look in my algorithm in order to fix it. After fixing all of the errors from the unit tests, I was able to say that the function works as it should. At the very least, I could be as confident that this new algorithm worked as well as the previous algorithm.

Of course, this is not bullet proof. If there's a bug in my code that the unit tests aren't checking for, then I won't know it. But of course, that exact same bug could have been in my slower algorithm as well. However, I can say with a high degree of confidence that I don't have to worry about the wrong output from that particular function. Pre-existing unit tests saved me hours, perhaps days, of trying to test the new algorithm to ensure it was correct.

That is the point of having unit tests regardless of TDD - that is to say, unit tests will do this for you in TDD and outside of TDD all the same, when you end up refactoring/maintaining the code. Of course, this should be paired with regular regression testing, smoke testing, fuzzy testing, etc, but unit testing, as the name states, tests things on the smallest, atomic level possible, which gives you direction on where errors have popped up.

In my case, without the existing unit tests, I would somehow have to come up with a method of ensuring the algorithm works all of the time. Which, in the end...sounds a lot like unit testing, doesn't it?

8

You can break code roughly into 4 categories:

  1. Simple and rarely changes.
  2. Simple and frequently changes.
  3. Complex and rarely changes.
  4. Complex and frequently changes.

Unit tests become more valuable (likely to catch important bugs) the further down the list you go. In my personal projects, I almost always do TDD on category 4. On category 3 I usually do TDD unless manual testing is simpler and faster. For example, antialiasing code would be complex to write, but much easier to verify visually than writing a unit test, so the unit test would only be worth it to me if that code changed frequently. The rest of my code I only put under unit test after I find a bug in that function.

It's sometimes difficult to know beforehand what category a certain block of code fits into. The value of TDD is you don't accidentally miss any of the complex unit tests. The cost of TDD is all the time you spend writing the simple unit tests. However, usually people experienced with a project know with a reasonable degree of certainty what category different parts of code fit into. If you aren't doing TDD, you should at least try to write the most valuable tests.

1
  • 1
    When working on code like you suggest with your antialiasing example, I find the best thing is to develop the code experimentally, then add some characterization tests to ensure that I don't accidentally break the algorithm later. Characterization tests are very quick and easy to develop, so the overhead of doing this is very low.
    – Jules
    Aug 9, 2014 at 4:00
1

Actually Uncle Bob mentioned a very interesting point in one of his Clean Coders videos. He said that the Red-Green-Refactor cycle can be applied in 2 ways.

1st is the conventional TDD way. Write a failing test then make the test pass and finally refactor.

2nd way is to write a very small piece of production code and immediately follow that by its unit test then refactor.

The idea is to go in very small steps. Of course you lose the verification from the production code that your test went from red to green, but in some cases where I was working mostly with junior developers who refused even to try to understand TDD it proved to be somewhat effective.

Again I repeat (and this was emphasized by Uncle Bob) the idea is to go in very small steps and to immediately test the piece of production code that was just added.

1
  • " ... the idea is to go in very small steps and to immediately test the piece of production code that was just added.": I do not agree. What you describe if good when you already have a clear idea of what you want to do and you want to work on the details, but you need to get the big picture first. Otherwise, by going in very small steps (test, develop, test, develop) you can get lost in the details. "If you do not know where you are going you might not get there."
    – Giorgio
    Aug 12, 2014 at 14:08
1

Whether it be unit, component, integration or acceptance tests, the important part is that it must be automated. Not having automated tests is a fatal mistake for any kind of software, from the simple CRUDs to the most complex calculations. The reasoning is that writing the automated tests will always cost less than the continuous need for running all tests manually when you don't, by orders of magnitude. After you have them written, you just have to push a button to see if they pass or fail. Manual tests will always take a long time to run, and depend on humans (living creatures who get bored, may lack attention and so on) to be able to check if the tests pass or fail. In short, always write automated tests.

Now, about the reason why your colleague might be against doing any kind of unit testing without TDD: It's probably because it is harder to trust tests written after the production code. And if you can't trust your automated tests, they are worth nothing. Following the TDD cycle, you must first make a test fail (for the right reason) to be allowed to write the production code to make it pass (and no more). This process is essentially testing your tests, so you can trust them. Not to mention the fact that writing tests before the actual code pushes you to design your units and components to be more easily testable (high levels of decoupling, SRP applied, etc...). Although, of course, doing TDD takes discipline.

Instead, if you write all the production code first, when you write the tests for it, you'll expect them to pass at first run. This is very problematic, because you may have created a test that covers 100% of your production code, without asserting the correct behavior (may not even perform any assertions! I've seen this happen), since you can't see it failing first to check if it is failing for the right reason. Thus, you may have false-positives. False-positives will eventually break the trust in your test suite, essentially forcing people to resort to manual testing again, so you have the cost of both processes (writing tests + manual tests).

This means you must find another way to test your tests, like TDD does. So you resort to debugging, commenting parts of the production code, etc., in order to be able to trust the tests. The problem is that the process of "testing your tests" is much slower this way. Adding this time to the time that you'll spend running ad-hoc tests manually (because you don't have automatic tests while you are coding the production code), in my experience, results in an overall process that is much slower than practicing TDD "by the book" (Kent Beck - TDD by Example). Also, I'm willing to make a bet here and say that truly "testing your tests" after they were written takes much more discipline than TDD.

So maybe your team can reconsider the "business and non-business reasons" for not doing TDD. In my experience, people tend to think TDD is slower compared to just writing unit tests after the code is done. This assumption is flawed, as you have read above.

0

Often the quality of tests depends on their provenance. I'm regularly guilty of not doing 'real' TDD - I write some code to prove that the strategy I'd like to use actually works, then cover each case that code is meant to support with tests afterward. Usually the unit of code is a class, to give you a general idea of how much work I'll happily do without test coverage - that is to say, not a large amount. What this means is that the semantic meaning of the tests matches up nicely with the system under test in its 'finished' state - because I wrote them knowing what cases the SUT fulfils and how it fulfils them.

Conversely, TDD with its policy of aggressive refactoring tends to obsolete tests at least as fast as you can write them as the public interface of the system under test changes. I personally find the mental workload of both designing the functional units of the application and keeping the semantics of the tests that cover it in sync to be too high to maintain my concentration and the test maintenance frequently slips. The codebase ends up with tests hanging around that don't test anything of value or are just plain wrong. If you have the discipline and mental capacity to keep the test suite up to date, by all means, practice TDD as rigorously as you'd like. I don't, so I've found it less successful for that reason.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.