Sense of unit tests without TDD

Question

We have new (quite big) project starting, that we planned to develop using TDD.

The idea of TDD failed (many business and non-business reasons), but right now we have a conversation - should we write unit tests anyway, or not. My friend says that there is no (or close to zero) sense in writing unit tests without TDD, we should focus only on integration tests. I believe the opposite, that there is still some sense in writing plain unit tests, just to make code more futureproof. What do you think?

Added: I think that this is not a duplicate of >>this question<< - I understand difference between UT and TDD. My question is not about differences, but about sense of writing Unit Tests without TDD.

Answer 1

TDD is used mainly (1) to ensure coverage, (2) and to drive maintainable, understandable, testable design. If you don't use TDD, you don't get guaranteed code coverage. But that in no way means that you should abandon that goal and blithely live on with 0% coverage.

Regression tests were invented for a reason. The reason is that in the long run, they save you more time in prevented errors than they take in additional effort to write. This has been proven over and over again. Therefore, unless you are seriously convinced that your organization is much, much better at software engineering than all the gurus who recommend regression testing (or if you plan on going down very soon so that there is no long run for you), yes, you should absolutely have unit tests, for exactly the reason that applies to virtually every other organization in the world: because they catch errors earlier than integration tests do, and that will save you money. Not writing them is like passing up free money just lying around in the street.

Answer 2

I have a relevant anecdote from something that's going on right now for me. I'm on a project that does not use TDD. Our QA folks are moving us in that direction, but we're a small outfit and it has been a long, drawn-out process.

Anyways, I was recently using a third-party library to do a specific task. There was an issue regarding the use of that library, so it's been put on me to essentially write a version of that same library on my own. In total, it ended up being about 5,000 lines of executable code and about 2 months of my time. I know lines of code is a poor metric, but for this answer I feel it's a decent indicator of magnitude.

There was one particular data structure I needed which would allow me to keep track of an arbitrary number of bits. Since the project is in Java, I chose Java's BitSet and modified it a bit(I needed the ability to track the leading 0s as well, which Java's BitSet doesn't do for some reason.....). After reaching ~93% coverage I started writing some tests that would actually stress the system I had written. I needed to benchmark certain aspects of the functionality to ensure they would be fast enough for my end requirements. Unsurprisingly, one of the functions I had overridden from the BitSet interface was absurdly slow when dealing with large bit sets(hundreds of millions of bits in this case). Other overridden functions relied on this one function, so it was a huge bottle neck.

What I ended up doing was going to the drawing board, and figuring out a way to manipulate the underlying structure of BitSet, which is a long[]. I designed the algorithm, asked colleagues for their input, and then I set about writing the code. Then, I ran the unit tests. Some of them broke, and the ones that did pointed me exactly to where I needed to look in my algorithm in order to fix it. After fixing all of the errors from the unit tests, I was able to say that the function works as it should. At the very least, I could be as confident that this new algorithm worked as well as the previous algorithm.

Of course, this is not bullet proof. If there's a bug in my code that the unit tests aren't checking for, then I won't know it. But of course, that exact same bug could have been in my slower algorithm as well. However, I can say with a high degree of confidence that I don't have to worry about the wrong output from that particular function. Pre-existing unit tests saved me hours, perhaps days, of trying to test the new algorithm to ensure it was correct.

That is the point of having unit tests regardless of TDD - that is to say, unit tests will do this for you in TDD and outside of TDD all the same, when you end up refactoring/maintaining the code. Of course, this should be paired with regular regression testing, smoke testing, fuzzy testing, etc, but unit testing, as the name states, tests things on the smallest, atomic level possible, which gives you direction on where errors have popped up.

In my case, without the existing unit tests, I would somehow have to come up with a method of ensuring the algorithm works all of the time. Which, in the end...sounds a lot like unit testing, doesn't it?

Answer 3

You can break code roughly into 4 categories:

1. Simple and rarely changes.
2. Simple and frequently changes.
3. Complex and rarely changes.
4. Complex and frequently changes.

Unit tests become more valuable (likely to catch important bugs) the further down the list you go. In my personal projects, I almost always do TDD on category 4. On category 3 I usually do TDD unless manual testing is simpler and faster. For example, antialiasing code would be complex to write, but much easier to verify visually than writing a unit test, so the unit test would only be worth it to me if that code changed frequently. The rest of my code I only put under unit test after I find a bug in that function.

It's sometimes difficult to know beforehand what category a certain block of code fits into. The value of TDD is you don't accidentally miss any of the complex unit tests. The cost of TDD is all the time you spend writing the simple unit tests. However, usually people experienced with a project know with a reasonable degree of certainty what category different parts of code fit into. If you aren't doing TDD, you should at least try to write the most valuable tests.

Answer 4

Actually Uncle Bob mentioned a very interesting point in one of his Clean Coders videos. He said that the Red-Green-Refactor cycle can be applied in 2 ways.

1st is the conventional TDD way. Write a failing test then make the test pass and finally refactor.

2nd way is to write a very small piece of production code and immediately follow that by its unit test then refactor.

The idea is to go in very small steps. Of course you lose the verification from the production code that your test went from red to green, but in some cases where I was working mostly with junior developers who refused even to try to understand TDD it proved to be somewhat effective.

Again I repeat (and this was emphasized by Uncle Bob) the idea is to go in very small steps and to immediately test the piece of production code that was just added.

Answer 5

Whether it be unit, component, integration or acceptance tests, the important part is that it must be automated. Not having automated tests is a fatal mistake for any kind of software, from the simple CRUDs to the most complex calculations. The reasoning is that writing the automated tests will always cost less than the continuous need for running all tests manually when you don't, by orders of magnitude. After you have them written, you just have to push a button to see if they pass or fail. Manual tests will always take a long time to run, and depend on humans (living creatures who get bored, may lack attention and so on) to be able to check if the tests pass or fail. In short, always write automated tests.

Now, about the reason why your colleague might be against doing any kind of unit testing without TDD: It's probably because it is harder to trust tests written after the production code. And if you can't trust your automated tests, they are worth nothing. Following the TDD cycle, you must first make a test fail (for the right reason) to be allowed to write the production code to make it pass (and no more). This process is essentially testing your tests, so you can trust them. Not to mention the fact that writing tests before the actual code pushes you to design your units and components to be more easily testable (high levels of decoupling, SRP applied, etc...). Although, of course, doing TDD takes discipline.

Instead, if you write all the production code first, when you write the tests for it, you'll expect them to pass at first run. This is very problematic, because you may have created a test that covers 100% of your production code, without asserting the correct behavior (may not even perform any assertions! I've seen this happen), since you can't see it failing first to check if it is failing for the right reason. Thus, you may have false-positives. False-positives will eventually break the trust in your test suite, essentially forcing people to resort to manual testing again, so you have the cost of both processes (writing tests + manual tests).

This means you must find another way to test your tests, like TDD does. So you resort to debugging, commenting parts of the production code, etc., in order to be able to trust the tests. The problem is that the process of "testing your tests" is much slower this way. Adding this time to the time that you'll spend running ad-hoc tests manually (because you don't have automatic tests while you are coding the production code), in my experience, results in an overall process that is much slower than practicing TDD "by the book" (Kent Beck - TDD by Example). Also, I'm willing to make a bet here and say that truly "testing your tests" after they were written takes much more discipline than TDD.

So maybe your team can reconsider the "business and non-business reasons" for not doing TDD. In my experience, people tend to think TDD is slower compared to just writing unit tests after the code is done. This assumption is flawed, as you have read above.

Answer 6

Often the quality of tests depends on their provenance. I'm regularly guilty of not doing 'real' TDD - I write some code to prove that the strategy I'd like to use actually works, then cover each case that code is meant to support with tests afterward. Usually the unit of code is a class, to give you a general idea of how much work I'll happily do without test coverage - that is to say, not a large amount. What this means is that the semantic meaning of the tests matches up nicely with the system under test in its 'finished' state - because I wrote them knowing what cases the SUT fulfils and how it fulfils them.

Conversely, TDD with its policy of aggressive refactoring tends to obsolete tests at least as fast as you can write them as the public interface of the system under test changes. I personally find the mental workload of both designing the functional units of the application and keeping the semantics of the tests that cover it in sync to be too high to maintain my concentration and the test maintenance frequently slips. The codebase ends up with tests hanging around that don't test anything of value or are just plain wrong. If you have the discipline and mental capacity to keep the test suite up to date, by all means, practice TDD as rigorously as you'd like. I don't, so I've found it less successful for that reason.