What are the disadvantages of test-first programming?

Question

It's all the rage nowadays. "Everyone" recommends it. That in and of itself makes me suspicious.

What are some disadvantages you have found when doing test-first (test-driven) development? I'm looking for personal experiences from knowledgeable practitioners--I can read the hypothetical musings of a hundred wannabes elsewhere on the internet.

I ask not because I am looking to hate TDD, but because it is my job to improve software development process, and the more we can learn about the problems people encounter, the better chance we have of improving the process.

Answer 1

There are quite a few, but the advantages far outweigh the disadvantages.

There's a steep learning curve.

Many developers seem to expect that they can be efficient with test-first programming right from day one. Unfortunately it takes a lot of time to gain experience and program at the same speed as before. You can't get around it.

To be more specific, it's very easy to get wrong. You can very easily (with very good intentions) end up writing a whole bunch of tests which are either difficult to maintain or testing the wrong stuff. It's difficult to give examples here - these kind of issues simply take experience to solve. You need to have a good feel of separating concerns and designing for testability. My best advice here would be to do pair-programming with someone who knows TDD really well.

You do more coding up front.

Test-first means you can't skip tests (which is good) and means you'll end up writing more code up front. This means more time. Again, you can't get around it. You get rewarded with code that's easier to maintain, extend and generally less bugs, but it takes time.

Can be a tough sell to managers.

Software managers are generally only concerned with timelines. If you switch to test-first programming and you're suddenly taking 2 weeks to complete a feature instead of one, they're not gonna like it. This is definitely a battle worth fighting and many managers are enlightened enough to get it, but it can be a tough sell.

Can be a tough sell to fellow developers.

Since there's a steep learning curve not all developers like test-first programming. In fact, I would guess that most developers don't like it at first. You can do things like pair-programming to help them get up to speed, but it can be a tough sell.

In the end, the advantages outweigh the disadvantages, but it doesn't help if you just ignore the disadvantages. Knowing what you're dealing with right from the start helps you to negotiate some, if not all, of the disadvantages.

Answer 2

Test-first assumes you are writing code that is:

• testable in a unit-test manner
• that what you are developing has an obvious approach and will not require extensive prototyping or experimentation
• that you will not need to refactor too heavily or that you have the time to repeatedly rewrite hundreds or thousands of test cases
• nothing is sealed
• everything is modular
• everything is injectable or mockable
• that your organization places a high enough value on low-defects to justify the resource sink
• that there is something of benefit to test at a unit test level

If your project does not meet those requirements you will have difficulty. The promoters of TDD have no good answers to this other to suggest that you redesign your product to better fall within those lines. There are situations where that is impossible or undesirable.

In practice there can also me a huge problem with people thinking the test-first tests actually prove anything about the correct function of the program. In many cases this is not true, but even in the cases where it is true it is far from a complete picture of correctness. People see hundreds of passing tests and assume it is safe to test less since before TDD they only did a few hundred test cases anyway. In my experience TDD means that you need to have even more integration tests as the developers will also have the false security and the pain of changing all the tests to do a big redactor can lead developers to make interesting work arounds.

Examples:

My personal best example is when writing security code for asp.net. If they are meant to run in a hostile environment from the machine config, they are gac'ed, signed and sealed, and because they are running against IIS god objects they are very difficult to mock very much of correctly. Add in some constraints for performance and memory use and you very quickly lose the flexibility to use placeholder objects in the remaining areas.

Any sort of micro controller or other low resource environment code may not be possible to do truly OO style design as the abstractions do not optimize out and you have low resource limits. The same can be said for high performance routines in many cases as well.

Answer 3

The biggest drawback I've seen is not with TDD itself but with practitioners. They take a dogmatic and zealot approach where everything must be tested. Sometimes (many times that is), that is not necessary. Also, it might not be practical (.ie. introducing an organization to TDD.)

A good engineer finds trade-offs and applies the right balance of when/where/how to apply test-first. Also, if you find yourself constantly spending much more time developing tests instead of actual code (by a factor of 2-3 or more), you are in trouble.

In other words, be pragmatic and reasonable with TDD (or anything in software development for that matter.)

Answer 4

I started doing TDD in early August 2009 and convinced my whole company to switch to it in September/October 2009. Presently, the whole dev team is fully converted, and committing untested code to the repo is considered a Bad Thing and thrown upon. It has been working great for us, and I cannot imagine switching back to cowboy coding.

However, there are two problems that are pretty noticeable.

The test suite has to be maintained

When you are serious about TDD, you'll end up writing lots of tests. Moreover, it takes some time and experience to realize what is the right granularity of tests (overdoing is almost as bad as underdoing it). These tests are also code, and they are susceptible to bitrot. This means that you have to maintain them as everything else: update it when you upgrade libraries they depend on, refactor from time to time... When you make big changes in your code, a lot of tests will suddenly become out of date or even plain wrong. If you are lucky, you can simply delete them, but a lot of times you'll end up extracting the useful bits and adapting them to the new architecture.

Testing abstractions leak from time to time

We are using Django, which has a pretty great testing framework. However, sometimes it makes assumptions that are slightly at odds with reality. For example, some middleware may break the tests. Or, some tests make assumptions about a caching backend. Also, if you are using a "real" db (not SQLite3), then preparing the db for the tests will take a lot of time. Sure, you can (and should) use SQLite3 and an in-memory db for tests you do locally, but some code will just behave differently depending on the database you use. Setting up a continuous integration server that runs in a realistic setup is a must.

(Some people will tell you that you should mock all the stuff like the database, or your tests aren't "pure," but that's only ideology speaking. If you make errors in your mocking code (and believe me, you will), your testsuite will be worthless.)

This all said, the problems I described start being noticeable only when you are quite advanced with TDD... When you are just starting with TDD (or working on smaller projects) test refactoring won't be an issue.

Answer 5

For me there's some deep psychological problem with tests whenever I try to apply them extensively, as in TDD: if they're there, I code sloppily because I trust that the tests will catch any problems. But if there are no tests to provide a safety net, I code carefully, and the result is invariably better than with tests.

Maybe it's just me. But I've also read somewhere that cars with all kind of safety bells & whistles tend to crash more (because the drivers know that the safety features are there), so maybe this is something to acknowledge; TDD can be incompatible with some individuals.

Answer 6

One situation in which test-first really gets in my way is when I want to quickly try out some idea and see if it can work before I write a proper implementation.

My approach is normally:

1. Implement something that runs (proof of concept).
2. If it works, consolidate by adding tests, improving design, refactoring.

Sometimes I do not get to step 2.

In this case, using TDD has turned out to have more disadvantages than advantages for me:

• Writing tests during the implementation of the proof of concept just slows me down and interrupts my flow of thoughts: I want to understand an idea and I do not want to waste time testing details of my first rough implementation.
• It may take longer to find out if my idea is worth something or not.
• If it turns out that the idea was useless, I have to throw away both my code and my nicely written unit tests.

So, when I have to explore some new ideas, I do not use TDD and only introduce unit tests when I get a feeling the new code is getting somewhere.

Answer 7

Disadvantages or Costs of TDD

Note: There is a range of different kinds of TDD. Regardless of unit, BDD, ATDD, or other variants many of the difficulties remain

Side effects

Whether it's mocking, fixtures or functional tests, dependencies on external states or systems are often the source of most complexity in tests, confusion in how to test, and biggest risk in getting it wrong. A few issues I've seen:

• Mocking: forget to assert the order of calls
• Mocking: mock doesn't match real call or response
• Fixture: test relies on unrealistic data, masking other issues
• Fixture: test an impossible state in production
• Functional: false build breaks because of dependent system being temporarily unavailable
• Functional: speed of test is very slow

You will have to change your approach to coding, for some it will be a drastic change.

Different people code in wildly different ways. In TDD you need to start with a test that asserts a specific behavior, and then implement so the test passes. I've seen and was a programmer whose programming was not conducive to TDD. It took me about 2 months when I initially started to get used to changing my development approach.

It takes time to understand what you care about testing and what you don't care about testing.

Every team should make an explicit decision on where they want to draw the line in testing. What things they value that they want tested, and what they don't. It is often a painful process learning how to write good tests, and what you actually care about testing. Meanwhile the code will continue to be in a state of flux until there is consistency in both style and approach.

Unit Test specific: large refactors

A large or fundamental refactor of a significant codebase with tens of thousands of unit tests will generate a huge cost in order to update all the tests. This will often manifest in pushback against doing a refactor even if it is the correct thing to do simply because the cost associated with doing it.

Answer 8

My analogy is barriers on a Scalextric track. If you put them on, you get far less cautious.

People also get a little bit space cadetish about their tests - because they run fine, they believe the code is fully tested whereas it is only the beginning of the testing process.

To my mind TDD is a stepping stone to BDD. A raft of tests that run doesn't really help support developers without knowing what the tests do. With BDD, the test output is in English which documents the testing and thereby builds understanding of the system.

Answer 9

The benefits of TDD is that it forces you to guard your code against people who don't understand it. Yes, this often includes yourself. But, what happens when the code isn't worth guarding? There is a lot of code that shouldn't even be there in the first place! So the problem with TDD is when it comes to developers who write bad code. TDD probably wont help them write good code, it is much more likely that they will write horrible tests as well. Thus in their case TDD will only add to the mess; badly written and/or redundant tests aren't any more fun than other forms of bad code.