Whats the real overhead of TDD once the entire team is used to it?

Question

What percentage of time is saved and costed doing TDD.

I assume this percentage of cost and reward changes during a projects life-cycle.

I'd imagine the initial phase has a lot more cost but little rewards attached. Further on (during re-factoring) you get the benefit of your tests.

I've heard anywhere from 30-50% of your time is writing unit tests. However that doesn't take into account the time saved from writing those tests. Whats the time saved as well as the time cost? In bug fixing and refactorablity?

Answer 1

I've heard anywhere from 30-50% of your time is writing unit tests. However that doesn't take into account the time saved

In my experience, it's more than 50%.

Once you've written the test, the solution has a tendency to come very easy. So I don't think it's odd to spend 70% - 75% of your time writing tests, but you're spending much less time writing the 'production code' (code-being-tested) and spending virtually no time in the debugger.

The sooner you find a bug, the cheaper it is to fix, and TDD helps with that tremendously. I've worked on projects where the last 2 months (of an 8 month project) were spent fixing bugs, and that phase would be almost entirely eliminated with TDD.

To me though, the real value is in maintenance. Inheriting a code base with tests makes you less scared to alter it. You feel like you didn't break anything when the tests still pass. Since you aren't scared to make changes you're willing to refactor if something isn't right. Which means the code can be made cleaner, the design can fit better, and, in theory, changes can be applied. Contrast that with voodoo code everyone's scared to touch.

Answer 2

Each time you run your unit tests, you save yourself the amount of time it would have taken to manually test your code.

The 30% to 50% of time you quote as being required to write your tests is also offset a great deal by the benefits of having a better (testable) software design.

---

Let's say it takes four times as long to write an automated test as it does to manually perform the test. That means that the fourth time you run your automated test, it pays for itself. Every time you run the automated test after that, it's free.

This holds true whether the test is an automated unit test, or an automated functional test. Not all functional tests can be automated, but many of them can. Plus, the automated test is more reliable than a person; it will run the test in exactly the same way, every time.

Having unit tests means that you can refactor the underlying implementation of a method (for performance or other reasons), and the unit tests will verify that the functionality of the method has not changed. This is especially true of TDD, where the unit test specifies the functionality of the method.

Answer 3

TDD is often measured towards code quality rather than time and cost spent. However, with better code quality, developers and any people working with them can work better (less time spent, less cost involved, happier, etc.). http://davidlongstreet.wordpress.com/2009/04/29/new-software-metric-wtfs-per-minute/

Writing tests is great for helping to automate verification of functional and non-functional requirements. One video that convinced me to adopt TDD (actually BDD, high level TDD): http://video.google.com/videoplay?docid=8135690990081075324#

• Writing functional tests can help spotting bugs/problems earlier during development phase. Assume you have large code base. With unit tests/specs, you only need to see "All tests passed"/"2 tests failed, see line xyz". You only need a team of developers to do both developing and testing. Without unit tests/specs, you have to manually compare printed statements with expected statements, and manually trace which methods/classes have bugs. You probably need two separate teams (developers and testers) to do this.

• Written tests help developers explain progress and problems faced.

• TDD helps fulfilling maintainability, adaptability, and flexibility of code. It encourages developers to write small testable chunks, and put them together into bigger testable chunks. The other way round (part of refactoring practice) also works, with condition that we have written solid tests. As a result, we can have nicely-written, modular code.

With TDD, we are glad to know when:

• a customer requests changes on requirements (satisfying requirements)
• better ways of writing code are discovered
• team mates have suggestions for code improvement
• we have to explain/pass our code to other people

TDD can be boring because development process takes small steps, and thus it becomes so predictable.

Answer 4

the important long-term measures are not only code quality and code confidence, but even moreso not burning out the team doing mindless testing

the short-term measures would be the ROI of automating the tests

for example: last week I made over 1000 code changes due to an internal-architecture shift, started the automated test suite, and went to sleep.

the tests took 28 minutes to run; they all passed. manually performing the same 40+ acceptance tests would take about 6 hours.

another example: in a prior iteration i had goofed up one of the test scenarios with a subtle bug that manual testing probably would not have found (the automated tests perform db integrity checks that manual testers almost never do). i had to run that test scenario about 50 times before i managed to figure it out and fix it. manually performing the test scenario operations would take about 50 minutes. So that's 41.6 man-hours of labor saved in one day

there is no way of calculating in advance the ROI of automated testing, because you cannot know exactly how many times you will need to run the tests.

but to me, the ROI of automated tests is nearly infinite

Answer 5

In our case, I'd estimate it's close to 40%. However, I don't think we went through a phase where it was any more than this. We have a code generator that spits out both a code skeleton that gets fleshed out by the developers and a test suite that likewise gets fleshed out. Most of our testing effort actually goes into tracking down (or creating) appropriate test data to ensure that we get complete coverage.

Answer 6

It can help a lot to restrict unit tests to complex algorithms, cases where they can be automatically generated, and regressions.

Component test often do a great job for rather trivial code, plus changing the implementation is a lot cheaper because the tests are only coupled to the interface.

Full coverage with fine grained unit tests has a huge overhead for changing or refactoring an implementation, which is exactly what they claim to make easy.

Answer 7

I have been in situations where it was required that even if you change a line in your code the unit test must highlight it. Yes unit tests are useful but ultimately it is a trade off on how much time you want to spend on it. Personally I am more biased toward detailed integration tests and having unit tests where the integration tests can not reach.