131

I'm a solo developer with a pretty time-constrained work environment where development time ranges usually from 1-4 weeks per project, depending on either requirements, urgency, or both. At any given time I handle around 3-4 projects, some having timelines that overlap with each other.

Expectedly, code quality suffers. I also do not have formal testing; it usually goes down to walking through the system until it somewhat breaks. As a result, a considerable amount of bugs escape to production, which I have to fix and in turn sets back my other projects.

This is where unit testing comes in. When done right, it should keep bugs, let alone those that escape to production, to a minimum. On the other hand, writing tests can take a considerable amount of time, which doesn't sound good with time-constrained projects such as mine.

Question is, how much of a time difference would writing unit-tested code over untested code, and how does that time difference scale as project scope widens?

  • Comments are not for extended discussion; this conversation has been moved to chat. – maple_shaft Jun 16 '16 at 12:50
  • 8
    You are solving the wrong problem. You are too busy and seem to have no project management support. Are you estimating project effort? Are you reserving 20% of your time for bug fixes, meetings, and other non-coding tasks? How much overtime are you working? – Tony Ennis Jun 16 '16 at 22:06
  • 20
    Do you realize that you're essentially saying "I have time to do it twice, but not time to do it once the right way."? – RubberDuck Jun 16 '16 at 23:07
  • 5
    @RubberDuck there is actually a point in the curve of project complexity measured as Time to Write vs Time to Test, where "Wring it twice", takes less time than "Write it and it's tests". I think it may be somewhere in the region of a bash oneliner. – Lyndon White Jun 17 '16 at 13:48
  • One time the developers got presents and thanks when a project was cancelled. I pointed out that we could have been even more productive had we known that the product wouldn't ship. So, this is a case where developing without testing would be advantageous. – JDługosz Jun 19 '16 at 5:03

12 Answers 12

148

The later you test, the more it costs to write tests.

The longer a bug lives, the more expensive it is to fix.

The law of diminishing returns ensures you can test yourself into oblivion trying to ensure there are no bugs.

Buddha taught the wisdom of the middle path. Tests are good. There is such a thing as too much of a good thing. The key is being able to tell when you are out of balance.

Every line of code you write without tests will have significantly greater costs to adding tests later than if you had written the tests before writing the code.

Every line of code without tests will be significantly more difficult to debug or rewrite.

Every test you write will take time.

Every bug will take time to fix.

The faithful will tell you not to write a single line of code without first writing a failing test. The test ensures you're getting the behavior you expect. It allows you to change the code quickly without worrying about affecting the rest of the system since the test proves the behavior is the same.

You must weigh all that against the fact that tests don't add features. Production code adds features. And features are what pay the bills.

Pragmatically speaking, I add all the tests I can get away with. I ignore comments in favor of watching tests. I don't even trust code to do what I think it does. I trust tests. But I've been known to throw the occasional hail mary and get lucky.

However, many successful coders don't do TDD. That doesn't mean they don't test. They just don't obsessively insist that every line of code have an automated test against it. Even Uncle Bob admits he doesn't test his UI. He also insists you move all logic out of the UI.

As a football metaphor (that's American football) TDD is a good ground game. Manual only testing where you write a pile of code and hope it works is a passing game. You can be good at either. Your career isn't going to make the playoffs unless you can do both. It won't make the superbowl until you learn when to pick each one. But if you need a nudge in a particular direction: the officials calls go against me more often when I'm passing.

If you want to give TDD a try I highly recommend you practice before trying to do it at work. TDD done half way, half hearted, and half assed is a big reason some don't respect it. It's like pouring one glass of water into another. If you don't commit and do it quickly and completely you end up dribbling water all over the table.

  • 67
    There is such a thing as too much of a good thing Neither you or Buddha have tested my grandmother's cookies :-) – Pierre Arlaud Jun 15 '16 at 13:05
  • 3
    @NickAlexeev I love that chart there. One thing it doesn't point out is that unit tests (which are typically automated) are really good at finding bugs when the code is modified. I'd love to see that split out into "bugs found before release" and "bugs found after release". Unit tests are the best line of defense against regression. – corsiKa Jun 15 '16 at 15:44
  • 3
    I think this is a very balanced answer: testing everything, even trivial things, can be a waste of time. Having good tests for the complex parts that can easily break can really help. I have just finished porting a small but non-trivial project from Java to C++. I first ported the tests and this guided me setting up the whole C++ implementation. Once all the tests were green, only a few easier classes needed to be ported and it went pretty smoothly. On the other hand, I do not have tests for all the code: this would have prolonged the implementation by at least 3, 4 days with little gain. – Giorgio Jun 15 '16 at 17:03
  • 5
    A slight disagreement with this : 'You must weigh all that against the fact that tests don't add features. Code adds features. And features are what pays the bills.' I would strongly suggest that it isn't features which pay the bills - it is working features. (or do people get paid for non-working deliverables ?). The rest of the answer I completely agree with. – Tony Suffolk 66 Jun 16 '16 at 20:06
  • 5
    @TonySuffolk66 you're correct, it's working features that pay the bills (barring flimflam salesmanship) However, people were creating working features long before TDD was a thing. They will long after it's gone. Remember, TDD is a disciplined way of testing. It is not the only disciplined way of testing. – candied_orange Jun 17 '16 at 3:38
111

I agree with the rest of the answers but to answer the what is the time difference question directly.

Roy Osherove in his book The Art of Unit Testing, Second Edition page 200 did a case study of implementing similarly sized projects with similar teams (skill wise) for two different clients where one team did testing while the other one did not.

His results were like so:

Team progress and output measured with and without tests

So in the end of a project you get both less time and fewer bugs. This of course depends on how big a project is.

  • 32
    The sample size is way too small to consider this to be scientific but I think it's representative of what lots of people experience. I find that when I do TDD, most of the extra time is spent fixing the bugs that cause my unit tests to fail, not writing the tests themselves. That's not really adding extra time, just shifting when you find and fix those issues. Any real extra time is in fixing problems that you wouldn't have found, at least not in the first go-round. – JimmyJames Jun 15 '16 at 15:38
  • 7
    @JimmyJames It's a case study, which is used extensively in business, and a lot in science when it's not (yet) possible to run a large scale reproducible experiment. There's psychology journals full of them. "Unscientific" is not the right word. – djechlin Jun 16 '16 at 2:12
  • 25
    Why do I think if the outcome of that case study had been showing the opposite, it would not have made it into the book ;-) ? – Doc Brown Jun 16 '16 at 10:40
  • 11
    @DocBrown I wonder how many case studies were made and discarded before they found one with the right answers :-) – gbjbaanb Jun 16 '16 at 12:13
  • 6
    @JimmyJames that almost certainly qualifies as science. Furthermore another scientist may read this study "n=1" case study, decide it's worth studying more, then run a large scale statistical study, or even a controlled experiment longitudinally, and confirm or deny it. That's exactly how science works. That's how it's supposed to work. you can read more on how science works here en.wikipedia.org/wiki/Scientific_method – djechlin Jun 16 '16 at 20:42
30

There is only one study I know of which studied this in a "real-world setting": Realizing quality improvement through test driven development: results and experiences of four industrial teams. It is expensive to do this in a sensible way, since it basically means you need to develop the same software twice (or ideally even more often) with similar teams, and then throw all but one away.

The results of the study were an increase in development time between 15%–35% (which is nowhere near the 2x figure that often gets quoted by TDD critics) and a decrease in pre-release defect density from 40%–90%(!). Note that all teams had no prior experience with TDD, so one could assume that the increase in time can at least partially attributed to learning, and thus would go down even further over time, but this was not assessed by the study.

Note that this study is about TDD, and your question is about unit testing, which are very different things, but it is the closest I could find.

  • 1
    It would be more interesting to impose additional constraints: no mutable state, following SOLID, static typing, no relying on null, functional over imperative, code contracts, static analysis, automated refactoring, no IoC containers (but DI) etc. I bet that value of unit tests would decrease (but not disappear). – Den Jun 21 '16 at 13:12
24

Done well, developing with unit tests can be faster even without considering the benefits of extras bugs being caught.

The fact is, I'm not a good enough coder to simply have my code work as soon as it compiles. When I write/modify code, I have to run the code to make sure it does what I thought it does. At one project, this tended to end up looking like:

  1. Modify code
  2. Compile application
  3. Run application
  4. Log into application
  5. Open a window
  6. Select an item from that window to open another window
  7. Set some controls in that window and click a button

And of course, after all that, it usually took a few round trips to actually get it right.

Now, what if I'm using unit tests? Then the process looks more like:

  1. Write a test
  2. Run tests, make sure it fails in the expected way
  3. Write code
  4. Run tests again, see that it passes

This is easier and faster then manually testing the application. I still have to manually run the application (so I don't look silly when I turn in work that doesn't actually work at all), but for the most part I've already worked out the kinks, and I'm just verifying at that point. I actually typically make this loop even tighter by using a program that automatically reruns my tests when I save.

However, this depends on working in a test-friendly code base. Many projects, even those with many tests, make writing tests difficult. But if you work at it, you can have a code base that's easier to test via automated tests than with manual testing. As a bonus, you can keep the automated tests around, and keep running them to prevent regressions.

  • 1
    And using something like nCrunch can cut steps 2 and 4, making the feedback loop even tighter. – Euphoric Jun 15 '16 at 7:22
  • "I still have to manually run the application" is an important observation, IMHO. No silver bullets. – Den Jun 21 '16 at 13:18
20

Despite there being a lot of answers already, they are somewhat repetitive and I would like to take a different tack. Unit tests are valuable, if and only if, they increase business value. Testing for testing's sake (trivial or tautological tests), or to hit some arbitrary metric (like code coverage), is cargo-cult programming.

Tests are costly, not only in the time it takes to write them, but also maintenance. They have to be kept in sync with the code they test or they're worthless. Not to mention the time cost of running them on every change. That's not a deal-breaker (or an excuse for not doing the truly necessary ones), but needs to be factored in to cost-benefit analysis.

So the question to ask when deciding whether or not (or of what kinds) to test a function/method, ask yourself 'what end-user value am I creating/safeguarding with this test?'. If you can't answer that question, off the top of your head, then that test is likely not worth the cost of writing/maintaining. (or you don't understand the problem domain, which is a waaaay bigger problem than a lack of tests).

http://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf

  • 1
    I'm not super familiar with BDD but will guess that it operates at a slightly coarser granularity than the method/function level and probably has a less tenuous connection to user-value. – Jared Smith Jun 15 '16 at 15:32
  • 1
    Related – Robbie Dee Jun 15 '16 at 16:09
  • 9
    "Testing for testing's sake (trivial or tautological tests), or to hit some arbitrary metric (like code coverage), is cargo-cult programming." So true and so well-said. Test in such a way that you feel like a cool badass - think about yourself as a ... spy, elite athlete ... DON'T test like a "government department". You know? – Fattie Jun 16 '16 at 12:06
  • 2
    @SteveJessop disagree, code coverage (in the sense of being a metric) is inherently arbitrary: the number of paths through a non-trivial program at the machine instruction level (i.e. the one that counts) will be greater than the number of atoms on Earth or possibly even the visible universe. This is not testable. So any claim of 'code coverage' will be to some heuristically chosen arbitrary threshold. Programmers are good at gaming such metrics at the expense of the things that actually matter. – Jared Smith Jun 17 '16 at 12:35
  • 2
    I would also say that tests provide business value approximately (albeit not precisely) whenever they fail, and the resolution is to improve the code under test. So if you're using TDD that pretty much automatically happens at least one per test. Tautological tests by definition cannot fail and so are useless. For "trivial" tests -- having worked with the Java TCK early in my career, I'm no longer surprised by how trivial of a test it is possible to fail when re-implementing an API from scratch ;-) But business value is almost always predicted heuristically, so is "arbitrary" too. – Steve Jessop Jun 20 '16 at 8:32
9

It depends on the person, as well as the complexity and shape of the code you're working with.

For me, on most projects, writing unit tests means I get the work done about 25% faster. Yes, even including the time to write the tests.

Because the fact of the matter is that software isn't done when you write the code. It is done when you ship it to the customer and they're happy with it. Unit tests are by far the most efficient way I know of to catch most bugs, isolate most bugs for debugging, and to gain confidence that the code is good. You have to do those things anyways, so do them well.

  • 7
    I think it's worth noting though, it's an acquired skill. I see so many people hear the claim that TDD really isn't even a time-sink upfront that pays itself off in the long run, it's just faster, period. then they try it for a day and it's painful because they have 0 experience, have read 0 books, no practice, they just expect it to magically work. there's no secret to TDD that makes you a better developer, you still need to practice, still need to think, still need to make good educated decisions. – sara Jun 15 '16 at 5:48
  • 1
    @kai - +1. I spent weeks reading about TDD before I tried it. I read everything I could find. I read books. I read through all the well-known agile blogs for examples. I read xUnit Test Patterns cover-to-cover. For the first few weeks, it still took me twice as long. – Jules Jun 15 '16 at 8:33
  • 2
    I agree. TDD is hard. The mindset is difficult. Anyone who says "Just write the tests first" and claims that it's free doesn't know how to do it. It takes practice. – duffymo Jun 15 '16 at 14:03
  • @kai: for similar reasons a lot of people can't touch-type. They tried it once and after a whole hour still weren't typing any faster than before ;-) – Steve Jessop Jun 17 '16 at 1:05
  • @SteveJessop I guess that's a pretty neat comparison. Or being really unfit and going out for a 10 minute jog, getting exhausted and wondering why you can't run 10 miles in an hour. it really illustrates how you need to work before you get the benefits. – sara Jun 17 '16 at 8:36
4

Question is, how much of a time difference would writing unit-tested code over untested code, and how does that time difference scale as project scope widens?

The problem gets worse as the age of the project increases: because whenever you add new functionality and/or whenever you refactor existing implementation, you ought to retest what's previously be tested to ensure that it still works. So, for a long-lived (multi-year) project, you might need to not only test functionality but re-test it 100 times and more. For this reason you might benefit from having automated tests. However, IMO it's good enough (or even, better) if these are automated system tests, rather than automated unit tests.

A second problem is that bugs can be harder to find and fix if they're not caught early. For example if there's a bug in the system and I know it was working perfectly before you made your latest change, then I'll concentrate my attention on your latest change to see how it might have introduced the bug. But if I don't know that the system was working before you made your latest change (because the system wasn't properly tested before your latest change), then the bug could be anywhere.

The above applies especially to deep code, and less to shallow code e.g. adding new web pages where new pages are unlikely to affect existing pages.

As a result, a considerable amount of bugs escape to production, which I have to fix and in turn sets back my other projects.

In my experience that would be unacceptable, and so you're asking the wrong question. Instead of asking whether tests would make development faster you ought to ask what would make development more bug-free.

A better question might be:

  • Is unit-testing the right kind of testing, which you need to avoid the "considerable amount of bugs" you've been producing?
  • Are there other quality control/improvement mechanisms (apart from unit-testing) to recommend as well or instead?

Learning is a two-stage process: learn to do it well enough, then learn to do that more quickly.

3

Some aspects to consider, not mentioned in the other answers.

  • Extra Benefit/Extra Cost depend on experience with writing unittests
    • with my first unit-test project the extra costs trippled because i had to learn a lot and i made a lot of mistakes.
    • after 10 years of experience with tdd i need 25% more coding time to write the tests in advance.
  • with more tdd-moduls there is still need for manual-gui-test and integration-testing
  • tdd only works when done from the beginning.
    • applying tdd to an existing, grown project is expensive/difficuilt. But you can implement regression-tests instead.
  • automated tests (unittests and other kind of tests) require maintanace const to keep them working.
    • having created test through copy&paste can make testcode-maintanace expensive.
    • with growing experience testcode becomes more modular and more easy to maintain.
  • with growing experience you will get the feeling when it is worth to create automated tests and when not.
    • example there is no big benefit to unittest simple getters/setters/wrappers
    • i donot write automated tests via the gui
    • i take care that the businesslayer can be tested

Summary

When starting with tdd it is difficuilt to reach the "more benefit than cost" state as long as you are under "time-constrained work environment" especially if there are "clever managers" that tell you to "get rid of the expensive, useless testing stuff"

Note: with "unit testing" i mean "testing moduls in isolation".

Note: with "regression testing" i mean

  • write some code that produces some output-text.
  • write some "regression testing" code that verifies that the result of the generation ist still the same.
  • the regression test let you know whenever the result changes (which might be ok or an indicator for a new bug)
  • the idea of "regression testing" is similar to approvaltests
    • ... taking a snapshot of the results, and confirming that they have not changed.
  • Needs proofreading (the literary equivilent of testing?) – JDługosz Jun 19 '16 at 5:10
3

Programmers, like people dealing with most tasks, underestimate how long it actually takes to complete it. With that in mind, spending 10 minutes to write a test can be looked at as time one could have spent writing tons of code when in reality, you would have spent that time coming up with the same function name and parameters you did during the test. This is a TDD scenario.

Not writing tests, is a lot like having a credit card; we tend to spend more or write more code. More code has more bugs.

Instead of deciding to have total code coverage or none at all, I suggest focusing on the critical and complicated part of your application and have tests there. In a banking app, that might be the interest calculation. An engine diagnostic tool may have complex calibration protocols. If you've been working on a project, you probably know what it is and where the bugs are.

Start slowly. Build some fluency before you judge. You can always stop.

3

There have been a long history of Programmers board promoting TDD and other test methodologies, I won't recall their arguments and agree with them, but here is additional things to consider that should nuance a bit:

  • Testing isn't equally convenient and efficient depending of context. I develop web software, tell me if you have a program to test the whole UI... right now I'm programming excel macros, should I really develop a test module in VBA ?
  • Writing and maintaing the test software is real work that counts in the short run (it pays off in a longer run). Writing relevant tests is also an expertise to get
  • Working as a team and working alone, haven't the same tests requirements because in team you need to validate, understand and communicate code you did not write.

I'd say testing is good, but make sure you test early and test where the gain is.

  • 1
    "Should I really develop a test module for VBA?" Damn right you should. rubberduckvba.com/Features#unitTesting – RubberDuck Jun 16 '16 at 23:23
  • There is a few reasons I won't use this because it doen't fit my needs (I'm on few days tasks at most, locked environment, successor won't bother 3rd parties). Good comment though, language is not an excuse in itself :) – Arthur Havlicek Jun 17 '16 at 6:25
  • All fair points @ArthurHavlicek. – RubberDuck Jun 17 '16 at 10:02
  • 2
    Writing tests is still trivial in VBA. Having all the fancy features that some unittest frameworks have? That's harder, but running a program called mainTest() that calls all your test modules isn't really that hard. – enderland Jun 17 '16 at 17:20
1

An oft overlooked benefit of TDD is that the tests act as a safeguard to make sure you aren't introducing new bugs when you make a change.

The TDD approach is undoubtedly more time consuming initially but the takeaway point is you'll write less code which means less things to go wrong. All those bells and whistles you often include as a matter of course won't make it into the code base.

There's a scene in the film Swordfish where if memory serves, a hacker is having to work with a gun to his head and being erm... otherwise distracted. The point is it is a lot easier to work when your headspace is in the code and you have time on your side rather than months down the line with a customer screaming at you and other priorities getting squeezed.

Developers understand that fixing bugs later is more costly, but flip that on it's head. If you could be paid $500 a day to code how you code now or $1000 if you wrote in a TDD way, you'd bite the hand off the person making you the 2nd offer. The sooner you stop seeing testing as a chore and see it as a money saver, the better off you'll be.

0

I can relate to your expierience - our code base had almost no tests and was mostly untestable. It took literally ages to develop something and fixing productions bugs took precious time from new features.

For a partial rewrite, I vowed to write tests for all core functionality. At the beginning, it took considerably longer and my productivity suffered noticeably, but afterwards my productivity was better than ever before.

Part of that improvement was that I had fewer production bugs which in turn led to fewer interruptions -> I had a better focus at any given time.

Furthermore, the abillity to test AND debug code in isolation really pays of - a suite of tests is vastly superior to a system which cannot be debugged except with manual setup, e. g. launching your app and navigate to the screen and do something...perhaps a few dozen times

But notice that there is a drop in productivity at the beginning, so start to learn testing on some project where the time pressure is not already insane. Also, try to start it on a greenfield project, unit testing legacy code is very hard, and it helps when you know how a good test suite looks like.

protected by maple_shaft Jun 16 '16 at 12:48

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.