15

Our organization is considering integrating unit testing into our software development workflow. I've heard lots of anecdotal stories about how it encourages better, easy to maintain, and well-planned code. As a programmer, I understand the benefits.

However, in order to build consensus around using unit testing in projects that might affect customers, I would like to demonstrate its value quantitatively, in terms of saving hours of developer time.

Does anyone in the community have any historical data or experience they would be willing to share as quantitative evidence in my case for unit testing?

5
  • 1
    I'm not 100% sure you will find what you are looking for. The act of performing unit testing doesn't necessarily save any developer time, as it takes time to write the tests and then fix defects found. However, what you will find is data relating to the cost savings of finding and fixing defects early and improved product quality (and customer satisfaction). Unit testing is just one step a series of processes designed to improve product quality. I think you should refocus your efforts - sell product quality, with unit testing as a single tool among many to begin to move toward that goal.
    – Thomas Owens
    Sep 7, 2011 at 23:40
  • 1
    "I would like to demonstrate its value quantitatively, in terms of saving hours of developer time.". And then, everyone will simply dispute your numbers. This is always doomed from the start. Are you open to suggestions for techniques that work? Or are you stuck on arguing about the numbers instead of doing something?
    – S.Lott
    Sep 8, 2011 at 1:04
  • 1
    All of these are duplicates: programmers.stackexchange.com/search?q=justify+unit+testing. What's wrong with all of the other questions just like yours?
    – S.Lott
    Sep 8, 2011 at 1:06
  • Historical data? What about 3 days of searching to find a hard to locate bug that was introduced by a simple "no-risk change"? A bug that would have been caught by a very simple unit test? A unit test that would have taken at most 5 minutes to code? Sep 8, 2011 at 6:51
  • How do you quantify leaving out features because you're afraid to break the application?
    – JeffO
    Sep 8, 2011 at 12:10

4 Answers 4

11

Unit testing is cheap.

Each time you run a unit test, you didn't have to pay a tester to perform that test manually, or perform the test yourself, by hand. A computer can perform the same tests over and over tirelessy, with perfect fidelity (executing the test in exactly the same way every time).

Each time an automated test catches a bug you missed during refactoring, you save money by preventing QA people from having to report it, and untold money and reputation capital by preventing that bug from going out the door to a customer.

Automated testing, especially unit testing, saves money because it catches bugs early. The later a bug is found, the more expensive it is (by orders of magnitude):

enter image description here

9
  • Thanks, Robert. I'm certainly in agreement, but it's going to be hard to sell this argument without some sort of quantitative evidence to back it up.
    – Adam
    Sep 7, 2011 at 23:15
  • Where's this chart from?
    – Aaronaught
    Sep 7, 2011 at 23:21
  • 1
    @Aaronaught: jonkruger.com/blog/2008/11/20/… Sep 7, 2011 at 23:23
  • 2
    Huh... doesn't seem to be based on any actual data. Not that I disagree with the general idea but it'd be nice to see something a bit more material.
    – Aaronaught
    Sep 7, 2011 at 23:26
  • @Aaronaught: It is from a Google image search for "Cost of Unit Testing." There are a bazillion such graphs; they all look the same. The original was probably from "The Mythical Man Month," but I don't have a copy in front of me right now to verify that. Sep 7, 2011 at 23:29
10

Here is a chart from my experiences with and without unit testing. How many of the problems on the right-hand column apply to your project?

                  With Unit Tests               Without Unit Tests
----------------- ----------------------------- -----------------------------
Development       Code/Test/Refactor            Cut and Paste/Hack 
Process           or Test/Code/Refactor         to minimize code change 
                                                and avoid complete retest

Module Coupling   Limited by continuous         Tangled due to need to
                  refactoring                   add features with minimal
                                                code change

Program Design    Evolving through Refactoring  Frozen due to risk of change

Velocity          Increasing as higher-level    Decreasing as code volume and
over Time         functions are factored out    coupling increase 

Bugs detected     Mostly by developer           Mostly by QA or Customer

Cost of bugs      Lower                         Higher

You should be able to point to specific instances where the code volume was increased unnecessarily, or the module coupling was increased to minimize the need for retest. However, if they don't believe they have a problem, and are not interested in improvement, there may be little you can do. Even when continued maintenance becomes uneconomical, they may believe that is simply the normal end of all software projects.

I have been successful in improving testing only when management knew they should be testing, but weren't sure how to do it economically. When an organization is uninterested in process improvement, great patience is needed to effect change. How patient are you?

5

I once did a statistic over a number of versions and various modules of a large system, measuring basically the number of unit test LOC per functional code LOC, plotted against the number of bugs found during manual testing per functional LOC. The result looked something like this:

More Bugs
^
|
| +
|+    +  +     +
|   +    +          +   +       
|  +        +    +     +    + + 
|+     + +         +      +  +  +   +
|    +        +        +    +    + +
---------------------------------------> More Unit tests

A linear interpolation said that there were nearly 50% less bugs in the code with the most unit tests, compared to that with none. But as you see, there is a large variance, most likely related to complexity of the modules, experience and competence of the developers and maturity of the code.

3

Unit testing is invaluable.

I have many experiences with this.

When we first didn't adopt Unit testing, we spent the majority of the time solving bugs and dealing with mad customers because things didn't work as expected. Debugging code when you have no idea of where the bug may be is a hassle, especially because you first have to reproduce it in order to solve it.

When we decided it was time to start working with UT, it all changed. We got no more calls from mad customers at all. It may sound excessive but it went exactly like this.

With UT you find the bugs before your customers or QA. This is the big deal.

Not the answer you're looking for? Browse other questions tagged or ask your own question.