How to Quantify the Value of Unit Testing [duplicate]

Question

Our organization is considering integrating unit testing into our software development workflow. I've heard lots of anecdotal stories about how it encourages better, easy to maintain, and well-planned code. As a programmer, I understand the benefits.

However, in order to build consensus around using unit testing in projects that might affect customers, I would like to demonstrate its value quantitatively, in terms of saving hours of developer time.

Does anyone in the community have any historical data or experience they would be willing to share as quantitative evidence in my case for unit testing?

Answer 1

Unit testing is cheap.

Each time you run a unit test, you didn't have to pay a tester to perform that test manually, or perform the test yourself, by hand. A computer can perform the same tests over and over tirelessy, with perfect fidelity (executing the test in exactly the same way every time).

Each time an automated test catches a bug you missed during refactoring, you save money by preventing QA people from having to report it, and untold money and reputation capital by preventing that bug from going out the door to a customer.

Automated testing, especially unit testing, saves money because it catches bugs early. The later a bug is found, the more expensive it is (by orders of magnitude):

Answer 2

Here is a chart from my experiences with and without unit testing. How many of the problems on the right-hand column apply to your project?

                  With Unit Tests               Without Unit Tests
----------------- ----------------------------- -----------------------------
Development       Code/Test/Refactor            Cut and Paste/Hack 
Process           or Test/Code/Refactor         to minimize code change 
                                                and avoid complete retest

Module Coupling   Limited by continuous         Tangled due to need to
                  refactoring                   add features with minimal
                                                code change

Program Design    Evolving through Refactoring  Frozen due to risk of change

Velocity          Increasing as higher-level    Decreasing as code volume and
over Time         functions are factored out    coupling increase 

Bugs detected     Mostly by developer           Mostly by QA or Customer

Cost of bugs      Lower                         Higher

You should be able to point to specific instances where the code volume was increased unnecessarily, or the module coupling was increased to minimize the need for retest. However, if they don't believe they have a problem, and are not interested in improvement, there may be little you can do. Even when continued maintenance becomes uneconomical, they may believe that is simply the normal end of all software projects.

I have been successful in improving testing only when management knew they should be testing, but weren't sure how to do it economically. When an organization is uninterested in process improvement, great patience is needed to effect change. How patient are you?

Answer 3

I once did a statistic over a number of versions and various modules of a large system, measuring basically the number of unit test LOC per functional code LOC, plotted against the number of bugs found during manual testing per functional LOC. The result looked something like this:

More Bugs
^
|
| +
|+    +  +     +
|   +    +          +   +       
|  +        +    +     +    + + 
|+     + +         +      +  +  +   +
|    +        +        +    +    + +
---------------------------------------> More Unit tests

A linear interpolation said that there were nearly 50% less bugs in the code with the most unit tests, compared to that with none. But as you see, there is a large variance, most likely related to complexity of the modules, experience and competence of the developers and maturity of the code.

Answer 4

Unit testing is invaluable.

I have many experiences with this.

When we first didn't adopt Unit testing, we spent the majority of the time solving bugs and dealing with mad customers because things didn't work as expected. Debugging code when you have no idea of where the bug may be is a hassle, especially because you first have to reproduce it in order to solve it.

When we decided it was time to start working with UT, it all changed. We got no more calls from mad customers at all. It may sound excessive but it went exactly like this.

With UT you find the bugs before your customers or QA. This is the big deal.