At a previous job (circa 2010), my manager mentioned that he's doing some research into quality metrics. The bottom line is to answer the question how many bugs should I expect? The goal is to try and predict how many bugs are incoming, and more importantly, how many bugs we probably haven't yet found.

There are some problems around this metric. Historically, we tend to look at "lines of code," which is a horrible way of measuring functionality. Functional points seem better, but I don't have an easy way to describe and measure how many function points a piece of software has.

The other issue is that quality will obviously vary (widely) depending on if you have really horrible QA/developers, or really awesome, or somewhere in the middle.

Another issue is, obviously, what you define as a bug. Some teams look at any feature change as a bug; others try to divide (eg. new functionality vs. broken but "works as expected" functionality).

Is there any research, or body of work, on helping software teams identify and predict how many bugs they should be finding?

  • 1
    I believe it was Code Complete that made a reference to a study (by IBM?) that found a fairly consistent 10:1 LoC to error ratio. Now how many of those are found by the IDE/programmer and never become bugs... who knows. – Telastyn Dec 6 '13 at 16:53
  • 1
    The 'how many bugs should I expect' is something that can be estimated - stevemcconnell.com/ieeesoftware/bp09.htm – user40980 Dec 6 '13 at 17:18
  • Nasa is one of the authorities on this. Research their work. The answer is related to x per y of code @ $ per line. Clearly at $10 you expect more bugs than at $1000, Nasa can tell you how many more. – mattnz Dec 6 '13 at 20:12
  • Color me crazy, but I find the whole idea of estimating the number of code defects a bit misguided. Sure, you can get some anecdotal idea of the quality of your code by counting the number of defects, but why not review the code to reduce the number of defects you get in the first place? Counting defects also doesn't speak at all to the severity of each defect; a single catastrophic defect trumps any effort to count them. – Robert Harvey Dec 9 '13 at 17:33

Yes: http://www.net-security.org/secworld.php?id=14871

"Over the past seven years, the Coverity Scan service has analyzed nearly 850 million lines of code from more than 300 open source projects including Linux, PHP and Apache. ... The analysis found an average defect density of .69"

And that's just bugs findable by Coverity's formal methods.

https://stackoverflow.com/questions/625344/how-many-bugs-is-too-many has more info in answers, suggesting anything up to 10 bugs per KLOC.

| improve this answer | |

how many bugs should I expect? [...] how many bugs we probably haven't yet found.

To answer that, define the words "bugs", "expect" and "found" :)

For one of my recent projects, we had initially added positive unit tests for our APIs. We found a set of bugs with a measurable "density" (i.e. measurable xx/1KLOC).

Then, we started adding negative test cases. We discovered another wave of bugs, that would have increased our density to an yy/1KLOC (with yy > xx).

Then we added testing with simulated low memory conditions (i.e. we tweaked our memory allocation APIs to fail on first allocation and called an API, then to fail on the second allocation and called the API and so on, until the API had enough successful allocation to succeed). We found a third wave of bugs, with most of them being "library left in invalid state when memory allocation fails".

Then we moved to fuzz testing, and full scenario testing (a particular sequence of API calls for achieving a given result) and got a new measurement of bugs.

TLDR: How many bugs you find depends on what you consider a "bug" (for us it wa only a bug if found after the end of the iteration), what you are testing for, your testing methodology and what resources you can put into testing (people, time, hardware, etc). If KLOC as a measurement is in there somewhere, it's a secondary measurement (i.e. after "what is a bug", "what you are testing for" and so on).

| improve this answer | |

Bug Estimation is an academic field of software engineering, and has quite a bit of research devoted to it. Most of the work I know of though, focuses on predicting future bugs rather than pure quality metrics.

Make sure everyone's on the same page as to what qualifies as a bug. This will largely depend on what you are trying to measure. For example, if you want to find the number of bugs found after a new feature is deployed, you would search for issues created for features a few days after the deployment.

Unfortunately, your estimation is only as good as the data you collect. If the dev team files multiple issues for the same bug, or if If you put low levels of effort into finding bugs, your estimate will reflect that. One possible mitigation techniques is to test your estimation against an estimate similar project but with a different dev/QA team. This, unfortunatley has its own pitfalls as well.

The most important aspect is to treat an estimate for what it is: an estimate. Know what your estimate means, and how it was made. Know when your estimate is relevant to the problem, and when it is not. Throw it out if it doesn't match your data. Never use one single metric as a standin for the quality of your code.

Here are some papers that you may find useful. Number 2. is a direct analysis of bugs in eclipse, which can provide an example of how to do your own estimate:

  1. https://dl.dropboxusercontent.com/u/18500843/01702038.pdf
  2. https://dl.dropboxusercontent.com/u/18500843/04228670.pdf
  3. https://dl.dropboxusercontent.com/u/18500843/chp%253A10.1007%252F978-1-4020-8741-7_20.pdf
| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.