9

I wanted to get an idea of categorizing bugs based on how easy is it to solve and how much benefit it will give me. for e.g., if there is a bug which will take say an hour (double file close etc.) to solve vs another which takes a day (segmentation fault). But if solving the first bug is not very important, then I'll probably work on the second one.

Are there any research papers which categorize bugs based on cost-benefit or similar metric?


Let's say it is possible to categorize bugs based on bug characteristics e.g. security vulnerability, memory error, logic error etc. On the other dimension there could be parameters like difficulty (easy, medium, hard). Are there other dimensions I should be looking for. To simplify things, I can assume two things:

  1. Every programmer in the team is equally capable of solving any bug
  2. There is no deadline
  • 4
    I'm not convinced one can accurately estimate the time it will take to fix a bug. – Mike Aug 2 '13 at 22:51
  • I agree with you. I'm not asking for a universal approach but an approximate one. There are certain bugs for which we can easily estimate the time (which might be wrong sometimes but that's okay). – A. K. Aug 2 '13 at 23:33
  • 1
    Don't categorize on time it will/may take. Categorize on importance: "show stoppers" (crashes, not starting, unusable ui), "correctness", "customer satisfaction", etc; and on urgency. Order them according to important and urgent; important but not urgent; not important and urgent; not important not urgent. (If you only look at urgent then the important not urgent things tend to get pushed out by urgent not important stuff). – Marjan Venema Aug 3 '13 at 9:56
  • 2
    This question was also posted here: pm.stackexchange.com/questions/9664/…. I don't think harm was done as this arguably could cross over into Project Management. I'm linking this so others who find this question can see all the answers. Hope this helps! :) – jmort253 Aug 3 '13 at 23:18
  • "Are there any research papers..." -- Questions asking us to recommend a tool, library or favorite off-site resource are off-topic for Programmers as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it. – gnat Aug 6 '13 at 23:34
11

The typical bug tracking system has two, maybe three fields identifying bug cost benefit ratio:

  1. Priority (assigned by business owner)
  2. Severity (classification of bug - critical to low)
  3. Estimated hours (a guess at how long it will take to do)

As you note, this does identify which bug is the important one to work on.

The system put forth in PEF/REV: A Multi-dimensional bug tracking metric adds more information on the business and developer components to bug cost benefit.

All values are on a 1 .. N scale (same top value for each).

PEF is identified by the business:

  • Pain - How painful the bug is
  • Effort - How much effort it takes to work around the bug
  • Frequency - How often does the bug occur

REV comes from the developer:

  • Risk - How risky is the fix to the system
  • Effort - How much effort is there to fix the bug
  • Verifiability - How easy is it to verify the bug fix

If, for example, you have a rarely happening crash that is easy to work around (turn on autosave), that could be a PEF of 7,1,1 (score 9). While fixing it may involve a change to a core module and have a REV of 9,3,2 (score 14).

On the other hand, you could have a annoyance that is always there (3,3,9 - score 15) that has a trivial fix (1,1,3 - score 5).

In this example, the annoyance looks to be a better cost/benefit thing to work on.

| improve this answer | |
  • I like this. Looks like it would be possible to apply this to "new features" as well. – Martin Wickman Aug 3 '13 at 16:45
  • This is very informative. Our team uses bugzilla, and I think it does not have similar features. – A. K. Aug 3 '13 at 17:05
  • 1
    @AdityaKumar Bugzilla is very customizable though this can be done with the addition of custom fields and then run reports against the PEF/REV values. – user40980 Aug 3 '13 at 17:55
  • @MartinWickman absoultely. The REV can be translated with nearly no changes. PEF would likely become a combination of Utility (how nice it is to have) and Frequency (How often it is used) and Value (How much the perceived value of the feature would be). (And thank you for getting me to think about that dimension) – user40980 Aug 3 '13 at 18:01
5

The problem you are describing is very common and the best answer might be "use your gut feeling".

I usually divide bugs in three categories: Crashing, Annoying, Cosmetic (they might be called 1, 2, 3 - that doesn't really matter) and then put down an overall estimate on the time needed to fix every bug (all bugs must have a rough updated estimate at all times).

When solving bugs Crashing > Annoying > Cosmetic and then I simply do a "shortest job first" to get as much initial throughput as possible.

I find it VERY difficult to calculate any kind of direct financial benefit from solving any bug - unless it is a very tightly scoped paid job.

One note you might need is that you really should live up to the fifth point on The Joel Test - this might be influenced by team size and distribution among various other "local" issues - but it is generally a sign of good practice.

| improve this answer | |
  • 1
    Agreed here, but what each of those classifications actually means is important to agree upon if you're working with others using them. "Crashing" seems pretty objective -- it either does or it doesn't. But then we get to the "Annoying"/"Cosmetic" part. How annoying? And to whom? Etc. Often there's another classification in between "Crashing" and "Annoying". Where "Crashing" may have no workaround, "Breaking" (if I may) may have a workaround. – tamouse Aug 3 '13 at 15:11
  • I aggree @tamouse - my example is translate from a (maybe poor) wording in my native tongue ;-) – Michael Banzon Aug 3 '13 at 18:42
3

Another consideration might be the type of test or testing organization that uncovers the bug, when trying to determine the impact and cost of the bug and fix. Unit or functional testing may point to something in the design that needs to be changed, and thus correcting early would be easier and less costly than later. System or integration testing may point to something that affects a wider variety of customers. And Standards testing, while often something non-critical to a large subset of customers, may lead to a loss of certification if not corrected and have a negative impact on the business.

That said, just because a particular test organization has a test fail shouldn't automatically make a bug "critical." For example, there might be a temptation to say something like "all system tests must pass before shipping, thus any system test that fails automatically results in a critical/high-severity bug." Hopefully no organization would make that statement (good test exit criteria is a different discussion), but the point is that the "severity" of bug should be judged on its impact to the customer, product, or company image rather than where or when it is uncovered.

One possible way to address that is to make a distinction between "severity" and "urgency." For example, as the release date nears there can be time pressure to determine if an apparently lower-severity bug might affect a large subset of customers, giving the bug a greater "urgency" and elevating work on that bug above (some) others at least until that determination can be made. A proper balance between the two, along with experience and good judgement, would help direct where time and effort is spent.

| improve this answer | |
2

In addition to what others have said about the classification of bugs, etc, also consider looking at Afferent Coupling (Ca) for determining the level of criticalness a particular bug might represent. If you have a bug in a module with a high Ca count, I might be looking at fixing that one first as it could benefit other components in the system. Ca helps you determine the level of responsibility, and responsible modules with bugs in them hurt the whole application (read more about Ca here: http://www.ibm.com/developerworks/java/library/j-cq04256/index.html).

With that in mind, we tend to categorize bugs and place them in priority based on their impact to the customer. Your customer will vary, but having them involved in the discussion will ultimately drive what bugs should be fixed over others. It's not real scientific, but as a whole team we tend to come to consensus on the priority and categorization of bugs, everyone has input on the "big ones" (all the stakeholders can give input), and we stack and rack from there.

| improve this answer | |
2

You can't determine the actual cost of all bugs. Some you can, for many it's very hard to quantify. For example, say you have a bug that results in all of the text on buttons are slightly misaligned. It makes your 1.0 product look a little sloppy. This doesn't cause your program to crash, or your users to lose data. Probably a pretty cheap bug, right?

Now, what if every customer of yours notices this problem, and every reviewer mentions it in reviews of your product. And what if this bug carries over from version 1.0 to 1.1 and 1.2. Your company may now have a reputation of being a bit sloppy in quality control. How expensive might this simple bug be in terms of potential lost sales for your company for future products? Or, how might this affect a review that your product gets -- your product may perfectly meet the needs of your customers, but only gets an 9 out of 10 because it looks a little sloppy.

So, you need to not only look at the cost of a specific bug in terms of what it costs to fix and it's direct impact on the user, but you have to consider it in the larger context of how it affects the perception of your product in the marketplace, and how that perception affects future sales.

This may seem far-fetched but it isn't. At the time that I write this you can go to another site on the web that has an article about how Apple has moved the "1" on their calendar icon over by a pixel or two. If you do a search you'll see several negative articles about this tiny little bug. Of course, this hasn't directly affected Apple's bottom line, but they promote themselves as the pinnacle of good design, so having a design bug affects that perception, even if only slightly.

| improve this answer | |
  • I like your idea that impact on customer/user may become a big driving force on which bugs to solve. – A. K. Aug 3 '13 at 18:11

Not the answer you're looking for? Browse other questions tagged or ask your own question.