4

We have been working with a shopping cart for DotNetNuke, and have had endless problems with the developer's releases of their product. Every release fixes one thing but new bugs pop up elsewhere.

I know that bugs are inevitable and that we cannot squash all of them at the time, but can someone please tell me what percentage of bugs should be stamped out before a product can be accepted as a stable release?

Improve this question
3
  • 1
    All bugs that your organization considers bugs that cannot wait for the next build release.
    – Ramhound
    Sep 12, 2011 at 13:08
  • All bugs should be resolved and properly tested prior to release. To do anything else will make you a normal business... Wouldnt you prefer to stand out from your compitition as a bastion of high quality?
    – SoylentGray
    Sep 12, 2011 at 16:14
  • Almost all if you want Joel to give you 12 stars.
    – Job
    Sep 12, 2011 at 20:49

7 Answers 7

Reset to default
23

I don’t think it’s a matter of percentage.

Each bug has to be evaluated on its own to decide whether it’s a show-stopper for the particular project in question, based on the likely cost of the bug if not fixed before release, and the likely cost of delaying release until the bug can be fixed.

(For example, Stack Overflow spent quite a while with a notification icon that didn’t display properly in Chrome. The costs of that bug were so low that it could happily be left for several weeks, as the team had bigger issues to focus on.)

And remember that “bugs” is just shorthand for “known bugs”. You could fix 100% of your known bugs and still not be in shape for launch, because you haven’t tested well enough.

Improve this answer
1
  • 1
    "bugs" is also a euphamism for "defect". They don't crawl into software, they are placed there.
    – Bryan Oakley
    Nov 15, 2011 at 13:46
6

It is not possible to give a percentage number that should be fixed - if you fix 90% of bugs, one of the remaining bugs could be so severe as to cripple your application.

Instead, all bugs should be classified by severity. A few suggestions for classifications would be:

  • Show Stopper
  • Critical
  • High
  • Medium
  • Low

By very definition, all Show Stoppers MUST be fixed, otherwise the application will not function. Critical bugs should also be fixed before shipping.

Any remaining bugs, such as High/Medium/Low priority bugs should then be considered, ideally they should all be fixed, but if this cannot be achieved due to time or budget constraints, then each remaining bug should be evaluated and it should be determined if it is still OK to ship with the issue remaining.

Improve this answer
3
  • Thanks Gavin. As it stands, most of the bugs that we have reported for this DNN Cart have either been Show Stoppers or Critical bugs. Every time they release new fixes, not only are there new ones but it means that we have reskin the entire shop to make it look like the client's site design which costs us money.
    – SixfootJames
    Sep 12, 2011 at 9:40
  • 2
    @SixFootJames - that's unfortunate. bear in mind you only get one chance to make a first impression. If a user tries to use the store and hits a show stopper bug, they will leave and never trust your store again. While it may cost a lot to wait, it may work out cheaper to do so than launch a buggy store and possibly ruin you and your clients reputation forever.
    – Gavin Coates
    Sep 12, 2011 at 11:11
  • 3
    @SixFootJames, if you are using someone else's product and it has a show stopper bug in every release, perhaps you would be better serverd replacing it with a product that works.
    – HLGEM
    Sep 12, 2011 at 21:58
3

Theres not really percentage, A good way to work is to monitor the rate of bug insertion - All else being equal, when it starts to fall, the product is becoming stable. If you say something like "Ship it when x% are fixed", you will probably ship unstable code

Therefore the answer is not as easy as "At percent x we call it a release". It's better to say "Once the insertaion rate is y% of the (maximum/typical/average) insertion rate, the software becomes a release candidate" Release candiates are tested for acceptance, and released or not after anaylisis of known and unknown bugs. You may decide to fix more severe bugs in release candidates, but you must not add features or make improvements.

Using statistical methods it is possible to "determine" unknown bugs. By that I mean get a statistical value of how many and how severe the ones you have not found are. Most of us have a gut instinct for this number, but few (including me) can do the maths.

Of couse, it you are not counting and like me, no good at math, you can go on gut instinct, and just ship it when its ready.......

Improve this answer
3

If you know a method for measuring the percentage of remaining bugs in a software, you will be the richest man of all times.

If you target only known bugs, a stable release should contain none.

Improve this answer
4
  • 1
    I have to disagree. Fixing a bug has a chance (10-20% by industry averages) of introducing another defect. Sometimes it's better to ship with a known defect than to risk shipping with a new, unknown defect. Also incorrect is your assertion that measuring remaining bugs cannot be done. It can and is done, although only in a few high end software houses (Think Nasa, Avionics and Military) .
    – mattnz
    Sep 13, 2011 at 4:58
  • If a bug fix introduces or reveals another bug, this is not what I call "stable". I work for space industry, I know how much effort is put to remove bugs and I know that the person able to reliably tell "there is no more bug in this software" is not born.
    – mouviciel
    Sep 13, 2011 at 7:24
  • I said that using statistical methods you can tell how many bugs are left to find. You could, in theory, get to position of saying "Statistically there no chance of any bugs", the same way cryptographers say that there is not way to brute force a 100000000 bit key, but for anything less than trivial software, it's impossible to put in the resources required to get any where close, let alone commercial suicide.
    – mattnz
    Sep 13, 2011 at 22:45
  • I agree, you can get an estimation of how stable a software is using statistical testing for fault forecasting. Anyway the main challenge of this method is the oracle problem, i.e., how to quickly (automatically) determine whether the outcome of a test is correct. And statistical testing needs a huge test suite for being relevant.
    – mouviciel
    Sep 14, 2011 at 7:30
0

When software gets close to shipping, you often get a lot more miner bug reports and bug reports that are asking you for new functionality. This shows that there are not many “show stoppers” that are preventing your testers using the app.

So talking about “percentage of bugs fixed” is not useful…

Improve this answer
0

Programming is not yet a science its more of a mixture of science and creativity mixed in with a few other things. One cannot simply measure quality by way of some counter in a bug report. For one there is no way of knowing all bugs in the system have been entered which would lead to a lower than actual score, ruining the analysis.

Improve this answer
1
  • You can't measure quality using a bug report, but you can measure quality, at least based on known issues. You can measure defects that affect each functional requirement and compute metrics based on the severity of the defect and priority of the requirement. You can measure complexity (which leads to an understanding of testability and maintainability). You can measure up-time, fault density, and compute MTTF. If you track the right information, you can produce a valid measure the overall quality of a system.
    – Thomas Owens ♦
    Sep 12, 2011 at 13:17
0

I always look at my bugs by how the program is going to be used and does it affect the user. If its a bad end system for developers that know what they are doing and they can avoid the bug with a little extra typing then you can relase it as an RC but not really a stable if its going to be used for users you have to think of every user like your grandparents they are always going to not know what to do.

Keep your known bugs as close to 0% as posiable for the amount of time. If you HAVE to relase something with bugs make sure you post the known bugs so people can be aware. And try to make sure your bugs are less critical. I was making a multilayered comment system. I had a bug that you could comment on any post but the first one. well that was nice but needed to be fixed because someone will want to comment on the first one.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.