13

There are some types of bugs which are very hard to reproduce, happen very rarely and seemingly by random. It can happen, that I find a possible cause, fix it, test the program, and can't reproduce the bug. However, as it was impossible to reliably reproduce the bug and it happened so rarely, how can I indicate this in a bugtracker? What is the common way of doing it?

If I set the status to fixed, and the solution to fixed, it would mean something completely fixed, wouldn't it?

Is it common practice to set the status to fixed and the solution to open, to indicate to the testers, that "it's probably fixed, but needs more attention to make sure" ?

Edit: most (if not all) bugtrackers have two properties for the status of a bug, maybe the names are not the same. By status I mean new, assigned, fixed, closed, etc., and by solution I mean open (new), fixed, unsolvable, not reproducible, duplicate, not a bug, etc.

Improve this question
2
  • 3
    This is somewhat specific to your bug tracker. What other values can you assign to status and solution?
    – scarfridge
    Apr 26, 2012 at 6:34
  • In some bug trackers, there is a status of resolved and another status of closed. Only QA people are permitted to set the status to closed, but developers can set status to resolved.
    – Brian
    Apr 26, 2012 at 17:55

4 Answers 4

Reset to default
9

Is it common practice to set the status to fixed and the solution to open, to indicate to the testers, that "it's probably fixed, but needs more attention to make sure"?

Common or not, this is the right thing to do anyway, and you laid out why yourself: no matter how, it is a good approach to

indicate to the testers, that "it's probably fixed, but needs more attention to make sure"

Side note even if particular bug tracker does not have field like one you describe as solution, developer can at least add a free-form comment explaining above.

...and if bug tracker does not allow to add comments to the issue then it must be replaced with one that does. Ability to add free-form clarifications is a critically important feature since issues vary too much to fit into some pre-defined form.

Improve this answer
7

The test team will decide if the issue has been resolved and if it can closed. If there are any more regressions, side effects of the fix, or if the fix itself is not effective in another scenario, the issue will be reopened. But if you have done enough developer testing, then better to mark it as fixed.

Improve this answer
1
  • +1 - This is the simplest answer. If you have tried your hardest, and the test teams test suite is strong enough, what more can you do?
    – ozz
    Apr 26, 2012 at 9:11
3

There are types of bugs which are very hard to reproduce, happen very rarely and seemingly by random. It can happen, that I find a possible cause, fix it, test the program, and can't reproduce the bug.

Actually, if I there is no reproducible test scenario, I would not even try to fix such a bug beforehand. If you want tester to take more attention on it, give them a chance to create a reproducible scenario.

For example, let's say you change the program, and a tester invests 1 hour on trying to reproduce the bug, and the bug does not pop up - was one hour enough? Or is testing further a waste of time because the bug was already fixed?

On the other hand, when you don't change the program, and the bug does not pop up in 1 hour, most probably the tester should invest another hour in trying different things. And when the tester invest one day and cannot reproduce the bug any more - is it really worth trying to fix it then?

Said that, you can think about how you model that process in your bug tracking system: not trying to fix it and handing it over to the testers may be a bug status like "open". If the testers cannot reproduce it, it is obviously "not reproducible". Hopefully, this does not happen, they find a reproducible scenario, you can find the root cause of your bug, fix it and set the status to "fixed". Try to avoid getting into something like "don't know if it's fixed".

Improve this answer
6
  • 4
    For certain bug types a reproducible test scenario simply doesn't exist. For example, a timing-related bug could happen 1 time in a million on average - but it's impossible to predict whether it will be on the 3rd or 532454th run. Nevertheless, such bugs are bugs and must be fixed.
    – Joonas Pulakka
    Apr 26, 2012 at 9:38
  • 3
    @Joonas Pulakka: I agree. And such bugs can depend on external circumstances. In case of embedded, they can depend on power surges caused by something out of your control. Not trying to fix it is not always the best solution, especially if I happen to find a code smell that I suspect it can be a cause of that bug. In this case, why shouldn't I fix it?
    – vsz
    Apr 26, 2012 at 10:04
  • 2
    @JoonasPulakka: to my experience about reproducible scenarios, in most cases where people say "it is not possible", they just missing the right idea to make things possible. In your example, one could set up a scenario with a "10 million run" loop, making it at least very probable to show up the bug in a reasonable amount of time.
    – Doc Brown
    Apr 26, 2012 at 10:56
  • 2
    @vsz: you should fix it, of course, but what I am suggesting is that one first should create a test (or give testers a hint what to test), and then fix it, not vice versa.
    – Doc Brown
    Apr 26, 2012 at 10:59
  • 2
    @DocBrown is right, another way to think about it is that sometimes bugs require a statistical approach to "reproduce" them. It might very well be that there is a very specific set of inputs/circumstances that reproduces the bug, but you might NOT have any idea what these inputs are and the set of possible inputs might be too huge to iterate through. In these cases, one approach is to collect stats about the occurence of bug every time you try to address it. It might take a long time, and the results might not give you 100% "confidence" in a statistical sense, but sometimes that all you have.
    – Angelo
    Apr 26, 2012 at 11:32
0

Sometimes the only evidence you have is purely statistical, e.g. it occurs once or twice a month, but otherwise seemingly unconnected to anything. These are overall the worst type of bug to diagnose and resolve that I have ever encountered, because you cannot tell whether your fixes have an effect with any certainty. The last one of these I had to resolve ended with a statistical fix: the symptom's frequency went down to 10% that we started with. The final piece was never found, or maybe it was, but nobody had any way to tell.

Two pieces of advice I have are (1) assume multiple causes could be in effect until you know otherwise, and (2) hypothesize how the symptoms could possibly exist, then tear apart every line of logic that is even remotely involved. Deep walkthroughs are sometimes the only means to a satisfying end.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.