10

If the customer has problems which are not reproducible because of complexity of actions they took and can't remember step by step, then coders are missing a test case.

Indeed, this issue is apparently authentic and critical, so do you think it is correct to turn away such bugs? What's wrong here? For me it looks as if the code is too complex so coders can't see all dependencies. They can't really say the code is definitely correct.

Please help me understand which behavior may be advisable in such situations. By the way, my company's program is technical selection software made with Delphi (standalone and web).

Improve this question
0

11 Answers 11

Reset to default
20

I think the answer here is going to depend on what your organization and its programmers want to do. If you are in a company with large, complex product offerings used by millions (say, Microsoft, though I have no idea how that company actually works internally), it may be that no one will ever have time to pay attention to the bug that has no steps to reproduce. If you are a smaller shop, it could be that someone will eventually have a little time to sit down and try to reproduce the bug.

Are you in a position where you potentially could try to reproduce the bug and provide steps to reproduce in the bug report? If so, and you have time, you may wish to go that route.

If not, I would at least recommend recording the bug in your bug tracking system (you do have one of those, right?), even if to mark immediately with "unable to reproduce" status or the like. That way, if the same bug happens to the same client or a different client a few months from now, you might be able to spot a pattern that will help the bug get fixed.

Finally, the unfortunate reality is that some bugs will not be considered worth the cost in personnel time to fix. If this bug represents an isolated incident or affects a very small percentage of clients, it may not be considered worth fixing from a business perspective, especially if there is a workaround available.

I hope this helps.

Improve this answer
5
  • @Andrew Thanks for quick reply. It's a quite small company (<25 empl). Yes I can try to reproduce and I've already created a bug report in our bugzilla. But my attempts to reproduce failed, also time is short (I sould 'fry other fish'). Actually why I don't get on with such things is that it could be serious problem when customer may lose project information - I think it shouldn't be acceptable the bug reoccurs. Just wondering how others challenge that. Well, lets see what time will tell.
    – seansilver
    Commented Mar 24, 2011 at 13:36
  • @seansilver If a bug is more serious (as you say, causes the client to lose project information), that may make the bug higher on the priority list from a business perspective too, especially if the client is considered "very important" in terms of amount of business you get from them. You're right, from a developer perspective, bugs really are not acceptable, but because time and money are scarce, economics demands that choices be made and tasks be prioritized. This bug is always something you can come back to if you are having a slow day, for example.
    – Andrew
    Commented Mar 24, 2011 at 15:16
  • These things happen, sometimes even with a detailed testcase. In my past employment we dealt with hardware as well, and sometimes even for the exact same binary and data, a bug will only show up probablistically, and even if you can occasionally observe it, diagnosing and fixing the cause can be very challenging. Often the customer has proprietary components as part of his system (either his data, but sometimes the OS envoronment) and the only hope is to send someone under proper non disclosure to the customer site.
    – Omega Centauri
    Commented Mar 24, 2011 at 16:03
  • 2
    @seansilver: If it's a real bug, it'll turn up again. Otherwise, if you can't narrow it down, you're going to be running in circles.
    – Satanicpuppy
    Commented Mar 24, 2011 at 23:11
  • +1 for getting a record of the bug even without a repro; there's also the possibility that your testers might see the record and have a clue what happened.
    – Ethel Evans
    Commented Mar 25, 2011 at 18:35
16

No

Customer -> "I have this issue"

Developer -> "I can't reproduce it, therefore it doesn't exist"

I'm sorry but that is simply not acceptable. More so it if is a critical part of an application.

Ignoring bugs is a "head in the sand" approach will inevitably lead to a negative situation in the long run.

Improve this answer
6
  • 8
    I agree with your example, but it sounds like the customer can't reproduce it either. Does this mean the bug does not exist? No, of course not. But should one spend time and energy on a bug that nobody, including the customer, can reproduce? Not in my book....
    – Jesse McCulloch
    Commented Mar 24, 2011 at 21:14
  • 6
    @Jesse McCulloch - This is why bugs have priorities. I'd argue that if the impact is low, the customer isn't complaining, and it is rare enough to be hard to reproduce you accept the bug, but de-prioritize it.
    – JohnFx
    Commented Mar 24, 2011 at 22:19
  • 4
    Seriously? "I had a bug, and I can't replicate it! Fix it!!!" And your response is to do...what? If it can't be replicated, by the customer or the programming staff, how can it be fixed?
    – Satanicpuppy
    Commented Mar 24, 2011 at 23:04
  • 3
    The "head in the sand" part is a little misplaced. It means ignoring something that is obviously there.
    – Rook
    Commented Mar 25, 2011 at 1:26
  • 4
    "my car is making an odd noise, sometimes".
    – magma
    Commented Jun 10, 2013 at 9:20
10

First of all, you cannot fix a bug if you cannot reproduce it, because you cannot ensure that your fix actually fix the issue.

This does not mean you should turn away the issue, as you need to keep the information for later in order to have as much information as possible WHEN you can reproduce the situation later. In other words, mark it in your issue tracker as "Cannot reproduce".

Improve this answer
6

There are several routes to go, and it comes down to a balance between cost and benefit.

One possibility is putting some logging and tracking-capabilities in place to make reproduction easier. Then you might not be able to reproduce the bug now, but perhaps in a few weeks or months time, the problem might become clear.

A second possibility is taking steps to recover and preserve information, ways of detecting lost information and restoring it.

If the customer is paying, you could explain how research on an unreproducible problem will cost time. Then the customer needs to decide wether it is worth it to investigate and to solve. (This might be the case if the complicated actions the customer performs were never foreseen.)

Even if the customer isn't paying, you can explain how you can't promise to solve it, but you can promise to put in effort to minimise the effects of it.

Improve this answer
3

Whether you can ignore hard-to-reproduce bugs or not depends on the kind of software you're developing. If you're developing video games, you'll probably ignore these problems. You already have the customer's money, and as long as the problem is rare enough not to influence product ratings, why bother. At the other end of the scale, if this "rare problem" means loss of money or even loss of human life, you'll definitely want to investigate. If you have repeat customers and you want to keep them happy, you'd probably look into it, too.

If you decide that it makes sense to solve the problem, the first step would probably be to change the code to get more information about the problem: For example, add logging to the code that's probably the reason for the bug.

Improve this answer
2

Do you have anything that describes your company's bugfix process? For example - how are bugs assigned to developers, what constitutes "fixed enough" to invoke a change in state (ie, a transition to the formal test team or to some sort of formal release cycle). Even if you work in a company with a lightweight process, someone there should be able to draw you a flow chart of how bugs enter and exit the system. Often an explanation of how this works will give some insight into what to do with bugs. Almost every bug fix process has some way for the developer to reject the bug, because it's quite possible to end up with duplicates, or bugs that are fixed as a side effect of a previously introduced fix.

There's also usually a severity rating - high severity bugs (that crash the system, introduce unrecoverable errors, or deny a feature from working utterly) usually require more diligent fixing efforts thatn low-severity bugs. If the bug can't be replicated and therefore isn't slowing your customer down anymore, you might be able to downgrade it's severity and put it into the "we'll fix it someday... but not today..." pool. That has the advantage that it's still around and active in case other similar bugs come in providing more information.

Also, what's the protocol for tracing the bug with the customer? Have you had the opportunity to communicate with the customer directly? Sometimes mutual agreeement is enough to give reason to decline the bug.

If you're a fairly new developer, this is a totally appropriate question to ask your management - they should be able to talk it through with you, hear what you've tried, and give you a judgement call on what to do in light of both the nature of the customer and the bug.

Improve this answer
2
  • thanks for your detailed information, actually the bugfix process is quite straightforward. Support/testing people record bugs and programmer fix it or not. Furthermore this task sharing is not fixed. So I'm really new developer in direct contact with customer and recorded a bug but someone else will be in charge to solve it - as soon as a test case exists. But what if even support staff declines? So I'm here to find some ideas how to break up (how to instill there should be a way).
    – seansilver
    Commented Mar 27, 2011 at 19:33
  • @seansilver - My best advice in the caes where you are recording bugs for someone else to try to fix is two fold: 1- get absolutely as much information about the user and their environment as you can - with the knowledge that this can come down to pure luck - often a customer will have done something that they did not know was significant that can totally change the nature of the issue. 2) learn as much about the area of the problem as you can from the developers - including any data or logging diagnostics you can do.
    – bethlakshmi
    Commented Mar 28, 2011 at 14:53
2

A reproducible test case is proof that a bug actually exists. A bug report with no proof is nothing but hearsay. It could be a real bug, or it could be caused by user error, a hardware problem on their system, or even cosmic rays flipping bits in RAM somewhere.

If I receive a bug report and I'm familiar with the codebase, I can generally get a good idea of what's going on from an informal "I tried to do X and it blew up" type report. If not, I'll ask for more information, any of the following:

  • An automated error report, if applicable. (You do have an exception-logger, right?)
  • A detailed list of reproduction steps.
  • A copy of their data
  • If all else fails, a screencap video showing the user's ability to reproduce the problem on demand.

If I can use any of these to get solid proof that a bug exists, it gives me a starting point to start digging into the code and tracking things down. But if I'm not able to reproduce the bug on demand, and neither is the user, then there's really nothing that I can do to fix it, because there's no proof that an actual bug exists in the first place. This is the point where I say "Sorry, I can't help you, but if you come up with a way to make it happen consistently, feel free to update the bug report."

Thankfully, I don't have to do that very often.

Improve this answer
6
  • +1: I don't know how these other people fix things...
    – Satanicpuppy
    Commented Mar 24, 2011 at 23:07
  • Sometimes bad things can happen which may or may not indicate bugs. For example, an AV component might unexpectedly change some setting because someone sat on the remote without realizing it. If the problem only ever occurs once, even if the layout of buttons would seem to make such an occurrence improbable, it may be hard to rule it out. If someone else reports the same problem, however, that would indicate there's likely some other cause.
    – supercat
    Commented Jan 28, 2015 at 17:43
  • Just reading this answer gives me a rash. It's an attitude that's all too common. "You have the audacity to complain? I'll make you rue the day." So, I, the customer, has to spend hours collecting crash reports, taking screenshots, writing out steps, apparently even making videos and at the end of all that you get in return is a one-line email "yeah, we can't reproduce that ..." Apple is very good at this. iDocs HD (ICSoft) proved to be absolute masters. All I can say is: you'll always win this game, but you won't be making friends.
    – Elise van Looij
    Commented Jan 29, 2015 at 13:31
  • @Elise: You have the audacity to complain? I'll make you rue the day. Not at all. Most developers (including myself) take pride in their work and want their product to be good-quality. But in order to fix a bug, we need to know the problem, not the symptoms. When you go to the doctor with a complaint of symptoms, the doctor can examine you directly to find out what's wrong. When you come to me with a bug report, I can't do that, so I need to deduce the cause by other means. It really is that simple: if I can't find what's causing the problem, I can't fix it.
    – Mason Wheeler
    Commented Jan 29, 2015 at 13:39
  • The doctor analogy is interesting. All a doctor has to go on initially, are the symptoms reported by the patient: "my shoulder hurts". Using his knowledge of the system, a doctor will then ask questions ("is the pain worse when you lie down?"), perform examinations, order tests and try to arrive at a diagnosis. Patients are not required at the door to hand in analyses of all their bodily functions, a set of total body-scans and a video compilation of their suffering. My beef is with the lousy bedside manner of many tech support. Hold my hand, don't order me around like an unpaid intern.
    – Elise van Looij
    Commented Jan 29, 2015 at 16:10
2

Record the bug. Document all you possibly can about it. What were they doing when it happened. <-- this is critical if you can find it. (WHEN it happened can matter too.)

Then in a while, when you work on a section of code, someone will find a "latent" bug.

And then it will be added to the bug tracker. And then that developer will think about the bug ; its projected appearance. And eventually the two bugs will be identified as the same bug.

True story

If you have a bunch of bugs with no steps to reproduce, they are data points.

This tells you : There are bugs out there.

They don't happen often.

So later, when a bug is found in the actual code... it might fit the data points you already have.

Google "Data mining"... your bug tracker might already know where a bug is

Improve this answer
1

I've had irreproducible bugs before where the best I could do at first was add logging to narrow it down to within 10,000 lines of code, but after a few iterations of that it became fixable.

I've also seen a bug with conditions that should be impossible to hit, where all I could do was explicitly check for that impossible condition, log a stack trace, and return cleanly instead of segfaulting. I still have no idea what the root cause is, but it's not affecting any customers at the moment, and if it ever does I will have more information to help fix it.

I've never had a bug where there wasn't something I could do to get closer to the answer, and customers are a lot happier even if all you can tell them is you added some code to keep them from getting hurt too bad if it happens again, and code to give you more information to help figure it out next time.

Improve this answer
0

How critical is critical in this case?

In our case, if there is a bug that is critical, and considering that our clients pay lots n' lots (tm) for our product, we create a quick fix team (2 ppl, test + dev) to find it, and squish it.

Otherwise, if it can't be repro'd, it gets closed as no repro. Like others have said, without repro steps, you're just guessing and shooting in the dark - we've got plenty of other critical bugs to work on that we can fix.

Improve this answer
0

It depends on two factors: the severity and probability. If the bug is fatal like data corruption or security violation you should keep digging till you find it and fix it because it will cause potential loss to customer's business. If it's not fatal but very probable and keep annoying the customer you have to find it as will cause problem to your organization by ruining the reputation of the product as the users will say it's a low quality buggy product. If it's extreme corner case and causes no harm to data and rarely happens then you may ignore it.

I also have to say that in most of these cases the developer should cooperate with testers to search for the bug as it might be difficult for one to find it alone.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.