Should QA find reproducible scenarios?

Question

Sometimes my QA team reports bugs, but neither I or them have any idea on how to reproduce them. This leads to very long and frustrating debugging sessions which sometimes do not even yield results.

My software is tied heavily with proprietary hardware so bugs can come from many directions at once.

Should I expect more from them than "your software crashed when I pressed a button" or should I figure myself what happened?

EDIT:

One of my coworker pointed out that we are probably all developers here so the results might suffer a little bias

Answer 1

QA should always try and make the bugs as easy for you to reproduce as possible and the bug description should contain the steps taken.

However, if they can't easily reproduce the bugs, they should still get entered into the bug database with suitable title/headings and a full description of what they did to cause the bug. The bug description should clearly state that they can't reproduce the bug (perhaps with some comment along the lines of "tried it five times, it happened once"). This way, if someone else sees the same bug, they can add to the bug database with their findings and also you get as much information as possible which further down the line could be vital in saving you time tracking down the problem.

Also, you get to filter the information - there might be a lot of bugs in different systems that you know are all linked to (eg) one area of the code - if QA don't report anything (as they can't reproduce them) then this information doesn't get to you.

Answer 2

It looks like your QA department is doing too much exploratory testing (ie. They don't have a good test plan).

Exploratory testing is good, and identifies problem areas, but from there they should be defining reproducible test cases (ie. a test plan) to perform that will reveal specific bugs.

There's a number of reasons why a correct repro is necessary (not just good, but necessary):

1. You have to repro so that you can debug/track down the cause.
2. QA will need to be able to verify the fix once you're done.
3. Further test passes will need to do regressions on previous bugs.
4. Known bugs can be discarded if the exposure is too small or repro is too unlikely.

So, as SteveCzetty notes: Close it as no repro and get back to work.

Answer 3

If the bug can't be reproduced consistently, how will QA ever know whether it was fixed?

Answer 4

Yes, you should expect more from them. They should be able to say:

1. Started new instance of program
2. I pressed button A
3. Entered "test text" into the TEST NAME field on Form #1
4. Pressed button B
5. Observed that the program crashed with this message (see attached screenshot).

If all they tell is "it crashed", they're not very good. Even if the above steps are not 100% reproducible, a large enough sample of these crashes might help narrow down the cause, once a pattern is detected.

Answer 5

My advice is to read those bugs and give them a good old think. If you can't figure out a potential cause, forget about them for now.

QA should document every issue they find, even if they have no idea how it happened. It's QA's job to try and reproduce issues, but realistically this won't always be possible. Sometimes it doesn't have anything to do with what they did in the last 10 minutes. Something got into an invalid state a day ago, and it just became apparent because of one of the last 10 steps.

With these "1 in 1000" bugs, QA should try to reproduce them for a bit. If they don't have success, they should document the bug, then try a little more.

The reason why they should get the bug entered fairly early on is that the programmer knows the code a lot better than QA, and might immediately know the problem. It could be the code they refactored. It could be that function they half implemented then forgot about. They may have no idea, but there's no sense in the tester wasting a few hours trying to reproduce a problem that's obvious to the person who coded it. The tester can always add more details to the bug later.

The bug should include as much info as possible. Whatever the tester can remember about the lead-up to the issue should be written down in painful detail. Any Crash logs, database snapshots, or relevant screenshots should be attached as well.

If the bug is never reproduced, great! It doesn't hurt having it in the database. If the program is released and a user reports a similar bug later, you can compare their experience to what's in the report and look for any similarities.

In my experience, the juiciest bugs aren't found from following test plans. Sometimes you have to let things stew for a few weeks in order to have the moon and stars align that cause a nasty bug. If QA can do some detective work and find some possible causes, give them a pat on the back. But sometimes, who knows what happened?

Answer 6

A lot of bugs aren't reproducible until you know how to fix them. That doesn't mean they don't need to be fixed. I fixed a bug last year that I still don't know how to reproduce. It requires some bizarre combination of a precisely-timed transmission error together with very specific garbage data in a certain memory location on the stack. Unfortunately, one of our customers is "lucky" enough to get into that condition all the time.

So, by all means encourage QA to include reproducability steps where possible, but don't fault them if they can't. Sometimes it will help you know where to add logging. Sometimes all it does is tell you what doesn't cause the bug, but a bug report is always useful.

Answer 7

If you mean should QA include the steps to reproduce the bug if they can, then the answer is yes. If you mean should they only record bugs they are able to reproduce, then absolutely not. Bugs are bugs, whether they only happen at midnight on a full moon, or every time you look at it. Some bugs are non-deterministic (classic example is uninitialized variable, where the value picked up is semi-random), that doesn't mean they shouldn't be recorded, investigated, and if possible fixed.

Non reproducible bugs should generally have a low priority, but they should definitely be recorded.

Answer 8

IMO, you should. QA aren't doing their job if they can't give you any reproduction steps. Don't waste your time trying to reproduce something that you can't, just close it as "Cannot reproduce" and move on.

Your time is much more valuable than that.

Answer 9

A bug report should contain:

• Steps to reproduce
• Actual behaviour
• Expected behaviour

E.g.:

• I deleted all suppliers from the database (using DELETE * FROM tSuppliers), opened the supplier dialog, and clicked on the Supplier drop-down list.
• The list contained the following message: GUPOS ERROR #0000000: SOMETHING WENT WRONG!. When I clicked on the message, the app disappeared from the screen, and the Task Manager.
• I expected to see either an empty list or (preferably) a message such as "No suppliers found". Clicking on the empty list or the message should have no effect. The app obviously shouldn't disappear without warning under any circumstances.

So, yes - it should contains the steps to reproduce. The fact that they don't feel the need to include this would seem to indicate that they think their job is to "break the software", rather than to identify faults.

Answer 10

QA should be able to reproduce the bugs based on the steps entered. If they tried hard , still could not reproduce, they should still enter the bugs with as much as details they have with the timestamp so that the developers can take a look at the application and debug logs for more details.

Answer 11

Money is at stake here. Why should any team member be able to create a poorly-defined, low-chances-of-success task that burdens a (hopefully) highly-paid developer?

This isn't about pecking order, hierarchy, arrogance, us vs. them, or anything like that. This is just about investing in activities that add value to the product.

When a problem can be demonstrated to negatively and measurably affect the value of the product, then it should be investigated, reproduced, and fixed. Fix your product defect pipeline to filter out the noise.

Answer 12

your QA team sucks

Go to them and tell them to read a document that any professional tester has to have printed right inside their brains (I was a tester once and I still have it right there, in my brain): How to Report Bugs Effectively.

Particularly, point them to the section "Show me how to show myself":

This is the era of the Internet. This is the era of worldwide communication. This is the era in which I can send my software to somebody in Russia at the touch of a button, and he can send me comments about it just as easily. But if he has a problem with my program, he can't have me standing in front of it while it fails. "Show me" is good when you can, but often you can't.

If you have to report a bug to a programmer who can't be present in person, the aim of the exercise is to enable them to reproduce the problem. You want the programmer to run their own copy of the program, do the same things to it, and make it fail in the same way. When they can see the problem happening in front of their eyes, then they can deal with it...

---

If they start yelling at you complaining that "bugs can come from many directions at once", tell them they suck even more than you thought before. Tell them that Testability is a feature they should evaluate among other project features and they should invest efforts into getting this feature right.

• Testability improvements one could get when there is a professional tester focused on it could be much like magic. I learned that myself few months ago. QA engineer working with our team gave me a handful of testability requests to develop for some components in our application. Things he asked about made very little sense to me but I just gave it to him assuming that he knows better as a professional. Soon after I finished, he came up with a tool that reduced testing efforts by order of magnitude. He said most of it was based on these cryptic requests I implemented, go figure.