How detailed should QA aim to be?

Question

When QA discovers an issue during testing should they...

• Log the bug with the test case and scenario information and move on testing?
• Try investigating to determine why the bug is happening?

I would think it is normally both, but some people are saying it is not QA's job to even try to find root causes. QA is just meant to point out bugs. Interestingly this sentiment is coming from QA not developers.

Some background

Imagine this is happening in a very small team (3-4 people) working in an agile-ish environment. The developers are continually adding new features and fixes. Unit test coverage is very low. This is mostly due to the fact the developers have been pushed to deliver features first. QA is relied heavily on verifying correctness and functionality. Generally the developers are open to working closely with QA on things.

The QA team is testing in parallel. As soon as new builds show up they start their regression and new functional tests. However there are no automated tests. All tests are manual. Luckily most tests are quick to execute since they are simple comparisons against expected results.

Added

In the overarching organization, there are mostly two breeds of QA. There are the automation QA who do what their title suggests. And there are testers. Some float between the two from project to project, but for the purposes of this question, the QA are meant to be testing.

Perhaps to rephrase, what is a appropriate level of detail QA should be recording about bugs when they come across it (only the Devs will read these details)?

• Is "Calculation X is wrong when using attached data file" sufficient?

• Or, "Calculation X was off by a some constant. When we change value Y1 in the data file, the difference is equal to the value of Y1."

Answer 1

Saying "QA" as a job role is akin to saying "health care professional." You've got nurses, nurses assistants, physicians, physicians assistants, nurse practitioner, surgeons, midwives, etc...

Trying to answer "how far QA should go" is much like trying to answer "what does a health care professional do?" - it depends on the job description the person was hired for and their capabilities.

I've seen QA be people that are just supposed to click and verify. I've seen QA where its programmers who are writing automated tests. I've seen QA where they have to put their stamp of approval on every release (or it doesn't go out). I've seen QA that has tried to do as little as possible and not claim any responsibility for the end product. I've seen QA where they take the A part of the title very seriously and are involved in the process of assurance of quality (making sure code reviews are done, getting deep into ISO 9000).

So what should your QA department do? What they need to. This is completely dependent on what you need them to do and what they are hired to do and what their responsibilities are. This is probably a discussion you should have with your management chain if it is necessary. Many programers will find they have had similar discussions about if they should be gathering requirements, writing documentation, preforming tests themselves - the same needs to be done and said of QA.

For issues of how detailed a bug report QA should give, it is an issue of quality vs quantity. If they are finding one bug a cycle and the test suite takes less than one cycle to do, then they have the opportunity (given that they have the appropriate skill set - because you're hiring good employees and paying competitively) they can then write a sufficiently detailed bug reports such that programmers can work from them more easily.

On the other hand, if they are expected to do 7 days of testing in 5, there is no way that they can do a complete root cause analysis (RCA) on the bugs they find - being under time pressure to finish the regression suite.

If QA is finding that each cycle there are sufficiently many bugs that while they do finish the tests, they can't do an RCA on each one, you will likely find the quality of the bug reports varying greatly. This becomes especially prevalent when there are no automated tests that the coders run - each time a new build is being done QA is finding itself having to make sure that each bit of functionality is still the same. Depending on the size of the application, this can mean lots of testing.

Related reading:

• Qa Is Not Qc and Quality Assurance Is Not Quality Control
• Software quality assurance

^{_{Minor rant - we work in an industry where people go for "code ninja" or "java wizard" as job title and such and then try to tell the "QA" people they should or shouldn't do something because of their job title? Read the job description. Do what is needed. If the job description needs to change, change it. If something needs to be done, do it.}}

Answer 2

In terms of "how deep should QA go", it really depends on the people involved and what kind of teams you have, as well as the project code base. At my current job, we don't really like the term QA any more because we feel that we are all responsible for ensuring quality. Our developers practice TDD and unit test as much of the code base as we can (as well as integration and pretty much any other automated tests we can build). We then have a small team of testers, these guys have been using the products pretty much since day one and know it inside and out (from a user perspective).

When they find a bug that slipped through the automatic testing they report it to us with as much detail as they can muster from a user view. They effectively attempt to pin down the issue as fast as possible into recreation steps and these steps then become the acceptance criteria. Ie: "if x y and z no longer cause j and k to happen, then the bug is resolved."

In terms of who's job is it to find the root cause, we say whoever finds the bug should attempt to give as much detail as possible to whoever works on a fix. This resolves the issue as fast as possible and knowing the extent of the issue up front will only help to speed this process up.

If your testers who act from a user view won't attempt to recreate the bug in a number of different ways to gain more information to resolve things quicker, then I'd say you've got an unhealthy separation of teams going on. Development and QA go hand in hand, friendly rivalry is great and beneficial but when one team starts getting awkward about things then you have a problem. Phrases like "well the acceptance criteria is technically met" or "I found the bug, it's not my job to suggest a fix", I think you need to be looking at how your teams are working together to achieve high quality rather than questioning who should be doing what specifically.

“I expect QA will find nothing [...] QA ought to wonder why they exist” - Uncle Bob.