6

When QA discovers an issue during testing should they...

  • Log the bug with the test case and scenario information and move on testing?
  • Try investigating to determine why the bug is happening?

I would think it is normally both, but some people are saying it is not QA's job to even try to find root causes. QA is just meant to point out bugs. Interestingly this sentiment is coming from QA not developers.

Some background

Imagine this is happening in a very small team (3-4 people) working in an agile-ish environment. The developers are continually adding new features and fixes. Unit test coverage is very low. This is mostly due to the fact the developers have been pushed to deliver features first. QA is relied heavily on verifying correctness and functionality. Generally the developers are open to working closely with QA on things.

The QA team is testing in parallel. As soon as new builds show up they start their regression and new functional tests. However there are no automated tests. All tests are manual. Luckily most tests are quick to execute since they are simple comparisons against expected results.

Added

In the overarching organization, there are mostly two breeds of QA. There are the automation QA who do what their title suggests. And there are testers. Some float between the two from project to project, but for the purposes of this question, the QA are meant to be testing.

Perhaps to rephrase, what is a appropriate level of detail QA should be recording about bugs when they come across it (only the Devs will read these details)?

  • Is "Calculation X is wrong when using attached data file" sufficient?

  • Or, "Calculation X was off by a some constant. When we change value Y1 in the data file, the difference is equal to the value of Y1."

  • 1
    "Imagine this .... - All tests are manual. Luckily most tests are quick to execute" - sorry, but that is too imaginary for me, – Doc Brown Dec 19 '14 at 4:31
  • @DocBrown - Well without going through exactly what processes QA does I kind of have to leave it general. A good portion is load data, export results, compare against a control. Rinse and repeat. There are good reasons why this can't be automated. Why it can't be unit/integration tested is another matter. – tyh Dec 19 '14 at 15:25
  • Also, while it is very easy to confirm if something is working. Investigating why it does not work is not. The flow of data internally is not straightforward and there are many points that feed into a single value. – tyh Dec 19 '14 at 15:30
  • @tyh I think the point Doc Brown was getting at with that comment is that any manual test takes significantly longer to run than if it were automated. Personally I fail to see how your code cannot be tested automatically, if your system is hard to test automatically then I'd feel that the system is probably not well designed. I've done many programs that have ended up impossible to add unit tests to, but I've also re-done many of those systems with unit tests. If it can't be tested then it's probably not a well crafted system imho. – Elliot Blackburn Dec 20 '14 at 22:44
12

Saying "QA" as a job role is akin to saying "health care professional." You've got nurses, nurses assistants, physicians, physicians assistants, nurse practitioner, surgeons, midwives, etc...

Trying to answer "how far QA should go" is much like trying to answer "what does a health care professional do?" - it depends on the job description the person was hired for and their capabilities.

I've seen QA be people that are just supposed to click and verify. I've seen QA where its programmers who are writing automated tests. I've seen QA where they have to put their stamp of approval on every release (or it doesn't go out). I've seen QA that has tried to do as little as possible and not claim any responsibility for the end product. I've seen QA where they take the A part of the title very seriously and are involved in the process of assurance of quality (making sure code reviews are done, getting deep into ISO 9000).

So what should your QA department do? What they need to. This is completely dependent on what you need them to do and what they are hired to do and what their responsibilities are. This is probably a discussion you should have with your management chain if it is necessary. Many programers will find they have had similar discussions about if they should be gathering requirements, writing documentation, preforming tests themselves - the same needs to be done and said of QA.

For issues of how detailed a bug report QA should give, it is an issue of quality vs quantity. If they are finding one bug a cycle and the test suite takes less than one cycle to do, then they have the opportunity (given that they have the appropriate skill set - because you're hiring good employees and paying competitively) they can then write a sufficiently detailed bug reports such that programmers can work from them more easily.

On the other hand, if they are expected to do 7 days of testing in 5, there is no way that they can do a complete root cause analysis (RCA) on the bugs they find - being under time pressure to finish the regression suite.

If QA is finding that each cycle there are sufficiently many bugs that while they do finish the tests, they can't do an RCA on each one, you will likely find the quality of the bug reports varying greatly. This becomes especially prevalent when there are no automated tests that the coders run - each time a new build is being done QA is finding itself having to make sure that each bit of functionality is still the same. Depending on the size of the application, this can mean lots of testing.

Related reading:

Minor rant - we work in an industry where people go for "code ninja" or "java wizard" as job title and such and then try to tell the "QA" people they should or shouldn't do something because of their job title? Read the job description. Do what is needed. If the job description needs to change, change it. If something needs to be done, do it.

  • I take your point and I will add some more details to see if it changes anything. I do agree with you on the fact that the role of QA can vary greatly from shop to shop and person to person. – tyh Dec 18 '14 at 22:36
3

In terms of "how deep should QA go", it really depends on the people involved and what kind of teams you have, as well as the project code base. At my current job, we don't really like the term QA any more because we feel that we are all responsible for ensuring quality. Our developers practice TDD and unit test as much of the code base as we can (as well as integration and pretty much any other automated tests we can build). We then have a small team of testers, these guys have been using the products pretty much since day one and know it inside and out (from a user perspective).

When they find a bug that slipped through the automatic testing they report it to us with as much detail as they can muster from a user view. They effectively attempt to pin down the issue as fast as possible into recreation steps and these steps then become the acceptance criteria. Ie: "if x y and z no longer cause j and k to happen, then the bug is resolved."

In terms of who's job is it to find the root cause, we say whoever finds the bug should attempt to give as much detail as possible to whoever works on a fix. This resolves the issue as fast as possible and knowing the extent of the issue up front will only help to speed this process up.

If your testers who act from a user view won't attempt to recreate the bug in a number of different ways to gain more information to resolve things quicker, then I'd say you've got an unhealthy separation of teams going on. Development and QA go hand in hand, friendly rivalry is great and beneficial but when one team starts getting awkward about things then you have a problem. Phrases like "well the acceptance criteria is technically met" or "I found the bug, it's not my job to suggest a fix", I think you need to be looking at how your teams are working together to achieve high quality rather than questioning who should be doing what specifically.

“I expect QA will find nothing [...] QA ought to wonder why they exist” - Uncle Bob.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.