How QA/testing process can be evaluated?

Question

I asked this question on Programmers about a crazy idea to evaluate how testers are doing their job. From the answers/comments, it seems community also considers that as a crazy idea. This is a spin off from that question.

From one of the answers in that question

On the other hand, we tend to let the customers verify the work of the QA teams, which is possibly not ideal. It is a very powerful feedback loop though.

As the answerer himself says, it is not ideal. The whole point of QA is to make sure that highest quality product gets delivered to the end user. We want our testers to report problems, not end users.

So my question is , if John is a Project Manager, what are the methods he can use to evaluate QA process? How he can identify QA guys who are doing it correctly or incorrectly? What are the obvious little things that should be noticed and acted upon?

Answer 1

Ultimately, the metrics that you choose depend on the specific questions that you want to answer about the quality of your product and processes:

• If you want to know how good your defect reports submitted are, track the number that are withdrawn or marked as not reproducible. You'd also want to track what organization (development, quality, customer support, etc.) submitted it. You want a high percentage of the reports to contain sufficient information to reproduce and then fix the issues.
• If you want to know how effective your testing is, look at test coverage. If you are doing requirements based coverage, map test cases to requirements and ensure that all requirements are covered. If your quality team is writing test code at a unit and integration level, look at code coverage.
• Look at defect containment metrics, such as total defect containment effectiveness (TDCE). Use the equation TDCE = (pre-release defects found) / (pre-release defects + post-release defects). You may need to be careful what you consider a pre-release / post-release defect. If you only want to count the effectiveness of your back-end quality activities, pre-release defects is only defects found by the quality organization and post-release defects is anything found after they finish their work. You could consider pre-release to be all defects found by anyone before release to a customer and post-release defects to be those found by customers or in-house after the release.
• Number of defects found per phase. This depends on your process, but examples would be requirements, design, code and unit test, integration test, acceptance test, post-release.

Something that is important to consider, though, is that your developers should be involved in quality assurance. They should be reviewing the requirements to ensure they are correct and usable, the designs to ensure that they can implement the software, the code should be both reviewed and unit tested. QA is more of a process to give you confidence that the product is of high quality, and not an organization that inspects or tests the final product.

Answer 2

Quality management, both assurance and control, is risk mitigation against poor performance and defects, both of which are probabilistic. Part of the probabilistic outcome is aleatory, meaning due to random variability where no action has any real effect. Trying to decipher how well your QM activity is doing when you have both random and non random effects would be near impossible. In other words, you will credit or blame QM on either a favorable and unfavorable result that would have occurred anyway.

There is also something odd about doing a quality analysis on the function that does quality analysis on the rest of the team and processes and outputs. In a project environment, where time and money are always constrained, I doubt that would yield any real value.

At the end of day, you measure your performance and outputs that the entire project team is a part of, including the quality management team. If performance and outputs are meeting objectives, then I believe it is safe to assume your quality management capability is partly to blame. If objectives are not being met, and your quality management team are producing findings that require attention, whether during QA or QC, then that is another good sign they are performing. If objectives are not being met but neither QA nor QC are introducing any findings, then that is a good sign they are under performing themselves. I think it is as easy as this; it should not require any more complex analysis than this.

Answer 3

Taking an ultra-simplistic view...

Defects are introduced into a product by the development operation. We (or more often, management) might say "If I get better developers (for "better" developers read "better people and better processes and practises) I will get fewer defects". This is undeniably true, but only up to a point- At some unknown point on the curve you will not be able to reduce defects by hiring "better" developers, but you will still get defects because, as we all know (except for management) it is technically and practically impossible to produce defect-free code.

So you hire testers to ensure that a) the product does what it is supposed to do so you all get paid and b) they find every single remaining bug left in the code... Except that you have a similar curve- At one level, hiring better testers means they will find more defects, but just like you cannot guarantee bug-free programming, nor can you ever do enough testing to guarantee you have found all the bugs. Some will always remain. And it is the I.T. equivalent of Sods Law that even though the remaining defects are so obscure your testing coverage does not find them, on day 1 the users will (or thereabouts).

The trick is knowing whether your developers are sufficiently good and your testers are sufficiently good that you are producing the best possible outputs to the users, accepting the fact that there will always be some residual defects after testing no matter how much you do.

But let's suppose you have cause to believe that some element of the delivery operation is not operating to maximum possible quality and the users are seeing more defects than they could be. Where do you go from there?

Well, measure the amount and types of defect found in System testing, in UAT testing, and then after the product goes Live. Now you know how many "findable" defects there were in your product release. You can then assess why UAT didn't find the ones that were reported in Live, and why System testing didn't find the ones that were reported in UAT. Lastly you can assess why the defects made it out of the development shop at all.

You will get a wide range of effects and answers, but when you have them you can devise a cost/benefit analysis for mitigating whatever issues you find. Then when you have mitigated those effects to the point where spending more time or money wouldn't give you any appreciable difference in the residual defect rates you have achieved all you can achieve and you have the best practical delivery operation possible for you...

After that, issues will still get found, and users will still get annoyed and management will still ask why it is possible to deliver code with defects "even after spending all this money to make our delivery the best it can be", but you will know you have, as an organisation, done the best you can and you may rest easy at night :)

Answer 4

The answer to your question is probably 2-stage.

First, QA has a policing role. This can be measured in nearly any project and you could use a fairly simple metric like ratio of defects caught by QA vs number of defects discovered (by anyone) in production. This will you how good your QA team is at catching defects and it's not a bad place to start. If a lot of defects are getting through, you can see if there are trends in the types of defects coming through or maybe defects are most often getting through when QA gets overwhelmed close to deadlines.

However, too many organizations stop here and that is both financially wasteful and ultimately ineffective. You can't solve the defect problem with testing alone. You want to leverage the perspective of your QA experts throughout the process. When I was managing a QA team, we'd have people embedded in all of the agile teams. When reviewing user stories and tasks, they QA team member would asking questions about use cases and potential hangups that would become test cases, but would more importantly help frame the work the developer would do. They'd also be present for architectural and tactical discussions as well as code reviews to create a broader perspective in the team. Finally, if your QA team members are asked to make quality their priority, they can provide a check against the pressures on developers to rush through tasks that we all know can occur in projects.

Leveraging the QA team as more than just a policing force will bring down the total number of defects created in development. Obviously, catching 95% of 100 defects in a project is a much more palatable scenario than 95% of 1000 defects. Once things are more integrated like this, you can look at the total generated defects as a measure, but you have to remember at that point that you're talking about a measure that reflects the whole organization, not just QA anymore.

Answer 5

If you're looking for quantitative metrics that show if QA is doing its job, here are a couple common ones:

number of open defects and # of total defects, the ratio between the two, the change in the ratio throughout the iteration.

number of P1, P2. P3. P4 defects found in pre-prod environment vs production environment

Average Defect cycle time

Here are some more qualitative things to look at that are strong indicators of performance:

How well is the defect documented? Does it at least have a severity, actual outcome, expected outcome documented?

Does your QA team write code? Do they automate tests? Do they write an automated test when a defect is fixed?

Does your QA write test cases? When do they write them - before a story is developed or right at the end when it needs testing?

Do they perform regressions? When? What is regressed in terms of business critical functionality? Is the regression automated?

Can your QA justify, in terms of business value, what they decided to test and how?

During retrospectives, does QA provide suggestions for improvement? Does the team listen?

How is the communication between QA and developers? What does Dev say about the value added by QA on the team?