-1

My question is similar to Leaving intentional bugs in code for testers to find:

We don't do this at our firm, but one of my friends says that his project manager asked every dev to add intentional bugs just before the product goes to QA...

...If you ignore the scenario were one of these intentional bugs gets shipped with the final product, what are the other drawbacks we should consider before even thinking of adopting this approach?

The difference is that I had different premises. I wanted to make a point without messing with anyone. Intended bugs not for motivation but for testing the testers, to point out that quality assurance process is dysfunctional.

I was working on a rather large and important feature. Critical part the of feature was that users could get a discount under certain circumstances. Problem that we had with that project was that QA was very sloppy and missed a lot of important bugs. On the other side project manager had a lot of faith in QA. In my opinion bad QA is worse than no QA, because it only adds illusions about quality. Couple of times I got into arguments with PM but I gave up and just decided that I should be my own tester and just assume that QA doesn't exist.

At some point a brilliant idea came to my mind - to test the QA. I found completely outrageous bug and decided that instead of fixing it I would leave it as is and see if the QA finds it. Bug was really bad. When user placed an order and got a discount, the browser redirects to payment provider and there user sees price without discount and is charged accordingly. This was a critical path bug.

Long story short, QA didn't find a bug, I fixed it before release of course and even created a ticket and politely announced it without blaming anyone to see the reaction. Both QA and PM didn't really cared, just said okay and that's it.

After that I finalized my conclusions about our QA but PM still had a lot of faith in him. I told some time after to assistant PM in private conversation, she told me to speak to his management, but I decided not to pursue and just carry on on my own because I really didn't know how to present the case.

So my questions are:

  1. Is it good idea to test testers by leaving bugs or even creating them?
  2. How to handle the situation where it happened, intentionally or not?
  3. How to otherwise point out low efficiency of QA?
  4. How to do it without forcing people losing face and creating tensions and conflicts?
Improve this question
12
  • 2
    And then how do you test the tester-testers? ;)
    – Mason Wheeler
    Jun 4, 2015 at 19:00
  • @SJuan76 to certain extend, I would even say to high. But I present a specific case which I failed to resolve and still don't know how I should have done it.
    – Andrey
    Jun 4, 2015 at 19:02
  • @MasonWheeler yeah you need a way to test that tester-testers indeed left a bug for testers. Or maybe they made a bug in bug code so that it negated it. Complicated stuff indeed.
    – Andrey
    Jun 4, 2015 at 19:03
  • 1
    @MasonWheeler sort of like taxes: who audits the auditors? I say, turtles all the way down.
    – user22815
    Jun 4, 2015 at 19:03
  • @SJuan76 I think this is meaningfully different because here, the motivation seems to be proving a deficiency with QA so that management might change something, while that question is more about ensuring QA remains effective over the long-term.
    – Ixrec
    Jun 4, 2015 at 19:04

3 Answers 3

Reset to default
8

Is it good idea to test testers by leaving bugs or even creating them?

No, you don't want your employees playing petty games about who can stump who, rather than delivering actual value.

If your QA department is not delivering value you likely:

  • Have a company who assumes testing is not worth paying for and hires duds for QA roles
  • Don't have documentation about functionality to test
  • Have bad/nonexistent tests for key functionality

How to handle the situation where it happened, intentionally or not?

You are a professional.

Professionals don't play silly jokes to "gotcha!" their coworkers. So, don't do it again.

How to otherwise point out low efficiency of QA?

Do your customers complain? Most people with pay levels able to affect this sort of problem don't care if a dev is upset.. but they care if the customer starts having significant problems, bugs, or otherwise lost revenue.

How to do it without forcing people losing face and creating tensions and conflicts?

Document how this affects the end user and show management that insufficient QA processes exist.

Realistically, you probably need a combination of:

  • More automated testing
  • Better defined QA requirements (if you have user requirements, convert these into stories/activities QA must do before a release)
Improve this answer
4
  • "don't do it again" I don't see any harm in playing such joke if I keep it to myself. I did it solely to understand if I can rely on QA or not. My job is to deliver quality product, from that moment I just assumed that I need to be my own QA, test more, write more unit tests etc.
    – Andrey
    Jun 4, 2015 at 20:37
  • 3
    @Andrey you can do whatever you want. As a professional, I'm not going to intentionally submit bugs just to "gotcha" our test team, I'm going to approach problems in my workplace professionally. And ignoring that, just imagine you do this and are gone suddenly for some reason - illness, vacation, new job, etc - and those intentional bugs get into production. Imagine how that conversation will go when your boss realizes you intentionally put serious bugs into the codebase...
    – enderland
    Jun 4, 2015 at 20:43
  • There were no immediate customers, it was public web site. We always received a lot of feedback from users, so variations were not really significant. Very small majority of users were paying so it is not always trivial to judge whether many clients suddenly became significantly unhappy. Lost revenue is also not immediately evident.
    – Andrey
    Jun 4, 2015 at 20:45
  • as a professional I must realistically assess the situation, if I don't frame anyone and keep those experiments private I think it may be useful instrument. In that particular case I knew that nothing would go live without me. Funny thing that in the end we still found couple of bugs actually after project went live.
    – Andrey
    Jun 4, 2015 at 20:51
1

I think this is the critical part of the issue here:

I was working on a rather large and important feature. Critical part the of feature was that users could get a discount under certain circumstances. Problem that we had with that project was that QA was very sloppy and missed a lot of important bugs. On the other side project manager had a lot of faith in QA. In my opinion bad QA is worse than no QA, because it only adds illusions about quality. Couple of times I got into arguments with PM but I gave up and just decided that I should be my own tester and just assume that QA doesn't exist.

I will answer your questions in the context of that paragraph.

Is it good idea to test testers by leaving bugs or even creating them?

No, you test your testers by measuring legitimate defects that made it through QA. Sure, if they miss an esoteric bug that is nigh impossible to reproduce or find then go easy on them. If they report a bug that project management chooses to ignore then it blows up with the customer you absolutely do not make it into an issue with QA.

How to handle the situation where it happened, intentionally or not?

If your QA are lazy and let obvious or major bugs through and your testing procedures would have caught them then this becomes a performance evaluation issue. I would not advocate employee performance being strictly related to bugs (e.g. you let 25% of the defects through this year, your annual raise is being docked 25%) but clearly if a QA employee is not finding obvious bugs (you know, his job) then maybe he should not be employed anymore.

How to otherwise point out low efficiency of QA?

Measure it, but be reasonable. See my explanation above. Everyone makes mistakes. But if a manager is hesitant to assign a QA employee to a high profile project because of performance concerns, well, his gut feeling just figured it out.

Yes, I have seen this happen.

How to do it without forcing people losing face and creating tensions and conflicts?

Not going to happen. You can be polite and professional all you want. At some point, QA are simply allowing too many bugs to slip through and management must address it or face economic consequences (e.g. customers jumping ship to competing products).

The alternative is to use management tools to address the poor performance: training, assign to an easy project, paired testing, and eventually, the other path: written reprimands for poor performance, and eventually, termination.

Improve this answer
0

Low-quality QA is wretched, and if you can't get Management to recognize that they're not doing their job, that's even worse. But the only person that's going to be blamed for intentionally putting in a terrible bug is you - for wasting everyone else's time, and for putting in a bug that, if QA is as bad as you say it is, will go into production and ruin the company.

Best case scenario - you get blamed and lose your job. Worst case scenario - you get blamed and everyone loses their job.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.