We don't do this at our firm, but one of my friends says that his project manager asked every developer to add intentional bugs just before the product goes to QA. This is how it works:

  1. Just before the product goes to QA, the development team adds some intentional bugs at random places in the code. They properly back up the original, working code to make sure that those bugs aren't shipped with the end product.
  2. Testers are also informed about this. So they will test hard, because they know there are bugs present and that not finding them might be considered as a sign of incompetence.
  3. If a bug (intentional or otherwise) has been found, they will be reported for the development team to fix. The development team then adds another intentional bug in a related section of the code just before the product goes to the second-level QA. The project manager says a tester should think like a developer and he/she should expect new bugs in sections where changes were made.

Well, this is how it goes. They say that this approach has following advantages.

  1. Testers will be always on their toes and they will test like crazy. That helps them to also find hidden (unintentional) bugs so developers can fix them.
  2. Testers feed on bugs. Not finding any bugs will affect their morale. So giving them an easy one to find will help their morale.

If you ignore the scenario where one of these intentional bugs gets shipped with the final product, what are the other drawbacks we should consider before even thinking of adopting this approach?

Some clarifications:

  1. They properly backup the original code in source control.
  2. When a tester finds the intentional bug, the development team just ignores it. If tester finds out an unintentional (original) bug, the development team first checks whether it is caused by any of the intentional bugs. That is, the development team first tries to reproduce that on the original working code and tries to fix it if they can.
  3. Just ignore the relationship issues between QA and development team. I specifically asked this question on Programmers, not on The Workplace. Consider that there is good rapport between QA and the development team, and they party together after working hours. The project manager is a nice, old gentleman who is always ready to support both teams (Godsend).

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

  • 57
    "A test should think like a developer"... interesting. I would have thought it was obvious that a tester should not think like a developer but like a user. – Trilarion Feb 4 '15 at 14:05
  • 10
    What happens if an intentionally introduced bug covers up another bug the testers could have found had that intentional bug not been introduced? For example, suppose a chunk of code has a fencepost problem and that the development team is unaware of this bug. A programmer decides to insert an intentional fencepost error at that spot. Now the code has a double fencepost error. Suppose the testers detect the error, but don't see that it's a double fencepost error. Congrats! The testers found an introduced bug. The original code will be restored to contain the original fencepost error. Oops! – David Hammen Feb 4 '15 at 18:40
  • 18
    I'm a QE. I'd rather find real bugs, thank you. I'd quit this company like it's on fire. Nobody gets to (intentionally) waste my time. – ArjunShankar Feb 5 '15 at 12:32
  • 6
    "We don't do this at our firm, but one of my friends says that his CTO asked every product manager to add extra features at the start of every feature development cycle..." – Marco Feb 5 '15 at 17:41
  • 3
    I suspect adding intentional bugs creates risk. What if an intentional bug actually fixes something unintended? The positive side effect is not reported, the code is removed and a real bug gets through QA. By their nature, these last minute "intentional bugs" will be ill considered, if not, the bugs are wasting too much developer time. – Jodrell Feb 6 '15 at 11:56

21 Answers 21

This sounds absolutely nutty. It is expending a great deal of effort for very questionable benefit, and the practice seems based on some faulty premises:

  • That QA won't work hard unless they know they are being tested every day (which cannot be good for morale)

  • That there are not enough unintentionally introduced bugs in the software for QA to find

  • That QA's job is to find bugs - it isn't; it is to ensure the software is production quality

  • That this kind of battle of wits between development and QA is in some way healthy for the company - it isn't; all employees should be working together against the company's competitors instead of each other.

It's a terrible idea and the project manager in question is a jerk/idiot who understands nothing about people and motivation. And it's bad for business.


To expand on my description of "QA's job:" QA definitely should be finding bugs - both in the code and in their test suites - as an artifact of doing their jobs, but the role shouldn't be defined as "you have to find bugs." It should be "you have to keep the test suites up-to-date to account for new features and ensure all high coverage of testing. If this does not result in finding bugs, then the testing procedures are not sufficiently sophisticated for the product.

  • 17
    That QA's job is to find bugs - it isn't; it is to ensure the software is production quality This requires some clarification. Isolating and fixing bug is one important process in shipping production quality software. – Krishnabhadra Jan 28 '15 at 11:26
  • 21
    Actually, in many companies, QA's job is to find bugs, and if there were new features added to a product, and QA just then runs a test suite which does not show up any bugs, I personally won't trust that test suite and supect it to be imcomplete. – Doc Brown Jan 28 '15 at 12:08
  • 8
    I agree, except for the last point. Having an adversarial sort of approach between QA and development (and business) is largely inevitable. Each group has their own desires and expertise. As a company, these need to balance out to work well. In my experience, "playing nice" just leads to the groups not pushing for their agenda, leading to stagnation or imbalance. The best companies I've seen have been ones where development, QA, and the business side push for their needs, but acts as a check on the others, leading to compromise on the best balance for the company. – Telastyn Jan 28 '15 at 12:44
  • 41
    I would add another point: An intentional bug could hide a true one that would have appeared if the intentional bug didn't stop the process (by throwing an exception for example) before. – nkoniishvt Jan 28 '15 at 17:58
  • 29
    If I were a QA guy and found out I was wasting my time researching and documenting purposefully introduced bullshit bugs, I would be finding a new job. – Kik Jan 28 '15 at 20:01

Well, based on what I've learned:

  1. It's not a school nor job interview;
  2. The testers are not children;
  3. It's not a game;
  4. It wastes company's money.

The QA are not there only to find bugs but also to worry about how intuitive the system is, what is the learning curve for the user, usability, and accessibility in general. For example: "Is the system ugly?", "Is the user color blind and stuff is red and green?" They should complain too.

The minimal requirements for a system to pass QA is usually described in a user story for that particular feature or in how magical the PO wanted the system to be in his head.

tl;dr

It's not only bugs, testers should grow out of this narrow view.

  • 25
    +1 Strongly agree with all 4 points, especially the first. The competitive approach that so many newer devs bring often reflects their previous 15 years of schooling - an extremely competitive environment - unlike the workplace where co-operation would be a better approach. – Michael Durrant Jan 28 '15 at 12:51
  • 1
    Much prefer this answer to the top answer. – Pharap Feb 2 '15 at 0:24
  • 1
    "The QA are not there only to find bugs but also [...]" - I just want to say that in many places, the terms software testing and quality assurance are used interchangeably. Yes, that's bad. Where I used to work, we had an employ who used state -- at every QA department meeting -- what we do here isn't quality assurance but quality control. (She meant this as a criticism of our QA department.) – Mario Apr 17 '15 at 13:41
  • 1. It is school. Every day is a school day. If you are working in an engineering discipline but you don't want to learn every day, you should get out of my office. It is also an interview. Performance should be measured every single day to make sure the department is getting value for money. 2. If my career has taught me anything, its that QA have the mental capacity of 14 years olds. They are children and should be managed like a herd of sheep. – Gusdor Jan 19 at 9:27
  • 1. It's not a school in the sense that people can collaborate and not compete against each other for grades, there's no such thing as copying your work as tasks must be completed only once and it's no shame to ask for help from a colleague. And 2. If your QA is that bad your problem is in HR, and those are the ones who should get out of the office. – SparK Jan 19 at 14:54

Bad idea.

From the tester's point of view: "So they will test hard, because they know there are bugs present and not finding them might be considered as their incompetence." Basically the devs are booby-trapping the code. Few people like doing work which is ultimately pointless (because the bugs are known in advance) but which still affect how they are perceived. If there are tangible punishments for not finding the booby-traps, more so. And do you know testers thrive on finding bugs? That sounds like a toxic confrontational environment; a QA should be happy if the code they are examining is high quality. Although if they are paid by the bug... http://thedailywtf.com/articles/The-Defect-Black-Market

From the dev's point of view: the QAs are being incentivised to find the bugs you know are there. That may well increase the likelihood of real bugs going out of the door; the QAs are spending at least some of their time looking for the sort of bug that is easy to plant, not really subtle ones. Plus there is a small chance that a booby-trap may make it out of the door.

  • 31
    If they pay per bug, then this – BЈовић Jan 28 '15 at 12:22
  • 12
    "incentivized to find the bugs you know are there" Excellent point. If an organization is doing this, it likely means that someone is breathing down the QA folks necks to make sure they find the planted bugs, so that will be their top priority. What if they get together and figure out, say, "Hey the planted bugs are almost always that it fails to save one field on an edit screen with a bunch of data" (or whatever). Then they'll spend an inordinate amount of time looking for that one type of bug, and increase the chance that they'll miss other types of bugs. – Jay Jan 28 '15 at 18:23
  • The first thing that jumped to my mind was Wally's going to code someone else a minivan this afternoon – Dan Neely Jan 30 '15 at 22:13
  • 10
    >real bugs going out of the door. I used to do Big Testing. You start with the thesis that (non-trivial) code always has bugs. QA are the heroes who find them before the customer does. The bugs are always there. If you introduce artificial bugs you are wasting time you could be spending finding the real bugs; time for testing is limited, you are reducing quality by adding unnecessary work. – RedSonja Feb 2 '15 at 8:14

I agree totally with the answers above as to why this is bad for motivation and just generally awful people management. However, there are probably sound technical reasons for not doing this as well:

Just before the product goes to QA, the dev team adds some intentional bugs at random places in the code. They properly back up the original, working code to make sure that those bugs aren't shipped with the end product.

  1. Based on the first statement, you never actually test your intended production code in these two passes.

  2. I would imagine you vastly increase the likelihood of accidentally including an 'intentional' bug into your released production code when trying to rush through a change for a customer. Might cause a few red cheeks at some point.

  3. I would imagine that this just trains your testers to think like your developers (i.e. how would Tom add a bug here) which probably makes them less likely to find the bugs that Tom hasn't thought about.

  • 42
    +1 for you never actually test your intended production code in these two passes. How you can even think about releasing without testing the production code is beyond me; if you're testing again without the intentional bugs then you're repeating your effort and wasting the initial effort. – adamdc78 Jan 30 '15 at 0:21

Edit

I want to be clear that this answer is only talking about the concept of testing your QA process, and I'm not defending the specific methodology portrayed in the question.

End Edit

There is a valid reason to check if your testing/checking is actually working. Let me give you an example from manufacturing, but the principle is the same.

It's typical when feeding material through a machine that the feeder might not push the material through far enough. This is called a "short feed" and to prevent this we might install a "short feed sensor" (typically a through-beam type sensor that's blocked by the material). This sensor detects the end of the material when it reaches the full feed length. At a certain point in the machine cycle we check that the sensor is blocked, and stop the machine if the check fails.

Now you have to think about how the test itself can fail. For instance, some dirt or other debris can block the sensor and it will always report "OK" and never stop the machine. Also, the nature of the sensor is that the receiver turns on when the beam hits it, so depending on the type of sensor you have installed, electrically you get an "ON" input when the sensor is not blocked. That means if the cable got cut or power was lost to that sensor, or the input failed, the logic of your program would read "OFF" and that would mean "blocked" or "OK".

To catch these failure modes of the test, we typically insert a second check to make sure the sensor is actually unblocked during a second part of the cycle. In this way we check that the test is actually still operating (as best we can).

Similarly there are many ways that a QA department can fail. Perhaps the automated tests didn't run and the report is looking at an old copy of the test data. Perhaps someone isn't doing their job right. Testing the QA department is a reasonable thing to do.

Obviously the draw-back is that a "test bug" could make it through the QA department and into the finished product. In the manufacturing industry there are sometimes cases where a known bad part, sometimes called a "Red Rabbit", is inserted into the process (typically by someone from QA) and they watch that part go through the process and measure how long it takes to find the part and remove it. Normally this part is painted bright red (or orange) so it can easily be tracked. Since someone is watching the part go through process during this test, the chance of it making it into the final product is practically nil. There are, of course, apocryphal stories of someone throwing a known bad part into the process to "see if the system can find it", and of course having to quarantine all the final parts produced that day and manually sorting them, but that's just a case of not performing your test with due diligence.

  • 1
    Comments are not for extended discussion; this conversation has been moved to chat. – yannis Jan 29 '15 at 19:02
  • Hello all. The discussion was getting a bit too long for comments. As you can see from my earlier (automated) comment, I've moved all comments to a dedicated chat room. If you want to continue discussing the answer, please do it in that chat room, and not here. Thanks. – yannis Jan 29 '15 at 19:07
  • 3
    So described approach could be used to test QA occasionally, not as a permanent process. – gerlos Feb 3 '15 at 11:40

Honestly, I'd call this behavior blatantly unethical and impractical. The PM is in need of some serious retraining, if not termination.

  • It demonstrates a fundamental lack of understanding of the concept of quality assurance. Testers should not think like developers: they should think like end users. The entire reason for having QA teams is that developers are inherently too close to the code; QA is supposed to maintain enough distance from the code that they can catch what the devs miss.
  • It wastes QA effort. Assuming that these bugs are not trivial -see below for when they are- it means that QA is spending time and resources investigating things that are already known, when they could be spending that effort seeking out what isn't known.
  • It wastes developer effort. For QA folks to catch these non-trivial bugs, developers must first write them. This requires yet further effort, spent not just coding the bugs, but also considering the software requirements and design.
  • It puts production at needless risk. It is only a matter of time before changes don't get merged in properly.
  • If it doesn't do the above, then it is pointless. If all of the known bugs are trivial, then they won't catch substandard workers: they will only catch people who are doing no work at all. There are better ways to do that.
  • It poisons the work environment. Your QA testers are professionals. They should be trusted to be professional until there is actual reason to suspect otherwise. When there is reason to suspect otherwise, there should be a proper investigation instead of these mind games. Anything else kills morale.

Seriously. Even if the PM's paranoia turns out to be well-founded in this specific case, this is not someone who has any business managing testers.

Personally, I feel uncomfortable with this approach.

The main thing that concerns me is the practicality of inserting intentional bugs. This seems to me to be difficult to do in any way that is predictable.

Any code changes (intentional or otherwise) risk having side-effects. These side-effects may well be revealed during testing but it may not be obvious (even to the developer who planted the bug) what the root cause is. It doesn't feel "safe", if you know what I mean (I'm speaking from my gut here).

Also, the tester will waste a lot of time testing code that isn't actually going to be released. Once the intentional bugs are removed, a complete re-test should be done anyway, in my opinion. That's the whole point of testing. Something changes, anything, and you re-test everything. Ok I know that doesn't ever happen in practice, but that's what regression testing is all about.

So, overall, not convinced.

On the other hand, we tend to let the customers verify the work of the QA teams, which is possibly not ideal. It is a very powerful feedback loop though.

  • 1
    I like the idea of feedback loop power! – jxramos Feb 6 '15 at 2:19

It's a bad idea for all the reasons already given, but bug seeding is a useful tool for a different purpose. You can use it to get a rough metric of how effective the QA process is.

In its simplest case, let's say you seed 100 bugs and they're representative of the full span of real bugs (I know, unlikely, but I'm simplifying). You do not tell QA you're doing this to avoid spoiling the experiment. At the end of the QA process let's say they found 60 of the 100 seeded bugs (and other real bugs). Now you know QA is finding 60% of the bugs.

You can extend this further by counting the number of real bugs QA found and apply the fake bug ratio. In our example, if QA found 200 real bugs then you can conclude they only found 60% of them, so 133 remain.

Of course, this is just a broad estimate with huge error bars. Writing realistic, representative bugs is hard. The bugs you write are likely to be easier for QA to find because developers are trained to not write bugs. It may be better to simulate a class of bugs such as off-by-one errors, Unicode mistakes, buffer overflows, and so on.

This should be applied to the entire QA process which would include developer unit testing, continuous integration, and, if available, a dedicated QA team.

This is a metric, and should not be hijacked as a management motivational tool.

  • This would be the only way that any meaningful data could be collected. But the amount of time and effort that would be required to determine the proper test cases to get meaningful results would bust any budget and schedule. And even if you were given the budget and schedule you would then have to get over the hurdle of ensuring you have people qualified to understand statistics and software well enough to be able to identify the proper subset of tests. I don't think you'll get all of that in one project. So in real life the best this method can do is get erroneous, if not misleading numbers. – Dunk Jan 29 '15 at 16:20
  • 1
    SQL injection is a good one to do this for, as you can just pick n sql statements at random to "break" – Ian Jan 29 '15 at 18:19
  • 1
    A big problem is that intentional bugs will tend to be very different from issues you would get naturally - you might simply be training your QA to think like the programmers. That's pretty much destroying the whole point of QA - to have a POV closer to the customer than to the code. A huge part of QA is being a sanity check against things that devs think intuitive (either for ignorance of the ignorance of users, or for proximity to code, time spent with UI etc.). Intentional bugs are not a well distributed sample. – Luaan Feb 6 '15 at 8:28

Bad idea.

This is the sort of logical, binary approach that developers often bring, but it is demotivating for the QEs. It simply demonstrates a lack of trust. QEs often get placed in these situations without much input from them, and it assumed that they are OK with it, and it is not their place to suggest otherwise.

This kind of thinking combines to QEs being manual testers only and not being motivated to understand the actual code under test.

I am a senior QE and this is a familiar issue in most organizations I have worked in.

  • 7
    My wife did QA for 8 years, and just left for dev -- primarily because of trust issues like this. It's just insulting to the tester. – Bryan Boettcher Jan 28 '15 at 15:23

I'd say bad idea.

One: Programmers are going to spend time putting deliberate bugs in the code, and some effort to save the good version. While the testers should presumably be testing everything, including the features with the planted bug, when they find one they will presumably have to go back and re-run that test to verify that this was indeed a bug (and not that the tester got confused in some way). At a minimum, testers are going to spend time writing up the planted bugs. Then the programmers have to spend time fixing the bug that they planted. This is a lot of effort that could be spent trying to write good code and write up real bugs.

Two: It sends a clear message to the testers that the programmers and/or management think they are not doing their jobs and must be treated as children. I can't imagine that this is good for morale. As a programmer, if I was given ambiguous or contradictory specs for a program and had to spend a bunch of time getting them clarified, and then after wasting hours or days my boss told me, "Oh, yeah, I deliberately put contradictory statements in the specs just to make sure you were really reading them", I think I would be really annoyed. If that happened regularly, that might well be enough to make me look for another job.

In real life, all but the most trivial code changes WILL have bugs. I have never had a problem with testers getting complacent because the first draft code they were given was so often 100% perfect. I've had to deal with lazy testers who don't do an adequate job, but they didn't get that way because the programmers were so perfect. The best testing person I ever worked with once told me that for a new software release, he set himself a personal goal to find 100 bugs. Okay, whether 100 is a realistic number depends on how big the product is and how extensive the changes are, but in our case, he almost always managed to meet that goal. Sometimes he had to stretch things, like calling a mis-spelled word in a message a "bug", but hey, it did have to be fixed.

Post script: If you do this, I bet that sooner or later the programmers are going to deliberately plant a bug, the testers don't find that particular one, and the programmers forget to put the good code back. So now a deliberately planted bug gets shipped to the customer.

  • That point about the spec in "Two" is an excellent analogy. – Kyralessa Apr 17 '15 at 18:06

I don't really think this a bad idea. There's just a lot of things that I would speculate work better:

  1. Make QA accountable for the quality any way you can. For example by making support their responsibility also. This will increase their motivation to make sure the products shipped have higher quality. It always takes less effort to discover an inadequacy (bug, obviously missing feature, counter-intuitive behavior) yourself then to try to understand what your upset user is trying to explain. And putting some of that responsibility even on developers might increase their motivation to help the QA do their job the best they can.

  2. Have multiple QA teams, which can the compete. You need to find a sensible metric of course. Definitely not just the number of issues. Factoring in the severity of the defect or the business value (as determined by stakeholders) of proposed enhancements should help.

It's hard to tell whether QA is "good enough". It's easier and possibly even better in the long run to find ways for QA to be "ever improving".

Still, there's one issue to be aware of if you introduce intentional bugs: How do you know the "correct" code ever actually was correct in the first place? After the 2nd QA you remove all intentional bugs that hadn't been discovered. There's no way of knowing that you aren't either just replacing them by code that is broken in a different way or that you are not enabling broken behavior that was unreachable before (exaggerated example: some dialog did not open because of an intentional bug, but the dialog itself is broken - you just don't find out because testers didn't get to see it).

  • 5
    If you left that first sentence out I would have +1'd you because everything else is good:) It is simply a terrible idea, bad is an understatement. The easiest way to make QA accountable is by keeping track of the number of bugs that make it into the field. That alone will accomplish EVERYTHING the proposed method claims to be its benefits. – Dunk Jan 28 '15 at 15:56
  • @Dunk: Keeping track of that number won't make your team better automatically, much like keeping score in a sport doesn't make you the best athlete you could be. In fact athletes do train, i.e. perform artificial tasks to increase their performance in a controllable fashion, which is not unlike what is being proposed here. Unless you have an idea how to get people to improve that number, it's of little value. – back2dos Jan 28 '15 at 18:01
  • I don't claim that it will improve anything. I only claim that it will accomplish everything the "insert false errors" method will accomplish but without all the costs and wasted time. What it will do is give an indication if too many defects are passing through QA. If it is determined to be the case then either the process or the people need to be re-evaluated. The "false error" method doesn't provide any more information than that, but it actually provides less useful information. So your costs are higher for less gain using the "false error" method. Like I said, a terrible idea. – Dunk Jan 28 '15 at 18:35
  • @Dunk Then you didn't read the question properly. It suggests that this method increases morale and also thoroughness. Also the number of bugs that make it through QA doesn't reliably measure the effectiveness of the QA team. It is equally affected by how many bugs the developers introduce. If they start using TDD and there is a sudden decrease in defects in the release, what does that say about the testers? Nothing. – back2dos Jan 29 '15 at 8:01
  • @Dunk As opposed to that, the "false error" actually does give you more information, under the assumption that the difficulty of finding them doesn't fluctuate erratically (which can be arranged). Because you know how many artificial defects there are, you can tell exactly what percentage of them were caught in QA. So the extra information you get is how effective QA is in detecting artificial defects. And that number certainly correlates more with their overall effectiveness than the one you suggested. – back2dos Jan 29 '15 at 8:12

As others have said, the developers should not be purposefully adding bugs in the software, but it is a legitimate strategy for your test suite to add bugs into the software as part of the testing process.

It's called mutation testing. The idea is to use software to automate the creation of small changes in the source code (called mutants). The changes are designed to create different behavior, for example, we could change

if x < 10:
    print "X is small!"

into

# we flipped the inequality operator
if x > 10:
    print "X is small!"

and a good unit test should detect that the mutant code fragment no longer works as expected and kills the mutant. When the original code passes the test, and all mutants (that don't happen to be functionally equivalent) fail the test, then you know that your code and your tests are strong.

I like the idea. Was it General Patton who said, "The more you sweat in peace, the less you bleed in war."

Putting intentional bugs "wastes time" of the testers. But that also makes them work harder, meaning that they will also do a better job of finding UNintentional bugs. (And you have a copy of the "original" so you don't have to live with what you've done.)

Finding more unintentional bugs will likely save you more grief in the long run that the cost of dealing with the intentional ones.

Plus, you can get an idea of how good your testers are, not a small benefit in itself.

  • 1
    I think that there are good parts to this. It's better to find a bug BEFORE it makes it into the wild, and I'd rather press my internal QA (That's what they are getting paid for after all right?) than respond to external attacks. Bug Hunting is a part, and as long as this kind of testing is handled properly, I don't see why it can't be a valuable part. – WernerCD Jan 28 '15 at 14:53
  • 1
    The copy of the "original" may not be bug-free and is also by definition untested (because the code was changed to add bugs). – Roger Rowland Jan 28 '15 at 15:06
  • 1
    In my experience, bugs are not isolated animals and they don't sit alone. Software is part of a system and bugs - intentional or not - affect the system. Unless of course we're talking about trivial pieces of software. – Roger Rowland Jan 28 '15 at 15:14
  • 18
    There is zero proof that this method would find even 1 more additional bug beyond the intentionally inserted bugs. There is zero proof that this would make QA work harder to find bugs. They may try less hard. Also, since you've wasted an entire running of the acceptance test procedure testing cycle while testing intentionally broken code(Our full blown test takes 3 weeks), you now have to retest with the actual code that is going to be deployed because the broken version is not the same build, so its tests are pretty much useless for validation of the "real" build. – Dunk Jan 28 '15 at 15:21
  • 6
    I'm guessing that Patton meant that you should have rigorous training and field exercises during peace time. The analogy would be to have rigorous classes in IT school or post-degree training. I'm pretty sure Patton didn't mean that officers should be instructed to shoot at their own troops from behind to keep the troops on their toes! – Jay Jan 30 '15 at 17:01

There is no basis for a reward or punishment on its own merit, but on the result of the behavior you're targeting. And sometimes there are unintended consequences. Is the goal to keep the QA team from slacking or to make some manager feel like he's actually contributing something without realizing he's just getting in the way.

Positive Outcome - The QA team works harder to find bugs. Who knows, maybe they see this as a challenge. It's a friendly game. Or they're just doing because they're being watched (Hawthorne Effect?).

Negative Outcome - They may not work harder and find the bug anyway. QA sees this as petty and adversarial. So now, they go into hyper-bug finding drive and return all sorts of nit-picky little problems. That font doesn't render properly when I take a screen shot and convert it to a pdf and view it at 500%.

No Impact - sound to me like this makes no difference, so why bother? You just risk wasting time and irritating people.

We could all agree that this won't work 90% of the time. That doesn't do much good to the other 10%. Test things for yourself. Are customers more satisfied with a release that has the intentional code bugs? Does it impact worker morale and productivity in other areas? Increase turnover? You tell us.

  • Definitely agree with this leading to nit-picky problems being reported. – Adam Johns Jan 28 '15 at 15:27
  • @AdamJohns - You never know for sure unless you try and test it. There are better ways, so this would almost be a last resort for me. – JeffO Jan 28 '15 at 16:56

Coming from a world where developers are expected to write and run the tests themselves, this "testing" "QA" silo you are referring to frightens and confuses me, so I'll try to answer from this perspective. As an aside, qualified QA engineers, from my perspective, (as is described well in @SparK's answer), should focus on the larger issues of making sure that the software fully satisfies the user stories and has overall "quality" (in regard to the domain the software is intended for), instead of hunting for bugs.

What drew me here is @JamesMcleod's mention of "defect injection" in the comments to the question. I actually think that having the developers think how they could inject bugs into the system is a great idea for targeting the concept of defense in depth. No single bug should ever be enough to bring down the entire system in an uncontrolled fashion (without clear actionable logging), cause any data corruption, or by itself expose a security vulnerability.

Having the developers of each component create intentional defects, handle those of other components and overall enter a more adversarial mindset about their software could possibly do much towards improving the robustness of the software. Even the immediate benefit could be significant - I'd require that during each such injection of a new kind of defect (that was hitherto untested), the developer would immediately cover it by a new test, which will be set with a flag that will allow the bug to live in the code base undisturbed for a short while, and then switched on before delivery (and the defect removed), to turn into a regular test that will make the test suite more comprehensive.

A related option is the use of feature flags to intentionally turn off features in particular components to examine how other components deal with that. I would also like to highly recommend reading the free book/article "Learning from First Responders: When Your Systems Have to Work" which describes such extensive testing of the software infrastructure to be used by Obama team for the 2012 election.

  • 4
    Rather than having developers "inject" bugs in the code, their time would be far better served identifying how those bugs could have gotten in the system to begin with and then correct the code to ensure those bugs can't happen or are handled properly by the software. The goal of developing a project is not to test the QA system, it is to build a usable robust and working system that does what its users want it to do. – Dunk Jan 29 '15 at 16:33

As others have already said, it's not the job of QA to solely find bugs. I would go further and say it's not their job at all, technically. Developers should be responsible for keeping their own code bug-free. Test suites should be run before new code is ever even committed, and if the test-suites fail, then it should never be making it to QA in the first place. Introducing bugs intentionally means you definitely can't pass your test suites, so why is your code going to QA?

The job of QA is to validate the application against the user stories it implements. They should test the flow, the UI, etc. and ensure that the user can do everything the user should be able to do, in the most usable and accessible way as possible. While doing this, of course, they may stumble upon bugs, but that's a side-effect of what they do, not what they do. Remember QA means Quality Assurance, not Bug-free Assurance.

This isn't necessarily as crazy as it sounds. It rather depends on your motivation. If you are looking for a stick to beat your test team with, well that would be crazy. On the other hand, one of the most difficult things in software development is to know how effective your testing approach is.

So if you structure it properly, you could use this technique to estimate how many unfound bugs remain in the product you are about to ship. So imagine that you've artificially seeded 100 bugs in your test build, and the testers find 50 of them. Then you can infer that there's a certain likelihood that if they also found 50 unseeded bugs, perhaps there are 50 left to find.

Of course, this is fraught with many problems. You could decide whether to ship based on these statistics, but in real life, you might find one very nasty issue, or a thousand minor irritations.

Still - knowledge is power, and without this technique, you have even less idea of the quality of your code base. If you can implement it respectfully and for the right reasons, I'd say "Why not?"

One thing nobody else has mentioned yet: mutation testing.

This is where an automated tool takes your source code and deliberately inserts bugs into it. (E.g., delete a randomly-chosen statement, change an AND to an OR, or whatever.) It then runs your full test suite, and checks whether the tests pass.

If all tests pass, then there are two possibilities:

  • The thing that was changed doesn't do anything. In other words, you have dead code.
  • The change introduced a bug which your test suite isn't catching. You need more tests.

Note that, unlike your proposal, everything I've described above is automated. You're not wasting developers' time inserting pointless bugs by hand. And you're not wasting testers' time finding known bugs. The only thing you're using up is machine time, which is much cheaper. (Machines don't get bored of doing the same test 20,000 times. Humans stop caring after a while!)

I would suggest that automated mutation testing is a far, far better approach than the manual scenario you're talking about.

Note that if you ask a developer to manually insert bugs, the kind of bug you get is probably not representative of the kind of accidental mistakes humans might make. (E.g., if you haven't realised there's a possible race condition, you're unlikely to insert a deliberate one either.) Whether an automated tool manages to be more objective remains to be seen, of course…

Although it is a bad idea in general (the other answers perfectly explain why), there is a few special situation where intentionally injecting bugs into the production code in a controlled, temporary manner can make sense.

When you refactor the test code - and you should, test code deserves the same attention to detail as production code - you might want to know whether the test code is still finding the bugs it's supposed to find.

You can then intentionally break the production code in order to verify if the tests still work.

There are multiple levels on which this is possible:

  • A developer that has just refactored some unit test might break the production code to verify that the unit test still finds what it's supposed to find.
  • A tester that has just refactored some acceptance test might break the production code to verify that the acceptance test still verifies what it's supposed to verify.
  • If the interface is stable and robust enough (i.e. protocol based), the company might want to keep a suite of known faulty product versions and run tests against them in order to regression test the test.

Whether these things make sense depends. If I'm a developer and it takes me just a minute to inject a bug, test the unit test, remove the bug - then why not. But I should have my editor, my cycle and my version control system under such a good control that I would not accidentally commit / deliver / check-in / push the bug. Same goes for the tester and the acceptance test.

Whether it makes sense for an organization to keep suites of known faulty product versions and regression test the test depends. For an online shop I wouldn't. For automotive embedded, aerospace embedded, banking cards or pay TV cards I would.

How much effort this is strongly depends on how decoupled the tests are from the production code. The more decoupled the tests are from the production code, the less effort it is to do this, the more cohesive the tests are with the production code, the more effort.

The reason is simply this: When your tests and your production code are cohesive, changing the production code requires changing the tests often, and that would break the dependency between the tests and the faulty production samples. You would then have to also maintain the faulty production samples. In rare cases even that can be worth the effort, and mocking as well as smart usage of a version control system can significantly reduce the effort, but it requires far above skilled developers.

The concept of intentionally injecting faults in production code is called sabotage, the injected fault is called saboteur.

A tester who is not taking the code-to-be-tested directly from the repository is doing it wrong. (1)

A developer who is checking in known-faulty code into the repository is doing it wrong. (2)


So at this stage, there is already no way for this scheme to work without one or both sides violating very basic premises of how development and testing ought to be done.


(1) Because you need to document which version you tested. A version tagged by a Git hash or a SVN revision number is something you can test, "the code Joe gave me" isn't.

(2) Because you just don't, outside of a test driver that is expecting failure.


This is an attempt at a shortest-possible, "elevator pitch" reason that should make immediate sense to devs, testers, and management alike.

  • 1
    This is a circular argument. You are saying "seeding bugs in a test build is wrong because developers shouldn't make a build with known faulty code". – Dominic Cronin Feb 5 '15 at 11:34
  • @DominicCronin: Nothing circular about it. Whatever gets commited to the repository should be the best possible quality. There's a whole bouquet of reasons there -- avoiding artificial change of code lines is one (w.r.t. "svn blame" and similar repository functions). The danger of "forgetting" to take the error out again. The problem that the testers basically could look up what has been "seeded" by looking at the repository change log. Many more reasons, with virtually no benefit to counterbalance. But I'm running out of space, and anyway the idea was to provide one, short reason. – DevSolar Feb 5 '15 at 11:43
  • @DominicCronin: Or, to put it differently -- there might be a case to be made for "seeding" a bug, but the line has to be drawn well before committing that to the repo. And on the other hand, while having "seeded" code for testing might have one or two things going for it, you should only ever test committed code. The two ideas -- each being already controversial on its own -- simply don't connect in any sensible way. – DevSolar Feb 5 '15 at 11:52

I recommend against deliberately injecting bugs into EVERY build you send to QA.

You could, from time to time, let's say once a year, do a covert "QA audit". Take a "tested and working" codebase, and as many small new features from your Todo list as possible. Implement them "a bit more sloppily" than you usually do. Think of the edge cases, write them down, but don't fix your code to take them into account. Send it off to QA.

If they find more non-working edge case bugs than you wrote down, it's certainly not your QA that needs supervision... ;-)

  • 2
    this doesn't seem to offer anything substantial over points made and explained in prior 16 answers – gnat Jan 30 '15 at 12:48

protected by maple_shaft Jan 28 '15 at 16:07

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.