Leaving intentional bugs in code for testers to find

Question

We don't do this at our firm, but one of my friends says that his project manager asked every developer to add intentional bugs just before the product goes to QA. This is how it works:

1. Just before the product goes to QA, the development team adds some intentional bugs at random places in the code. They properly back up the original, working code to make sure that those bugs aren't shipped with the end product.
2. Testers are also informed about this. So they will test hard, because they know there are bugs present and that not finding them might be considered as a sign of incompetence.
3. If a bug (intentional or otherwise) has been found, they will be reported for the development team to fix. The development team then adds another intentional bug in a related section of the code just before the product goes to the second-level QA. The project manager says a tester should think like a developer and he/she should expect new bugs in sections where changes were made.

Well, this is how it goes. They say that this approach has following advantages.

1. Testers will be always on their toes and they will test like crazy. That helps them to also find hidden (unintentional) bugs so developers can fix them.
2. Testers feed on bugs. Not finding any bugs will affect their morale. So giving them an easy one to find will help their morale.

If you ignore the scenario where one of these intentional bugs gets shipped with the final product, what are the other drawbacks we should consider before even thinking of adopting this approach?

Some clarifications:

1. They properly backup the original code in source control.
2. When a tester finds the intentional bug, the development team just ignores it. If tester finds out an unintentional (original) bug, the development team first checks whether it is caused by any of the intentional bugs. That is, the development team first tries to reproduce that on the original working code and tries to fix it if they can.
3. Just ignore the relationship issues between QA and development team. I specifically asked this question on Programmers, not on The Workplace. Consider that there is good rapport between QA and the development team, and they party together after working hours. The project manager is a nice, old gentleman who is always ready to support both teams (Godsend).

Answer 1

This sounds absolutely ridiculous. It is expending a great deal of effort for very questionable benefit, and the practice seems based on some faulty premises:

• That QA won't work hard unless they know they are being tested every day (which cannot be good for morale)

• That there are not enough unintentionally introduced bugs in the software for QA to find

• That QA's job is to find bugs - it isn't; it is to ensure the software is production quality

• That this kind of battle of wits between development and QA is in some way healthy for the company - it isn't; all employees should be working together against the company's competitors instead of each other.

It's a terrible idea and the project manager in question is a jerk/idiot who understands nothing about people and motivation. And it's bad for business.

---

To expand on my description of "QA's job:" QA definitely should be finding bugs - both in the code and in their test suites - as an artifact of doing their jobs, but the role shouldn't be defined as "you have to find bugs." It should be "you have to keep the test suites up-to-date to account for new features and ensure all high coverage of testing. If this does not result in finding bugs, then the testing procedures are not sufficiently sophisticated for the product.

Answer 2

Well, based on what I've learned:

1. It's not a school nor job interview;
2. The testers are not children;
3. It's not a game;
4. It wastes company's money.

The QA are not there only to find bugs but also to worry about how intuitive the system is, what is the learning curve for the user, usability, and accessibility in general. For example: "Is the system ugly?", "Is the user color blind and stuff is red and green?" They should complain too.

The minimal requirements for a system to pass QA is usually described in a user story for that particular feature or in how magical the PO wanted the system to be in his head.

tl;dr

It's not only bugs, testers should grow out of this narrow view.

Answer 3

Bad idea.

From the tester's point of view: "So they will test hard, because they know there are bugs present and not finding them might be considered as their incompetence." Basically the devs are booby-trapping the code. Few people like doing work which is ultimately pointless (because the bugs are known in advance) but which still affect how they are perceived. If there are tangible punishments for not finding the booby-traps, more so. And do you know testers thrive on finding bugs? That sounds like a toxic confrontational environment; a QA should be happy if the code they are examining is high quality. Although if they are paid by the bug... http://thedailywtf.com/articles/The-Defect-Black-Market

From the dev's point of view: the QAs are being incentivised to find the bugs you know are there. That may well increase the likelihood of real bugs going out of the door; the QAs are spending at least some of their time looking for the sort of bug that is easy to plant, not really subtle ones. Plus there is a small chance that a booby-trap may make it out of the door.

Answer 4

I agree totally with the answers above as to why this is bad for motivation and just generally awful people management. However, there are probably sound technical reasons for not doing this as well:

Just before the product goes to QA, the dev team adds some intentional bugs at random places in the code. They properly back up the original, working code to make sure that those bugs aren't shipped with the end product.

1. Based on the first statement, you never actually test your intended production code in these two passes.

2. I would imagine you vastly increase the likelihood of accidentally including an 'intentional' bug into your released production code when trying to rush through a change for a customer. Might cause a few red cheeks at some point.

3. I would also think that this tends to train your testers to think like your developers (i.e. how would Tom add a bug here) which probably makes them less likely to find the bugs that Tom hasn't thought about.

Answer 5

Edit

I want to be clear that this answer is only talking about the concept of testing your QA process, and I'm not defending the specific methodology portrayed in the question.

End Edit

There is a valid reason to check if your testing/checking is actually working. Let me give you an example from manufacturing, but the principle is the same.

It's typical when feeding material through a machine that the feeder might not push the material through far enough. This is called a "short feed" and to prevent this we might install a "short feed sensor" (typically a through-beam type sensor that's blocked by the material). This sensor detects the end of the material when it reaches the full feed length. At a certain point in the machine cycle we check that the sensor is blocked, and stop the machine if the check fails.

Now you have to think about how the test itself can fail. For instance, some dirt or other debris can block the sensor and it will always report "OK" and never stop the machine. Also, the nature of the sensor is that the receiver turns on when the beam hits it, so depending on the type of sensor you have installed, electrically you get an "ON" input when the sensor is not blocked. That means if the cable got cut or power was lost to that sensor, or the input failed, the logic of your program would read "OFF" and that would mean "blocked" or "OK".

To catch these failure modes of the test, we typically insert a second check to make sure the sensor is actually unblocked during a second part of the cycle. In this way we check that the test is actually still operating (as best we can).

Similarly there are many ways that a QA department can fail. Perhaps the automated tests didn't run and the report is looking at an old copy of the test data. Perhaps someone isn't doing their job right. Testing the QA department is a reasonable thing to do.

Obviously the draw-back is that a "test bug" could make it through the QA department and into the finished product. In the manufacturing industry there are sometimes cases where a known bad part, sometimes called a "Red Rabbit", is inserted into the process (typically by someone from QA) and they watch that part go through the process and measure how long it takes to find the part and remove it. Normally this part is painted bright red (or orange) so it can easily be tracked. Since someone is watching the part go through process during this test, the chance of it making it into the final product is practically nil. There are, of course, apocryphal stories of someone throwing a known bad part into the process to "see if the system can find it", and of course having to quarantine all the final parts produced that day and manually sorting them, but that's just a case of not performing your test with due diligence.

Answer 6

Honestly, I'd call this behavior blatantly unethical and impractical. The PM is in need of some serious retraining, if not termination.

• It demonstrates a fundamental lack of understanding of the concept of quality assurance. Testers should not think like developers: they should think like end users. The entire reason for having QA teams is that developers are inherently too close to the code; QA is supposed to maintain enough distance from the code that they can catch what the devs miss.
• It wastes QA effort. Assuming that these bugs are not trivial -see below for when they are- it means that QA is spending time and resources investigating things that are already known, when they could be spending that effort seeking out what isn't known.
• It wastes developer effort. For QA folks to catch these non-trivial bugs, developers must first write them. This requires yet further effort, spent not just coding the bugs, but also considering the software requirements and design.
• It puts production at needless risk. It is only a matter of time before changes don't get merged in properly.
• If it doesn't do the above, then it is pointless. If all of the known bugs are trivial, then they won't catch substandard workers: they will only catch people who are doing no work at all. There are better ways to do that.
• It poisons the work environment. Your QA testers are professionals. They should be trusted to be professional until there is actual reason to suspect otherwise. When there is reason to suspect otherwise, there should be a proper investigation instead of these mind games. Anything else kills morale.

Seriously. Even if the PM's paranoia turns out to be well-founded in this specific case, this is not someone who has any business managing testers.

Answer 7

Personally, I feel uncomfortable with this approach.

The main thing that concerns me is the practicality of inserting intentional bugs. This seems to me to be difficult to do in any way that is predictable.

Any code changes (intentional or otherwise) risk having side-effects. These side-effects may well be revealed during testing but it may not be obvious (even to the developer who planted the bug) what the root cause is. It doesn't feel "safe", if you know what I mean (I'm speaking from my gut here).

Also, the tester will waste a lot of time testing code that isn't actually going to be released. Once the intentional bugs are removed, a complete re-test should be done anyway, in my opinion. That's the whole point of testing. Something changes, anything, and you re-test everything. Ok I know that doesn't ever happen in practice, but that's what regression testing is all about.

So, overall, not convinced.

On the other hand, we tend to let the customers verify the work of the QA teams, which is possibly not ideal. It is a very powerful feedback loop though.

Answer 8

It's a bad idea for all the reasons already given, but bug seeding is a useful tool for a different purpose. You can use it to get a rough metric of how effective the QA process is.

In its simplest case, let's say you seed 100 bugs and they're representative of the full span of real bugs (I know, unlikely, but I'm simplifying). You do not tell QA you're doing this to avoid spoiling the experiment. At the end of the QA process let's say they found 60 of the 100 seeded bugs (and other real bugs). Now you know QA is finding 60% of the bugs.

You can extend this further by counting the number of real bugs QA found and apply the fake bug ratio. In our example, if QA found 200 real bugs then you can conclude they only found 60% of them, so 133 remain.

Of course, this is just a broad estimate with huge error bars. Writing realistic, representative bugs is hard. The bugs you write are likely to be easier for QA to find because developers are trained to not write bugs. It may be better to simulate a class of bugs such as off-by-one errors, Unicode mistakes, buffer overflows, and so on.

This should be applied to the entire QA process which would include developer unit testing, continuous integration, and, if available, a dedicated QA team.

This is a metric, and should not be hijacked as a management motivational tool.

Answer 9

Bad idea.

This is the sort of logical, binary approach that developers often bring, but it is demotivating for the QEs. It simply demonstrates a lack of trust. QEs often get placed in these situations without much input from them, and it assumed that they are OK with it, and it is not their place to suggest otherwise.

This kind of thinking combines to QEs being manual testers only and not being motivated to understand the actual code under test.

I am a senior QE and this is a familiar issue in most organizations I have worked in.

Answer 10

I'd say bad idea.

One: Programmers are going to spend time putting deliberate bugs in the code, and some effort to save the good version. While the testers should presumably be testing everything, including the features with the planted bug, when they find one they will presumably have to go back and re-run that test to verify that this was indeed a bug (and not that the tester got confused in some way). At a minimum, testers are going to spend time writing up the planted bugs. Then the programmers have to spend time fixing the bug that they planted. This is a lot of effort that could be spent trying to write good code and write up real bugs.

Two: It sends a clear message to the testers that the programmers and/or management think they are not doing their jobs and must be treated as children. I can't imagine that this is good for morale. As a programmer, if I was given ambiguous or contradictory specs for a program and had to spend a bunch of time getting them clarified, and then after wasting hours or days my boss told me, "Oh, yeah, I deliberately put contradictory statements in the specs just to make sure you were really reading them", I think I would be really annoyed. If that happened regularly, that might well be enough to make me look for another job.

In real life, all but the most trivial code changes WILL have bugs. I have never had a problem with testers getting complacent because the first draft code they were given was so often 100% perfect. I've had to deal with lazy testers who don't do an adequate job, but they didn't get that way because the programmers were so perfect. The best testing person I ever worked with once told me that for a new software release, he set himself a personal goal to find 100 bugs. Okay, whether 100 is a realistic number depends on how big the product is and how extensive the changes are, but in our case, he almost always managed to meet that goal. Sometimes he had to stretch things, like calling a mis-spelled word in a message a "bug", but hey, it did have to be fixed.

Post script: If you do this, I bet that sooner or later the programmers are going to deliberately plant a bug, the testers don't find that particular one, and the programmers forget to put the good code back. So now a deliberately planted bug gets shipped to the customer.

Answer 11

I don't really think this a bad idea. There's just a lot of things that I would speculate work better:

1. Make QA accountable for the quality any way you can. For example by making support their responsibility also. This will increase their motivation to make sure the products shipped have higher quality. It always takes less effort to discover an inadequacy (bug, obviously missing feature, counter-intuitive behavior) yourself then to try to understand what your upset user is trying to explain. And putting some of that responsibility even on developers might increase their motivation to help the QA do their job the best they can.

2. Have multiple QA teams, which can the compete. You need to find a sensible metric of course. Definitely not just the number of issues. Factoring in the severity of the defect or the business value (as determined by stakeholders) of proposed enhancements should help.

It's hard to tell whether QA is "good enough". It's easier and possibly even better in the long run to find ways for QA to be "ever improving".

Still, there's one issue to be aware of if you introduce intentional bugs: How do you know the "correct" code ever actually was correct in the first place? After the 2nd QA you remove all intentional bugs that hadn't been discovered. There's no way of knowing that you aren't either just replacing them by code that is broken in a different way or that you are not enabling broken behavior that was unreachable before (exaggerated example: some dialog did not open because of an intentional bug, but the dialog itself is broken - you just don't find out because testers didn't get to see it).

Answer 12

As others have said, the developers should not be purposefully adding bugs in the software, but it is a legitimate strategy for your test suite to add bugs into the software as part of the testing process.

It's called mutation testing. The idea is to use software to automate the creation of small changes in the source code (called mutants). The changes are designed to create different behavior, for example, we could change

if x < 10:
    print "X is small!"

into

# we flipped the inequality operator
if x > 10:
    print "X is small!"

and a good unit test should detect that the mutant code fragment no longer works as expected and kills the mutant. When the original code passes the test, and all mutants (that don't happen to be functionally equivalent) fail the test, then you know that your code and your tests are strong.

Answer 13

I like the idea. Was it General Patton who said, "The more you sweat in peace, the less you bleed in war."

Putting intentional bugs "wastes time" of the testers. But that also makes them work harder, meaning that they will also do a better job of finding UNintentional bugs. (And you have a copy of the "original" so you don't have to live with what you've done.)

Finding more unintentional bugs will likely save you more grief in the long run that the cost of dealing with the intentional ones.

Plus, you can get an idea of how good your testers are, not a small benefit in itself.

Answer 14

There is no basis for a reward or punishment on its own merit, but on the result of the behavior you're targeting. And sometimes there are unintended consequences. Is the goal to keep the QA team from slacking or to make some manager feel like he's actually contributing something without realizing he's just getting in the way.

Positive Outcome - The QA team works harder to find bugs. Who knows, maybe they see this as a challenge. It's a friendly game. Or they're just doing because they're being watched (Hawthorne Effect?).

Negative Outcome - They may not work harder and find the bug anyway. QA sees this as petty and adversarial. So now, they go into hyper-bug finding drive and return all sorts of nit-picky little problems. That font doesn't render properly when I take a screen shot and convert it to a pdf and view it at 500%.

No Impact - sound to me like this makes no difference, so why bother? You just risk wasting time and irritating people.

We could all agree that this won't work 90% of the time. That doesn't do much good to the other 10%. Test things for yourself. Are customers more satisfied with a release that has the intentional code bugs? Does it impact worker morale and productivity in other areas? Increase turnover? You tell us.

Answer 15

Coming from a world where developers are expected to write and run the tests themselves, this "testing" "QA" silo you are referring to frightens and confuses me, so I'll try to answer from this perspective. As an aside, qualified QA engineers, from my perspective, (as is described well in @SparK's answer), should focus on the larger issues of making sure that the software fully satisfies the user stories and has overall "quality" (in regard to the domain the software is intended for), instead of hunting for bugs.

What drew me here is @JamesMcleod's mention of "defect injection" in the comments to the question. I actually think that having the developers think how they could inject bugs into the system is a great idea for targeting the concept of defense in depth. No single bug should ever be enough to bring down the entire system in an uncontrolled fashion (without clear actionable logging), cause any data corruption, or by itself expose a security vulnerability.

Having the developers of each component create intentional defects, handle those of other components and overall enter a more adversarial mindset about their software could possibly do much towards improving the robustness of the software. Even the immediate benefit could be significant - I'd require that during each such injection of a new kind of defect (that was hitherto untested), the developer would immediately cover it by a new test, which will be set with a flag that will allow the bug to live in the code base undisturbed for a short while, and then switched on before delivery (and the defect removed), to turn into a regular test that will make the test suite more comprehensive.

A related option is the use of feature flags to intentionally turn off features in particular components to examine how other components deal with that. I would also like to highly recommend reading the free book/article "Learning from First Responders: When Your Systems Have to Work" which describes such extensive testing of the software infrastructure to be used by Obama team for the 2012 election.

Answer 16

One thing nobody else has mentioned yet: mutation testing.

This is where an automated tool takes your source code and deliberately inserts bugs into it. (E.g., delete a randomly-chosen statement, change an AND to an OR, or whatever.) It then runs your full test suite, and checks whether the tests pass.

If all tests pass, then there are two possibilities:

• The thing that was changed doesn't do anything. In other words, you have dead code.
• The change introduced a bug which your test suite isn't catching. You need more tests.

Note that, unlike your proposal, everything I've described above is automated. You're not wasting developers' time inserting pointless bugs by hand. And you're not wasting testers' time finding known bugs. The only thing you're using up is machine time, which is much cheaper. (Machines don't get bored of doing the same test 20,000 times. Humans stop caring after a while!)

I would suggest that automated mutation testing is a far, far better approach than the manual scenario you're talking about.

Note that if you ask a developer to manually insert bugs, the kind of bug you get is probably not representative of the kind of accidental mistakes humans might make. (E.g., if you haven't realised there's a possible race condition, you're unlikely to insert a deliberate one either.) Whether an automated tool manages to be more objective remains to be seen, of course…

Answer 17

As others have already said, it's not the job of QA to solely find bugs. I would go further and say it's not their job at all, technically. Developers should be responsible for keeping their own code bug-free. Test suites should be run before new code is ever even committed, and if the test-suites fail, then it should never be making it to QA in the first place. Introducing bugs intentionally means you definitely can't pass your test suites, so why is your code going to QA?

The job of QA is to validate the application against the user stories it implements. They should test the flow, the UI, etc. and ensure that the user can do everything the user should be able to do, in the most usable and accessible way as possible. While doing this, of course, they may stumble upon bugs, but that's a side-effect of what they do, not what they do. Remember QA means Quality Assurance, not Bug-free Assurance.

Answer 18

This isn't necessarily as crazy as it sounds. It rather depends on your motivation. If you are looking for a stick to beat your test team with, well that would be crazy. On the other hand, one of the most difficult things in software development is to know how effective your testing approach is.

So if you structure it properly, you could use this technique to estimate how many unfound bugs remain in the product you are about to ship. So imagine that you've artificially seeded 100 bugs in your test build, and the testers find 50 of them. Then you can infer that there's a certain likelihood that if they also found 50 unseeded bugs, perhaps there are 50 left to find.

Of course, this is fraught with many problems. You could decide whether to ship based on these statistics, but in real life, you might find one very nasty issue, or a thousand minor irritations.

Still - knowledge is power, and without this technique, you have even less idea of the quality of your code base. If you can implement it respectfully and for the right reasons, I'd say "Why not?"

Answer 19

Although it is a bad idea in general (the other answers perfectly explain why), there is a few special situation where intentionally injecting bugs into the production code in a controlled, temporary manner can make sense.

When you refactor the test code - and you should, test code deserves the same attention to detail as production code - you might want to know whether the test code is still finding the bugs it's supposed to find.

You can then intentionally break the production code in order to verify if the tests still work.

There are multiple levels on which this is possible:

• A developer that has just refactored some unit test might break the production code to verify that the unit test still finds what it's supposed to find.
• A tester that has just refactored some acceptance test might break the production code to verify that the acceptance test still verifies what it's supposed to verify.
• If the interface is stable and robust enough (i.e. protocol based), the company might want to keep a suite of known faulty product versions and run tests against them in order to regression test the test.

Whether these things make sense depends. If I'm a developer and it takes me just a minute to inject a bug, test the unit test, remove the bug - then why not. But I should have my editor, my cycle and my version control system under such a good control that I would not accidentally commit / deliver / check-in / push the bug. Same goes for the tester and the acceptance test.

Whether it makes sense for an organization to keep suites of known faulty product versions and regression test the test depends. For an online shop I wouldn't. For automotive embedded, aerospace embedded, banking cards or pay TV cards I would.

How much effort this is strongly depends on how decoupled the tests are from the production code. The more decoupled the tests are from the production code, the less effort it is to do this, the more cohesive the tests are with the production code, the more effort.

The reason is simply this: When your tests and your production code are cohesive, changing the production code requires changing the tests often, and that would break the dependency between the tests and the faulty production samples. You would then have to also maintain the faulty production samples. In rare cases even that can be worth the effort, and mocking as well as smart usage of a version control system can significantly reduce the effort, but it requires far above skilled developers.

The concept of intentionally injecting faults in production code is called sabotage, the injected fault is called saboteur.

Answer 20

A tester who is not taking the code-to-be-tested directly from the repository is doing it wrong. (1)

A developer who is checking in known-faulty code into the repository is doing it wrong. (2)

---

So at this stage, there is already no way for this scheme to work without one or both sides violating very basic premises of how development and testing ought to be done.

---

(1) Because you need to document which version you tested. A version tagged by a Git hash or a SVN revision number is something you can test, "the code Joe gave me" isn't.

(2) Because you just don't, outside of a test driver that is expecting failure.

---

This is an attempt at a shortest-possible, "elevator pitch" reason that should make immediate sense to devs, testers, and management alike.

Answer 21

It’s an awful idea, and here’s the reason: Artificial bugs and “natural” bugs don’t behave the same. With an “artificial” bug, the developer tests for a specific situation and makes the program misbehave in that situation. QA would have to create that specific situation to find the bug. So they have to look not for bugs but for triggers.

Testing requires experience. Over time someone in QA will get a feel how to find bugs. Having to find triggers is also something that can be learned, but it isn’t useful. You could have one person in QA specialising in this and getting really good at finding artificial bugs, and be useless at finding any real bugs. I suppose management might be happy, but it’s just a a waste of time.

Answer 22

I recommend against deliberately injecting bugs into EVERY build you send to QA.

You could, from time to time, let's say once a year, do a covert "QA audit". Take a "tested and working" codebase, and as many small new features from your Todo list as possible. Implement them "a bit more sloppily" than you usually do. Think of the edge cases, write them down, but don't fix your code to take them into account. Send it off to QA.

If they find more non-working edge case bugs than you wrote down, it's certainly not your QA that needs supervision... ;-)