Is creating a completely duplicate system for quality assurance (QA) of another a bad practice?

Question

At work we have a quite complicated system. Let's call this system, System_A. Our QA team has created another system, call this system, System_B, to test System_A.

The way System_B is used is as follows. We generate inputs (using System_B itself), IN, process such inputs back through System_B and generate outputs, O_B. So the process is as follows:

System_B(IN) -> O_B.

We then do the same for System_A to generate its own outputs, O_A:

System_A(IN) -> O_A.

At any time, it is assumed that O_B is the expected output, and O_A is the observed/actual output. Implied is that O_B is the "gold" source (the truth). However, we have run into a combinations of problems.

• O_A is wrong, O_B is right
• O_A is right, O_B is right
• O_A is wrong, O_B is wrong
• O_A is right, O_B is wrong

Who determines what's right if O_B is assumed to be always right (or what's expected)? Well, it turns out that O_B is sometimes (or often) wrong with human inspection and analyzation. Things will pass QA using this process, and real users will complain, and we go back to finding that O_B was wrong after all.

The question is this: is it a bad practice to create a "test system" to test the real system?

• What about the slippery slope? Then, can't we argue we need yet another system to test the "test system"?
• The cost is definitely prohibitive, as developers now need to learn at least 2 code bases, with perhaps System_B's complexity larger than System_A. How would we quantify how good or bad having System_B around is to the organization?
• One of the original "compelling" reasons to create System_B was to "automate" testing. Now we are very proud that we are fully automated (because System_B generates the input to bootstrap the process of using itself to also generate the output). But I think we've done more harm and introduced more complexity, in an unquantifiable way. Is the job of QA to be fully automated? Is that reason enough to justify to create a parallel system?
• My real concern is this, even though we all know System_B is wrong (quite often). If System_B is so good at processing the input and its output is the gold source, why not replace System_A with System_B? To that, no one at work is able to provide a satisfactory response.

Any guidance on this issue is appreciated.

Answer 1

• O_A is wrong, O_B is right

Fix A

• O_A is right, O_B is wrong

Fix B

• O_A is right, O_B is right

Hopefully, they also agree.

• O_A is wrong, O_B is wrong

Hopefully, they don't agree so you'll know to do something about it.

No process catches everything. Sure, you've doubled your code but think of it like the 2 in O(2n). Automated QA all the way to the integration tests is a wonderful thing. Manual testing is a drag on inovation. Especially on cross cutting changes that would demand a full test. Also, since you'll be having different programmers implement the same thing, you can have them race.

It should be different people to increse the odds of getting different bugs. I don't advise creating system B by coping from system A. All that gives you is a regresion test. You could get the same thing simply by saving old copies of O_A to compare with new ones.

The question is this: is it a bad practice to create a "test system" to test the real system?

If it is, then all testing is bad.

• What about the slippery slope? Then, can't we argue we need yet another system to test the "test system"?

Yes, we can argue that. We will call this 3rd system, system_A. :P

• The cost is definitely prohibitive, as developers now need to learn at least 2 code bases, with perhaps System_B's complexity larger than System_A. How would we quantify how good or bad having System_B around is to the organization?

By the number of happy customers that pay us to play with nerf guns. You've freed your self of the cost of manual testing. You've created something whose usefulness will be proven every time a bug is caught by it. Bug's caught early cost far less than bugs reported late.

• One of the original "compelling" reasons to create System_B was to "automate" testing. Now we are very proud that we are fully automated (because System_B generates the input to bootstrap the process of using itself to also generate the output). But I think we've done more harm and introduced more complexity, in an unquantifiable way. Is the job of QA to be fully automated? Is that reason enough to justify to create a parallel system?

The complexity of System_B is wonderfully isolated from System_A. It is not any harder to add features to System_A because System_B exists. It is infact easier because System_B gives them the confidence to go fast.

• My real concern is this, even though we all know System_B is wrong (quite often). If System_B is so good at processing the input and its output is the gold source, why not replace System_A with System_B? To that, no one at work is able to provide a satisfactory response.

Is this a typo? System_B is quite often wrong so it's the gold standard you want to use to replace System_A?

I'm going to assume you meant System_A is often wrong. Doesn't really matter which one is wrong most often. Whichever one is wrong is the one that needs work. These systems don't decide right and wrong, developers do. What the testing does is produce a disagrement that means something is not right. Developers figure out what that is. Remember, there is no gold standard being produced here. There is only agreement or disagreement. Disagreement demands that work needs to done. Part of that work is figuring out where.

Answer 2

When you have a system in production that is actually used by customers, having a QA system to verify bugfixes and new functionality is an absolute must. From the quality standpoint, it should be as close a replica of the production system as possible. That way, if you ensure that the production system and its QA system are identical, what works on one, should work on the other. If it is not the case, then the systems are not identical, the inputs were not identical, and/or the outputs were misinterpreted.

This goes doubly so if your system is mission critical and is required to be available 24/7. You then cannot affort the luxury to not have a QA system, because you must absolutely minimize the downtime on the production system. And if it is a 24/7 system, then the exact replica of the production system is a very, very strong recommendation.

Now, the obvious drawback of this approach is the cost. It doubles the hardware costs, and increases deployment and maintenance costs. Plus, a continuous replication of data from the production system to QA would have to be implemented, so that we would minimize the possibility of different results due to the difference in the data that the systems work with.

Some balance usually can be found by downsizing some of the components of the QA system relative to the production system, so that most of the functionality can be properly tested. Those are usually redundant servers, size of servers or number of workstations. However, it is my experience that some bug is always found exactly in the part that was downsized, and then it is a nightmare to reproduce the problem and implement the fix while maintaining minimum allowed downtime in the production system.

Answer 3

Any time you test a system you have to have to know what your expected outcome is.

The problem with having a system generate this expected outcome is obviously 'how do i test that system'

Even so its not usual to see people use spreadsheets for example to generate sets of expected outcomes.

At the end of the day though you really need a human to interpret the requirements of the system and manually produce the expected result. If you have a system do it and only check the differences then you are skipping your testing.