At work we have a quite complicated system. Let's call this system, System_A. Our QA team has created another system, call this system, System_B, to test System_A.
The way System_B is used is as follows. We generate inputs (using System_B itself), IN, process such inputs back through System_B and generate outputs, O_B. So the process is as follows:
System_B(IN) -> O_B.
We then do the same for System_A to generate its own outputs, O_A:
System_A(IN) -> O_A.
At any time, it is assumed that O_B is the expected output, and O_A is the observed/actual output. Implied is that O_B is the "gold" source (the truth). However, we have run into a combinations of problems.
- O_A is wrong, O_B is right
- O_A is right, O_B is right
- O_A is wrong, O_B is wrong
- O_A is right, O_B is wrong
Who determines what's right if O_B is assumed to be always right (or what's expected)? Well, it turns out that O_B is sometimes (or often) wrong with human inspection and analyzation. Things will pass QA using this process, and real users will complain, and we go back to finding that O_B was wrong after all.
The question is this: is it a bad practice to create a "test system" to test the real system?
- What about the slippery slope? Then, can't we argue we need yet another system to test the "test system"?
- The cost is definitely prohibitive, as developers now need to learn at least 2 code bases, with perhaps System_B's complexity larger than System_A. How would we quantify how good or bad having System_B around is to the organization?
- One of the original "compelling" reasons to create System_B was to "automate" testing. Now we are very proud that we are fully automated (because System_B generates the input to bootstrap the process of using itself to also generate the output). But I think we've done more harm and introduced more complexity, in an unquantifiable way. Is the job of QA to be fully automated? Is that reason enough to justify to create a parallel system?
- My real concern is this, even though we all know System_B is wrong (quite often). If System_B is so good at processing the input and its output is the gold source, why not replace System_A with System_B? To that, no one at work is able to provide a satisfactory response.
Any guidance on this issue is appreciated.