Should we test a code version against another

Question

I am required to write tests for an API at work to make sure we don't break it as we develop on it. So far, the solutions that cam out are as follow :

• "prod" code and "devel" code running on the same DB (with a dump of the production database);
• "prod" code and "devel" code running on their own DB.

Both solutions require a stand-alone application to make a diff of the results given by both codes. Hence, I'm also required to write something allowing to make diffs between JSON, XML or any other format used by the API.

According to me, this is over-thinking. I think we would be better of writing simple functional tests that successfully pass which will be included in the API code project. As we develop upon, we would simply run the tests and fix what failed passing as it previously did.

My solution had been dismissed by a coworker as being not right not efficient for a few given reasons :

• we need real datas;
• creating a dataset for tests is impossible;
• we don't want to rewrite tests every time we change something;
• we need to generate thousands of urls randomly;
• although I don't know what PHPUnit can or cannot do, I think something written in bash will be more optimized, or Python to get advantage of multi-threading (!?).

To me, this looks like complete non-sense and totally misses the purpose of testing. Firstly, our database schema is rather simply : a score of denormalized tables with a handful of columns, lots of them are of the boolean type. Thus, writing a relevant dataset looks quite simple to me.

Secondly, I think (I may be wrong) that testing is about determining what your code is supposed to do in any possible circumstances. There's no need to write more than a few "hard coded" (not randomly generated at least) urls that will test each test one of those cases.

Thirdly, rewriting tests as the behavior of the code change IS required and a benefit by itself, not something to dodge because it appears to be something natural.

Finally, my thoughts are also driven by the fact that as far as the self-taught that I am knows, I never heard about the kind of test they want me to write. To me, this is a clue that we are talking about something really specific that require an even more specific solution (but our API is rather simple and mostly a read-only), or that we're getting damn wrong. My coworker which is the most trusted by my superior doesn't seem well versed into testing... I ever get rebuked for trying to make clear that what they call "unit-testing" actually is something between integration and functional testing with a huge smoke flavor (yeah, really), and that we should make clear what we want to test for which purpose, or that we're not supposed to test private methods. He replied with anger that his tests work and that I am getting "too much theoric" and shouldn't listen to "people that love portmanteau words" because it's useless... That said, I simply don't trust in him, on this point at least !

Does anyone ever had to write such tests which really were the best solutions ? Why was it the best one ? Should I try to make my point of view clearer and / or catch my superior to "force" trying my "basic solution" first ?

Thank you,

Answer 1

The kind of testing you describe definitely does exist. It is a special form of regression testing, and as such makes sure that your devel version behaves identical to the prod version, i.e. that you don't introduce any backwards incompatibilities.

It is most often used when doing rewrites or clones of existing software, to ensure that the behaviors are a perfect match. For example, the Samba team has a variation of this: the test machine has two network cards, one network card is connected to a Windows server, the other network card is connected to a Samba server. The test machine sends random identical commands to both machines and compares the answers. Whenever it detects a difference between the responses, it starts a backtracking search to try and find the simplest sequence of commands that still generates the different responses, and then logs that sequence together with the two responses in a database for a human to take a look at.

So, the typical use case for this kind of testing is when you don't actually know what the version you are testing against does, either because it is a poorly documented legacy system (in the rewrite case) or because it belongs to someone else (in the Samba trying to be compatible with Windows case).

The Merb framework used acceptance tests to define the public APIs (i.e. the APIs that applications can rely on) and semi-public APIs (i.e. the APIs the Merb internals can rely on) and distinguish them from private APIs (that are allowed to change without notice). These tests were separated into different directories, and there were clear rules what you are allowed to do with those tests during a release cycle. For example, you are not allowed to delete or modify tests in the public directory, only add new ones unless doing a major release. (Or, interpreted the other way around: modifying or deleting a test in the public directory is a backwards-incompatible change in the public API and this requires a new major version.)

Something like this sounds more applicable to your situation: a set of acceptance tests describing the public API for your customers, a set of acceptance tests describing the internal APIs for your developers, and then of course functional tests for individual slices of functionality, unit tests for individual units of behavior, integration tests for individual points of communication between units of behavior.

The public API tests can also be used as regression tests. When you release a new version, you can run the tests of the old version against the new version, to see whether you have broken anything, and what you have broken, so you can either document it (if it is an intended incompatibility) or fix it.