I am required to write tests for an API at work to make sure we don't break it as we develop on it. So far, the solutions that cam out are as follow :
- "prod" code and "devel" code running on the same DB (with a dump of the production database);
- "prod" code and "devel" code running on their own DB.
Both solutions require a stand-alone application to make a diff of the results given by both codes. Hence, I'm also required to write something allowing to make diffs between JSON, XML or any other format used by the API.
According to me, this is over-thinking. I think we would be better of writing simple functional tests that successfully pass which will be included in the API code project. As we develop upon, we would simply run the tests and fix what failed passing as it previously did.
My solution had been dismissed by a coworker as being not right not efficient for a few given reasons :
- we need real datas;
- creating a dataset for tests is impossible;
- we don't want to rewrite tests every time we change something;
- we need to generate thousands of urls randomly;
- although I don't know what PHPUnit can or cannot do, I think something written in bash will be more optimized, or Python to get advantage of multi-threading (!?).
To me, this looks like complete non-sense and totally misses the purpose of testing. Firstly, our database schema is rather simply : a score of denormalized tables with a handful of columns, lots of them are of the boolean type. Thus, writing a relevant dataset looks quite simple to me.
Secondly, I think (I may be wrong) that testing is about determining what your code is supposed to do in any possible circumstances. There's no need to write more than a few "hard coded" (not randomly generated at least) urls that will test each test one of those cases.
Thirdly, rewriting tests as the behavior of the code change IS required and a benefit by itself, not something to dodge because it appears to be something natural.
Finally, my thoughts are also driven by the fact that as far as the self-taught that I am knows, I never heard about the kind of test they want me to write. To me, this is a clue that we are talking about something really specific that require an even more specific solution (but our API is rather simple and mostly a read-only), or that we're getting damn wrong. My coworker which is the most trusted by my superior doesn't seem well versed into testing... I ever get rebuked for trying to make clear that what they call "unit-testing" actually is something between integration and functional testing with a huge smoke flavor (yeah, really), and that we should make clear what we want to test for which purpose, or that we're not supposed to test private methods. He replied with anger that his tests work and that I am getting "too much theoric" and shouldn't listen to "people that love portmanteau words" because it's useless... That said, I simply don't trust in him, on this point at least !
Does anyone ever had to write such tests which really were the best solutions ? Why was it the best one ? Should I try to make my point of view clearer and / or catch my superior to "force" trying my "basic solution" first ?