16

The only real logic is in the query syntax for the external API. I don't want to test whether it queries the api, I want to test that it queries it in such a way that the correct data will be returned. For example, some pseudo-code:

function retrieve_related_data(id)
{
  query = "[potentially long, syntactically complex query that
            uses param id to get some data]";
  results = api_wrapper.query(query);
  return results;
}

A more concrete example with a made up API:

function retrieveLifeSupportingObjectsWithinRegion(id)
{
  query = "
    within region(" + id + ") as r
    find objects matching hydration>0 and temp_range has 75
    send name, id, relative(position, r)        
  ";
  results = astronomicalObjectApiWrapper.query(query);
  return results;
}

The query is in a syntax custom to the API and is complex and there are multiple ways to achieve the same or similar results. The purpose of the function is not to get data identified by id but to find a subset of other data based on a fuzzy relationship to the data identified by id that also meets a few other requirements. The other requirements are always the same regardless of id but may change over time as the system is modified. For example, if the example api added support for gravity information, we may want to change the query to also use gravity to refine the results. Or maybe we come up with a more efficient way to check the temp range, but it doesn't change the results.

What I want to test is that for a given input id the correct set of data is returned. I want to test this so that if someone messes up the query such that it is no longer returning the correct data based on id that it will fail, but I also want people to be able to modify the query to refine it without needing to also modify the test.

Options I've considered:

  1. I could stub the api, but that would either be too simple (check that the id is present in the query and then return an expected set of data if it is or an unexpected set if not), too brittle (check that the query string is exactly what is in the function), or too complex (check that the query used is syntactically correct and will result in the correct data being returned).

  2. I could submit the query to the real api, but the expected results could change over time as the data in the external system changes, outside of the control of the test system.

  3. I could look at setting up a test install of the real api in order to control the data it has, but that is a lot of effort.

I'm leaning towards #2 and making this more of an integration test that doesn't get run often and seeing how often changes in the data of the external system causes the test to break. I think that would be simplest for now, but I am wondering if there are alternatives that I am not thinking of or better ways to tackle this issue. Any advice would be appreciated.

  • I was thinking of this as a unit test, but maybe it is really an integration test or a low-level acceptance test? – Joshua Coady Mar 25 '17 at 19:26
  • 2
    Is it a read-only API? Or, are you able to write data you can reliably verify your reads against? – svidgen Apr 1 '17 at 19:18
  • Is this question different from "how to test that my sql (=complex syntax) returns the the correnct data"? With databases you usually have a few integration tests that test the crud-sql-syntax and a mocked repository fasade to verify businesslogic – k3b Jul 17 '17 at 15:46
7

It may seem that validating the external API response we would be testing our function, but it would not be totally true. Somehow, we would be testing the external API and the environment where the API is running at.

Our tests should be addressed to guarantee the expected behaviour of the code we have written, not the one written by 3rd parties.

To a certain degree, we have to trust on the proper function of the APIs and libs we rely on. For example, we usually don't test the framework components we implement.

Why do I say so?

What I want to test is that for a given input id the correct set of data is returned

What would be tested here? As you said, the data and its correctness are not under our control, so we would be constraining the success of the testing phase to an external agent which we don't have any control. These tests are a candidate to become non-deterministic and definitively, we don't want these sort of tests in our building pipeline.

A different concern is to validate the contract. I would find to be quite useful a contract tests1 to ensure that the integration is still working as expected, before any release or deployment.

I want to test this so that if someone messes up the query such that it is no longer returning the correct data based on id that it will fail

What if the query is ok, but the data is wrong due to bugs in the API? Not only the data is out of our control. The logic is too.

Implementing functional tests or end-to-end test may help here. You could address these test to validate certain execution paths so that if the APIs returns wrong data, this will probably cause unexpected behaviours and outputs. On the other hand, I would expect the API to throw errors if my queries were bad-formatted.

But I also want people to be able to modify the query to refine it without needing to also modify the test.

I suggest implementing a tool for such purpose. It could be as simple as:

  • A class that runs as a test but it doesn't belong to the test plan
  • A shell script + curl

Or something more sophisticated. For example, a standalone client.

In any case, the function under the question, well worth two kinds of tests:

  • Unit Test. As you said, you have to stubb the external API, but that's the whole purpose of unit tests. Testing our code isolating the dependencies.

  • Integration test. Check that the code not only sends the correct request but that it properly handles response content, errors, redirects, etc. Do tests for all these cases, but don't test data.

Side note: Your question is similar to -how do you test the SQL statements of the app-?

Related questions:


1: You might be interested in @DocBrown's answer regarding this topic

  • "The problem (IMO) is that you are too focused on testing the external API." - I don't see anything indicating that the asker is interested in testing the external API at all. Also, you say "stub the external API", but do you have any suggestions as to whether the asker should use the "too simple" option, the "too brittle" option, the "too complex" option, or some fourth option? – Tanner Swett Mar 26 '17 at 22:41
  • The OP question is asking how to tests a function that call to a external API. But reading the his doubts, seems to me that he puts too much emphasis on testing the query and its results. I made 4 suggestions: (1) don't do test of the API. (2) don't use the integration tests as workbench for tunning the query. Make a tool instead. (3) back to the main question, do its unitary and Integration test. But not validating the content of the API response. (4) Ask to the project manager if they should/can make a test suite of the external API as part of the test plan of the project. – Laiv Mar 27 '17 at 5:04
  • 1
    I consider the query itself as "code wrote by us". The end-goal is automated testing to alert us if we introduced a bug in our code. Taking it to what you said about SQL statements, I guess it is similar to that -- I guess my question is like how do you test that your code is querying an external API in a manner that the API will respond as intended (assuming a nominal response). I think what you are saying is to leave that out of the unit and integration tests, but if the queries are mission critical, we could setup some other automated tests separately to test the live external api. – Joshua Coady Mar 27 '17 at 23:39
  • 1
    IMO , the best way is doing functional test , the change in the where clause that is never going to be true, will cause a different behaivor in one or more of my functional tests. U.T are only a little piece in the test plan. – Laiv Apr 1 '17 at 7:37
  • 2
    Thanks for being a sounding board. I guess ultimately, even though the query has custom logic in it, it is outside of the domain of unit testing because the query is code that is "executed" outside of the system being tested. I just needed someone to tell me that multiple times in different ways before I saw it ;) – Joshua Coady Apr 1 '17 at 23:39
2

I have seen unit checks which check the generated query string matches an expected value.

However. This was in my opinion if limited use. The query syntax was complicated, possibly buggy, so A there were endless possibilities to check and B even if the string was 'correctly' generated, unexpected results could be returned in the live environment.

I think you are right to go for your option 2. run integration tests against the live instance.

As long as they are non destructive, these are the first tests you should write, as they will catch, although not identify the cause of, any error.

Going for option 3 'deploy a test instance with dummy data' is superior. But doesn't affect your test writing, as you can point the same tests at the test server if and when it becomes a good use of time to deploy one.

0

It depends on the API, but if possible, go for option #3 (private testing instance).

Stubbing the API (option #1) is the worst option, because of the reasons you mentioned, and going this route will probably do more harm than good (lots of wasted time).

Running against the real API (option #2) makes the tests flaky and unreliable, and after a few false positives people will just stop using them. Not only can the data change, but the service might be down as well. In my opinion, this is akin to having no tests for the queries, and relying on integration/system tests to find the issues. That said, if the API data rarely changes and the API itself is almost always up, then this might be a viable option. Most APIs don't fit this description.

Eventually it comes down to how important and complex these queries are: if there are more than a handful, and some of them are complex enough that you feel the need to test them, I would invest the effort of setting up a private instance for testing. It will pay for itself just like other unit tests.

  • So basically, you are telling that unit tests (#1) and integration tests (#2) are harmful? While #3 may seem the best among them It could be also the most expensive too. It has to be maintained every time the API changes. Without #2 you won't be aware of possible bugs-changes in the real API till you app is in production (too late for taking measures). Ok, #1 seems uneeded because there're not lines of code to test... today... Tomorrow, how knows... – Laiv Apr 3 '17 at 19:33
  • I'm saying bad tests are harmful, definitely. Flaky tests waste a lot of time and effort, and cause people to lose faith in unit tests as a whole. Tests that break with implementation changes (#1) or just randomly when data changes (#2) are not good tests. – Michal Tenenberg Apr 4 '17 at 7:15
  • Integration test don't test data. That's it. They can't break the testing, just validate the integration. Testing is not matter of faith is matter of having good habits that add value to the application – Laiv Apr 4 '17 at 11:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.