The only real logic is in the query syntax for the external API. I don't want to test whether it queries the api, I want to test that it queries it in such a way that the correct data will be returned. For example, some pseudo-code:

function retrieve_related_data(id)
  query = "[potentially long, syntactically complex query that
            uses param id to get some data]";
  results = api_wrapper.query(query);
  return results;

A more concrete example with a made up API:

function retrieveLifeSupportingObjectsWithinRegion(id)
  query = "
    within region(" + id + ") as r
    find objects matching hydration>0 and temp_range has 75
    send name, id, relative(position, r)        
  results = astronomicalObjectApiWrapper.query(query);
  return results;

The query is in a syntax custom to the API and is complex and there are multiple ways to achieve the same or similar results. The purpose of the function is not to get data identified by id but to find a subset of other data based on a fuzzy relationship to the data identified by id that also meets a few other requirements. The other requirements are always the same regardless of id but may change over time as the system is modified. For example, if the example api added support for gravity information, we may want to change the query to also use gravity to refine the results. Or maybe we come up with a more efficient way to check the temp range, but it doesn't change the results.

What I want to test is that for a given input id the correct set of data is returned. I want to test this so that if someone messes up the query such that it is no longer returning the correct data based on id that it will fail, but I also want people to be able to modify the query to refine it without needing to also modify the test.

Options I've considered:

  1. I could stub the api, but that would either be too simple (check that the id is present in the query and then return an expected set of data if it is or an unexpected set if not), too brittle (check that the query string is exactly what is in the function), or too complex (check that the query used is syntactically correct and will result in the correct data being returned).

  2. I could submit the query to the real api, but the expected results could change over time as the data in the external system changes, outside of the control of the test system.

  3. I could look at setting up a test install of the real api in order to control the data it has, but that is a lot of effort.

I'm leaning towards #2 and making this more of an integration test that doesn't get run often and seeing how often changes in the data of the external system causes the test to break. I think that would be simplest for now, but I am wondering if there are alternatives that I am not thinking of or better ways to tackle this issue. Any advice would be appreciated.

  • I was thinking of this as a unit test, but maybe it is really an integration test or a low-level acceptance test?
    – Josh Coady
    Commented Mar 25, 2017 at 19:26
  • 2
    Is it a read-only API? Or, are you able to write data you can reliably verify your reads against?
    – svidgen
    Commented Apr 1, 2017 at 19:18
  • Is this question different from "how to test that my sql (=complex syntax) returns the the correnct data"? With databases you usually have a few integration tests that test the crud-sql-syntax and a mocked repository fasade to verify businesslogic
    – k3b
    Commented Jul 17, 2017 at 15:46
  • This depends so much on the maker of the API. For integration testing we had a party that gave use a "test-endpoint" where we could test on, a lot of different scenario's and what exact response the api would give in the scenario's.
    – Pieter B
    Commented Jun 28, 2023 at 11:26

3 Answers 3


It may seem that by validating the API response you would be testing your function, but it would not be true. You would be testing the API and its environment.

Your tests should focus on validating the behaviour of the code you have written, not the one written by 3rd parties. To some extent, you have to trust in others' code. For the same reason you don't test frameworks and libraries, you won't test external services.

What I want to test is that for a given input id the correct set of data is returned

What would you be testing? As you said, the data and its correctness are not under your control, so you would be conditioning the success of the tests to an external agent over which you have no influence. In these circumstances, tests become non-deterministic and definitively, you don't want that to happen.

You could, instead, test contracts1 which don't involve your code. A simple and homemade implementation is possible with Postman and Newman. Once requests are recorded, and test programmed it only takes you to schedule the execution. The idea is to execute these tests in different pipelines and at a different time and pace than the rest of the code.

I want to test this so that if someone messes up the query such that it is no longer returning the correct data based on id that it will fail

What if the query is ok, but the data changed or is wrong due to bugs in the API? Not only the data is out of our control, the logic too. Moreover, what you have considered "correct" can be eventual. Maybe, the "correct" state of the data is achieved later without you knowing when, how or why.

Implementing functional and end-to-end tests may help here. You could address these tests so that if the API returns wrong data, you see it by detecting misbehaviour in your code.

But I also want people to be able to modify the query to refine it without needing to also modify the test.

I would suggest implementing an instrumental test. Instrumental tests are test classes, executed as test classes but they aren't included in the test plan.

Command line shells or Python scripts can be good alternatives too. One way or another, make it an instrument. One everyone can run, at any time without being forced to know the rest of the source code.

Related questions:

1: You might be interested in @DocBrown's answer regarding this topic

  • "The problem (IMO) is that you are too focused on testing the external API." - I don't see anything indicating that the asker is interested in testing the external API at all. Also, you say "stub the external API", but do you have any suggestions as to whether the asker should use the "too simple" option, the "too brittle" option, the "too complex" option, or some fourth option? Commented Mar 26, 2017 at 22:41
  • The OP question is asking how to tests a function that call to a external API. But reading the his doubts, seems to me that he puts too much emphasis on testing the query and its results. I made 4 suggestions: (1) don't do test of the API. (2) don't use the integration tests as workbench for tunning the query. Make a tool instead. (3) back to the main question, do its unitary and Integration test. But not validating the content of the API response. (4) Ask to the project manager if they should/can make a test suite of the external API as part of the test plan of the project.
    – Laiv
    Commented Mar 27, 2017 at 5:04
  • 1
    I consider the query itself as "code wrote by us". The end-goal is automated testing to alert us if we introduced a bug in our code. Taking it to what you said about SQL statements, I guess it is similar to that -- I guess my question is like how do you test that your code is querying an external API in a manner that the API will respond as intended (assuming a nominal response). I think what you are saying is to leave that out of the unit and integration tests, but if the queries are mission critical, we could setup some other automated tests separately to test the live external api.
    – Josh Coady
    Commented Mar 27, 2017 at 23:39
  • 1
    IMO , the best way is doing functional test , the change in the where clause that is never going to be true, will cause a different behaivor in one or more of my functional tests. U.T are only a little piece in the test plan.
    – Laiv
    Commented Apr 1, 2017 at 7:37
  • 3
    Thanks for being a sounding board. I guess ultimately, even though the query has custom logic in it, it is outside of the domain of unit testing because the query is code that is "executed" outside of the system being tested. I just needed someone to tell me that multiple times in different ways before I saw it ;)
    – Josh Coady
    Commented Apr 1, 2017 at 23:39

I have seen unit checks which check the generated query string matches an expected value.

However. This was in my opinion if limited use. The query syntax was complicated, possibly buggy, so A there were endless possibilities to check and B even if the string was 'correctly' generated, unexpected results could be returned in the live environment.

I think you are right to go for your option 2. run integration tests against the live instance.

As long as they are non destructive, these are the first tests you should write, as they will catch, although not identify the cause of, any error.

Going for option 3 'deploy a test instance with dummy data' is superior. But doesn't affect your test writing, as you can point the same tests at the test server if and when it becomes a good use of time to deploy one.


It depends on the API, but if possible, go for option #3 (private testing instance).

Stubbing the API (option #1) is the worst option, because of the reasons you mentioned, and going this route will probably do more harm than good (lots of wasted time).

Running against the real API (option #2) makes the tests flaky and unreliable, and after a few false positives people will just stop using them. Not only can the data change, but the service might be down as well. In my opinion, this is akin to having no tests for the queries, and relying on integration/system tests to find the issues. That said, if the API data rarely changes and the API itself is almost always up, then this might be a viable option. Most APIs don't fit this description.

Eventually it comes down to how important and complex these queries are: if there are more than a handful, and some of them are complex enough that you feel the need to test them, I would invest the effort of setting up a private instance for testing. It will pay for itself just like other unit tests.

  • So basically, you are telling that unit tests (#1) and integration tests (#2) are harmful? While #3 may seem the best among them It could be also the most expensive too. It has to be maintained every time the API changes. Without #2 you won't be aware of possible bugs-changes in the real API till you app is in production (too late for taking measures). Ok, #1 seems uneeded because there're not lines of code to test... today... Tomorrow, how knows...
    – Laiv
    Commented Apr 3, 2017 at 19:33
  • I'm saying bad tests are harmful, definitely. Flaky tests waste a lot of time and effort, and cause people to lose faith in unit tests as a whole. Tests that break with implementation changes (#1) or just randomly when data changes (#2) are not good tests. Commented Apr 4, 2017 at 7:15
  • Integration test don't test data. That's it. They can't break the testing, just validate the integration. Testing is not matter of faith is matter of having good habits that add value to the application
    – Laiv
    Commented Apr 4, 2017 at 11:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.