14

Assume you are using APIs from a vendor, how to make sure their API is working as expected?

My main concern is sometimes the vendor pushed the changes to their code and break the API, we want to have some sort of automatic software to test them continually. How to deal with this?

|improve this question
  • Depending on the language, there may be tools that can help (I'm thinking Pex for C# libraries/APIs). – Steven Evers Dec 21 '12 at 16:53
10

Short answer: you need a test suite for a third-party vendor API - so you will have to develop one.

Don't expect anyone else to do it for you, and don't expect a "magic bullet" for generating automatically the right tests.

Some things you could try additionally:

  • ask the vendor if they provide a list of "breaking changes" for each new release
  • ask them how they care about API compatibility/inform them that this is an important feature for you
  • check if the API provides specific testing hooks, logging output or something like that for parts which could not be tested easily either
  • wrap important API calls with your own logging code, writing input and related output of the API to a log file, this will make it easier to debug things if something unexpected happens
  • add assertions to API calls to check pre- and postconditions, so if a new release of the API shows up unexpected behaviour within your application, you get informed early by an error message

If these things work or not depends on who is your vendor and what kind of API you have in mind. An API which produces some inspectable output like files is much easier to test than an API which controls some physical device where you have to observe the behaviour of the thing to decide whether the API call was successful or not.

|improve this answer
  • I totally agree, but it seems to me, that the questioner hasn't experience with testing units and doesn't know the scheme of work with them. I mean: find critical points, write test units, run all tests, debug, during debugging find new critical points, write new units, repeat the last 4 steps after every API change. – Gangnus Dec 21 '12 at 9:17
  • @Gangnus: IMHO the OP did not tell us anything about his former experiences with unit testing. If he has issues with that, I am sure he knows to ask a more specific question. Furthermore, the topic here is not "unit tests", but "automated integration tests". Those require typically a different scheme than, for example unit-testing in a TDD style manner. – Doc Brown Dec 21 '12 at 10:25
  • Yes, he hadn't said that. But if he asks "how to make sure their API is working as expected" not mentioning test unites, "it seems to me", that he doesn't know them. As for automated integration tests, he doesn't know them even with higher probability, he mentioned merely "some sort of automatic software". You are awaiting from the people the same knowledges as yours, but in these themes 99% of programmers (including me) know far less. and 90% far far far less. – Gangnus Dec 21 '12 at 11:06
0

Based on the poster's phrasing, it's more than just testing, IMO. After you write your unit test for the API and make sure everything is working as expected, you need to monitor third-party APIs so you catch issues before the users do. That's the real risk with third-party APIs - it's not your code and you have no control over how much testing was done on the API or when/if it changes.

(Disclaimer: product names used here) If you use soapUI to write your API tests, those tests can be re-used in AlertSite as an operational monitor to make sure the API keeps working as expected. If it fails the test, you can get alerted before your users call you and complain that your app doesn't work.

|improve this answer
0

Implement learning tests for your area of interest (features that you plan to use). Learning tests are integration tests that are written by the developer against the public contract of the API. The tests should not be written against the internal implementation details even if the source code for the API is available. This kind of learning tests serves two purpose -

  1. It dramatically improves your understanding of the third-party API.
  2. The tests help to verify whether the claimed new version is actually backward compatible or not.
|improve this answer
0

There are 2 approaches for this issue...

your app is live in production with real user traffic:

if you have an app in production that has live traffic and depends on an external api you have no choice but to closely monitor and have good thresholds to know as fast as possible when the external api makes changes without notifying.

you should always take into account that :

  • api's change over time
  • the api vendor can have bugs
  • api vendors test-kits can have bugs or not fully cover all the production api's functionality

your app is an installation and has planned version/releases:

in this case you have a grace period to fail... live user aren't effected immediately of the external api breaking changes.

in my opinion this is a more easy task. write a test (full end-to-end test) that makes real transactions/http/requests to you application that invoke the external api and check that there are no failures. no test-kits no mocks real transaction.

after this task is done you can choose to run this every 24H , 1 min etc...

good practices:

  • automate everything
  • have a person you can quickly contact from the vendor of the external api
  • dont blindly trust the vendor test everything
  • fail fast - if your service depends heavily on the external api dont let your service crash. fail fast and return proper error messages

tools:

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.