My team and I are integrating with a 3rd party company and using their API to perform different CRUD operations. Their API isn't always reliable though. Maybe 0.1% of the time an API call just fails with a 500 error, and then when you try again, it works fine. Occasionally, they will have times where the API call fails with a 500 error more than 90% of the time. After ~10 attempts, it will finally work. It usually lasts for about 4 hours and occurs every few months. We experienced this yesterday where 90% of the time the API call would fail.

The 3rd party told us that we should implement retry logic when receiving a 500 error. For idempotent operations, I could see this being useful. However, for non-idempotent operations it seems like it could be dangerous. For example, one API call might be to send an Email. We've noticed at times that the API might return a 500 error, even though the email was sent. If I retry the API call 10 times, I might get 10 500 errors, but the recipient might still get 10 emails. This could be very bad.

Should we actually be adding retry logic for 500 errors that are not idempotent?

My two thoughts on when we should add retry:

  1. When we are okay with the operation running more than once
  2. When the operation failing is worse than the operation succeeding 10 times (or whatever the retry limit is).

I've never had to add an retry logic to API calls as getting 500 errors is very slim. Is it reasonable to have to do this?

  • 3
    Shouldn't you consider moving to an API which is a bit more reliable? I can't imagine a case where someone would continue using an API with such high rate of failure and such unpredictable behavior. – Arseni Mourzenko Dec 6 at 19:24
  • @ArseniMourzenko, we don't have the option of using a different API. – Eric S Dec 6 at 19:32
  • A little bit of an aside but you might want to consider implementing a circuit breaker to minimize the impact of these periods or instability. – JimmyJames Dec 6 at 21:40

This boils down to what is worse for the specific non-idempotent operation:

  • when it is executed more than once, or

  • when it is executed not at all.

This is not a technical design decision, it depends on the specific operation and the consequences of such failures in the domain.

As a technical measure, you could check if there are additional options for validating if a certain operation has happened (at least with a certain probability). Let us take the email example: if the API allows you to send emailing requests with an included blind-copy request, you may utilize this feature to send your own system such a blind copy. That gives you an opportunity to implement an additional checkpoint if the original email was send or not. Of course, the blind copy may not reach your system in time because of a network lag, but you can reduce the frequency of duplicate email operations through the API.

Or maybe the API itself will allow you to inspect certain status or logging information for certain operations. Though these calls can fail as well, at least you may be able to lower the failure rate to an acceptable degree by using these features.

As a general engineering principle: if you have an unreliable component in your system, create enough redundancy to make the system "reliable enough".

Determine if your API call is "at-least-once", "at-most-once" or "exactly-once" (http://www.cs.unc.edu/~dewan/242/f97/notes/ipc/node27.html) in terms of urgency. For at-least-once, write a retry logic with some kind of backoff (eg, exponential backoff). For other cases, implement better monitoring and alerting so that you know when things go bad. Move failed operations to a queue and implement a UI that will let you manually pick operations from the queue and run them.

As a rule of thumb, yes. 5xx are "server errors" and in the world of REST APIs it's a common occurrence. NOT as frequent as your case, but common.

E.g. for our APIs hosted in AWS using API Gateways and Java Lambda throw 500/503/504 once every 2 days on an average. And usually the first retry goes through. Issues are transient infrastructure issues, gateway connectivity, lamba not reachable, network issues...

ALL REST calls should be retried with exponentially increasing delay between retries for 5xx errors. There are libraries available in all libraries, don't use for loops.

Idempotency of course complicates things a bit, but I would stand by rule of thumb. E.g. 503, 504 surely means you can retry. The API has to make some guarantees or give guidance about it's behavior for 500s.

Of course I second others' opinion that no paid API should be down for 4 hours.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.