Should you retry 500 API errors?

Question

My team and I are integrating with a 3rd party company and using their API to perform different CRUD operations. Their API isn't always reliable though. Maybe 0.1% of the time an API call just fails with a 500 error, and then when you try again, it works fine. Occasionally, they will have times where the API call fails with a 500 error more than 90% of the time. After ~10 attempts, it will finally work. It usually lasts for about 4 hours and occurs every few months. We experienced this yesterday where 90% of the time the API call would fail.

The 3rd party told us that we should implement retry logic when receiving a 500 error. For idempotent operations, I could see this being useful. However, for non-idempotent operations it seems like it could be dangerous. For example, one API call might be to send an Email. We've noticed at times that the API might return a 500 error, even though the email was sent. If I retry the API call 10 times, I might get 10 500 errors, but the recipient might still get 10 emails. This could be very bad.

Should we actually be adding retry logic for 500 errors that are not idempotent?

My two thoughts on when we should add retry:

1. When we are okay with the operation running more than once
2. When the operation failing is worse than the operation succeeding 10 times (or whatever the retry limit is).

I've never had to add an retry logic to API calls as getting 500 errors is very slim. Is it reasonable to have to do this?

Answer 1

This boils down to what is worse for the specific non-idempotent operation:

• when it is executed more than once, or

• when it is executed not at all.

This is not a technical design decision, it depends on the specific operation and the consequences of such failures in the domain.

As a technical measure, you could check if there are additional options for validating if a certain operation has happened (at least with a certain probability). Let us take the email example: if the API allows you to send email requests with an included blind-copy request, you may utilize this feature to send your own system such a blind copy. That gives you an opportunity to implement an additional checkpoint if the original email was sent or not. Of course, the blind copy may not reach your system in time because of a network lag, but you can reduce the frequency of duplicate email operations through the API.

Or maybe the API itself will allow you to inspect a certain status or logging information for certain operations. Though these calls can fail as well, at least you may be able to lower the failure rate to an acceptable degree by using these features.

As a general engineering principle: if you have an unreliable component in your system, create enough redundancy to make the system "reliable enough".

Answer 2

Determine if your API call is "at-least-once", "at-most-once" or "exactly-once" in terms of urgency.

For at-least-once, write a retry logic with some kind of backoff (eg, exponential backoff). For other cases, implement better monitoring and alerting so that you know when things go bad. Move failed operations to a queue and implement a UI that will let you manually pick operations from the queue and run them.

Answer 3

The real hard problem is this: If you make a request that must be fulfilled exactly once, and you get a 500 error, has the request been fulfilled or not? All other problems can be solved.

Possible solutions are:

1. The API provider guarantees not to perform any action if an error 500 is returned. That seems difficult, since these 500 errors shouldn’t happen in the first place.
2. You design all the requests in such a way that they can be repeated. For example, a request “send an email if it hasn’t been sent yet”.
3. Your design has the ability to check independently whether a request has been sent. For example, after making a $200 payment to x, you might be able to get a list of all payments and check if the $200 payment is in there. So you know whether to repeat the request or not.

Of course you have the problem that the server sometimes doesn’t work correctly, so it’s hard to say what can be guaranteed.

Answer 4

As a rule of thumb, yes. 5xx are "server errors" and in the world of REST APIs it's a common occurrence. NOT as frequent as your case, but common.

E.g. for our APIs hosted in AWS using API Gateways and Java Lambda throw 500/503/504 once every 2 days on an average. And usually the first retry goes through. Issues are transient infrastructure issues, gateway connectivity, lamba not reachable, network issues...

ALL REST calls should be retried with exponentially increasing delay between retries for 5xx errors. There are libraries available in all libraries, don't use for loops.

Idempotency of course complicates things a bit, but I would stand by rule of thumb. E.g. 503, 504 surely means you can retry. The API has to make some guarantees or give guidance about it's behavior for 500s.

Of course I second others' opinion that no paid API should be down for 4 hours.

Answer 5

IMO the correct answer is read the API documentation, try to retry on errors specificially indicated as retryable, and don't blindly retry on generic 500 until you've weighed the pros and cons.

Example

Most web services document how their API works at a more granular level than HTTP status codes. For example, some AWS services include a response header called x-amzn-ErrorType which classifies the error at a more detailed level than the HTTP status code can provide. For example, the AWS Lambda Invoke operation has a detailed list of error messages. Some of them are obviously retryable, and some of them are obviously not retryable, just by looking at the description.

In the AWS case, the language-specific SDKs have logic which decides whether or not to retry, not based on the HTTP status code, but based on the granular error code provided by the specific service (e.g. Lambda) and whether the service considers that error code retryable.

Generalizing

For HTTP status code 500 in general, there is nothing inherently retryable about it. The definition of a 500 is:

500 INTERNAL SERVER ERROR

The server encountered an unexpected condition that prevented it from fulfilling the request.

That is a very general error code.

Whether this very general error might succeed upon a retry depends on why the error occurred.

If the server has a programming error that causes a null pointer exception every time you send a particular input, then you can retry it until you are blue in the face but the result will always be the same (and you may take down the server or find yourself paying for resources you didn't use).

On the other hand, if the server has some kind of temporary problem related to a resource it depends on (maybe a bad disk, or some downstream web service it depends on) then the problem may be quickly corrected such that a retry will work.

So I suggest reading the API documentation carefully, including any additional metadata that is available to more precisely describe the error. If you get a generic 500 with little or no useful metadata, you'll have to weigh the pros and cons of blindly retrying (are you going to pay for it, are you putting the downstream web service at risk from a retry storm, are you putting your own availability at risk by doing useless retries that will make your clients time out, etc.).

Answer 6

As a general rule, 500 errors should be retried. There is a rule of thumb for most error codes:

• 2xx = Everything that can be done has been done.
• 4xx = You (the caller) did something wrong or asked for something nonsensical.
• 5xx = We (the server) did something wrong.

4xx errors generally boil down to a request that is inherently bad and cannot be handled. 5xx errors are used to indicate that the server encountered a private issue, and there is no indication that your request is provably invalid.

That doesn't mean your request must have been valid. For example, let's say you send a request to fetch an item but you use an ID that doesn't exist. A server that handles the request well will give you a 404 response (or a 204. I've heard arguments either way and I think the distinction is contextual). However, if the server has a bug which leads to a null reference exception, you're likely to get a 500 response.

As the caller, when you get a 500 response, you don't know what went wrong. All you know is that the server did not actively tell you that your request is nonsensical. Which means that you cannot assume that there's no point in trying to make the same request again.
For example, maybe you received the 500 because the server database was offline. If you wait a bit and try again, it may succeed now that the database is online.

But when you receive a 4xx error, the server has actively stated that your request is unresolvable and there's no point in retrying.

This is a bit of an overgeneralization, fringe exceptions exist. For example, status 429 (too many requests) means that you (the caller) have exceeded your allotted amount of calls, but it's likely that your same request will be processed correctly if you wait until you've been allotted more requests (e.g. if you hit the daily maximum, it will work again tomorrow).

---

However, sul4bh is correct that this can be contextual, and that some applications may not want you to send the same request multiple times.

For example, consider a banking transaction. You send the request, and get a 500. But in reality, the server did actually process your request, but it encountered a trivial bug in the formatting of the response message.
Send your request again may be harmful, as you'll end up making several banking transactions.

However, I don't think that this is up to you (the caller) to decide when you are allowed to fire the same request and when you are not. It's the server's responsibility to clearly communicate the true state of things to the caller, and in the above scenario, it failed to do so. You can't be held responsible for being given bad information and responding correctly to the information given.

If I make a payment via online banking, and I get a 500 when I submit the transaction, I'm going to try it again. It's the most sensical response to a 500 error. To that effect, my bank actually has implemented a check. If it notices that you are making a second transfer in quick succession for the same amount and/or to the same account, it will tell you that it did in fact register the first transaction.

So this is very dependent on the server. They are expected to clearly communicate to you, and when you assume clear communication, a 500 means you should retry at a later time because the cause of the issue was serverside, not in your request.
If the server cannot clearly communicate, or the context of the application requires the caller to err on the side of never taking the same action twice, then the server's administrators need to clearly communicate that to their API's consumers, and should make reasonable efforts towards preventing erroneously taking the same action twice.