REST: How to determine transient exceptions?

Question

As I try to deal with the possibility of failure when I invoke RESTful endpoints or in general any HTTP endpoint I've been wondering if there is any standard or pattern in the HTTP specification or in the industry to deal with the problem of determining when an exception is transient and when it is not particularly for retryability purposes.

Since a remote service might fail at any moment, determining when we ought to retry an operation should to be a fundamental feature of our distributed architecture.

• I started by considering the method idempotency first (i.e. get, put, delete are safe to retry according to our service contracts).
• Then I considered the status code where for e.g. 4xx are not retried, but 5xx errors may be retried depending on the type of error, i.e. whether it is transient or not. For example, if the remote service uses a database and I get a 5xx error caused by a query timeout, then it's safe to retry since the condition is transient and the service invocation will most likely succeed if I retry. However, there could be other type of errors that are not transient and that I would prefer to avoid retrying. For example, in the past we have had errors caused due to a DBA adding a new constraint to a table and then a service that was working before started failing with a non-transient error like a constraint violation.

In the past, we have made the mistake of infinitely retrying a 500 error thinking that they are always transient and that the remote service will eventually recover and be able to handle the request. Particularly in computer-to-computer interactions (orchestrations) where we'd like to avoid at all costs propagating an exception since it would require complex compensating transactions and publishing partially processed requests in a DLQ for later human intervention.

In general I'd like to know how do people in the industry usually deal with this issue, how is this property of "transientness" conveyed from the server to the client using the HTTP protocol.

Should I resort to using customized http status codes or should I communicate this property in a header or in a property of the body?

If there is a standardize solution for this?

That would be awesome because if there is, I could expect third-party services to consume my services knowing they will behave correctly and at the same time it means I could also consume theirs without having to reconfigure my retry protocol for their particular implementation.

I welcome any suggestions on retryability protocols to design good service contracts that help me build good service citizens of our ecosystem that may potentially need to integrate with third-party services in the future.

So far I am particularly amazed on discovering that a distributed architecture pattern like this does not make of retryability a first class citizen, but it could be that I am mistaken in my interpretation of how the implementation should work. Can anybody please point me into the right direction?

Answer 1

For your transient failures, don't return 500. Return 503 Service Unavailable. The query timeout is being caused by "a temporary overload", not by an underlying error in the code. In your response, include a Retry-After header indicating how long the client should wait before retrying the request.

Answer 2

Retrying infinitely is probably not a good idea. If something goes down and does not come up quickly, it's a waste of your resources. How I've handled this situation was having a fixed number of retries, and an increasingly longer delay between each retry. In my experience, if something does not respond within several retries, it will probably be down for hours. But that's just my experiences...

The details of how often to retry and how often to wait for each retry will probably depend on the specific service you're interacting with, and you'll just have to rely on your experience with that service.

One service my application talks to times out occasionally, but the retries are usually successful. It has never returned a 5xx error code (well, I see 504 in the logs for the timeout, but never any others). So far, I've never had to retry more than 5 times, waiting at least 30 seconds between each retry. Their server is known to be slow, and has been overloaded before. Another service (from a different vendor) always responds very quickly, but sometimes returns a 5xx code (usually 500 or 503), and a 1 or 2 retries usually resolves that problem. If anything takes more than the allowed amount of retrying, then it is logged as a serious error and the application just keeps on working on other things.

One other thing: If you are writing both the service and the client, you could add more detailed information to the response message for a 5xx error. That way, if you know the service you're writing can recover from some errors but not others, you could include that detail in the response so the client knows to retry or not. As an example, I've seen one service that has a rate limit. When that limit is exceeded, the 5xx error response comes with a "retry-after" header which is the number of minutes after which the limit will reset and then you can retry. So in that case, a custom header (which needs to be documented for other users) contains information about a specific retry scenario.