11

As I try to deal with the possibility of failure when I invoke RESTful endpoints or in general any HTTP endpoint I've been wondering if there is any standard or pattern in the HTTP specification or in the industry to deal with the problem of determining when an exception is transient and when it is not particularly for retryability purposes.

Since a remote service might fail at any moment, determining when we ought to retry an operation should to be a fundamental feature of our distributed architecture.

  • I started by considering the method idempotency first (i.e. get, put, delete are safe to retry according to our service contracts).
  • Then I considered the status code where for e.g. 4xx are not retried, but 5xx errors may be retried depending on the type of error, i.e. whether it is transient or not. For example, if the remote service uses a database and I get a 5xx error caused by a query timeout, then it's safe to retry since the condition is transient and the service invocation will most likely succeed if I retry. However, there could be other type of errors that are not transient and that I would prefer to avoid retrying. For example, in the past we have had errors caused due to a DBA adding a new constraint to a table and then a service that was working before started failing with a non-transient error like a constraint violation.

In the past, we have made the mistake of infinitely retrying a 500 error thinking that they are always transient and that the remote service will eventually recover and be able to handle the request. Particularly in computer-to-computer interactions (orchestrations) where we'd like to avoid at all costs propagating an exception since it would require complex compensating transactions and publishing partially processed requests in a DLQ for later human intervention.

In general I'd like to know how do people in the industry usually deal with this issue, how is this property of "transientness" conveyed from the server to the client using the HTTP protocol.

Should I resort to using customized http status codes or should I communicate this property in a header or in a property of the body?

If there is a standardize solution for this?

That would be awesome because if there is, I could expect third-party services to consume my services knowing they will behave correctly and at the same time it means I could also consume theirs without having to reconfigure my retry protocol for their particular implementation.

I welcome any suggestions on retryability protocols to design good service contracts that help me build good service citizens of our ecosystem that may potentially need to integrate with third-party services in the future.

So far I am particularly amazed on discovering that a distributed architecture pattern like this does not make of retryability a first class citizen, but it could be that I am mistaken in my interpretation of how the implementation should work. Can anybody please point me into the right direction?

2 Answers 2

10

For your transient failures, don't return 500. Return 503 Service Unavailable. The query timeout is being caused by "a temporary overload", not by an underlying error in the code. In your response, include a Retry-After header indicating how long the client should wait before retrying the request.

1
  • I didn't know about the Retry-After header. This approach makes a lot sense and it is quite simple, as I'd hope this kind of thing should be.
    – edalorzo
    Feb 5, 2017 at 15:40
1

Retrying infinitely is probably not a good idea. If something goes down and does not come up quickly, it's a waste of your resources. How I've handled this situation was having a fixed number of retries, and an increasingly longer delay between each retry. In my experience, if something does not respond within several retries, it will probably be down for hours. But that's just my experiences...

The details of how often to retry and how often to wait for each retry will probably depend on the specific service you're interacting with, and you'll just have to rely on your experience with that service.

One service my application talks to times out occasionally, but the retries are usually successful. It has never returned a 5xx error code (well, I see 504 in the logs for the timeout, but never any others). So far, I've never had to retry more than 5 times, waiting at least 30 seconds between each retry. Their server is known to be slow, and has been overloaded before. Another service (from a different vendor) always responds very quickly, but sometimes returns a 5xx code (usually 500 or 503), and a 1 or 2 retries usually resolves that problem. If anything takes more than the allowed amount of retrying, then it is logged as a serious error and the application just keeps on working on other things.

One other thing: If you are writing both the service and the client, you could add more detailed information to the response message for a 5xx error. That way, if you know the service you're writing can recover from some errors but not others, you could include that detail in the response so the client knows to retry or not. As an example, I've seen one service that has a rate limit. When that limit is exceeded, the 5xx error response comes with a "retry-after" header which is the number of minutes after which the limit will reset and then you can retry. So in that case, a custom header (which needs to be documented for other users) contains information about a specific retry scenario.

5
  • Interesting, so the recommendation is not to pay too much attention to "transientness". Retry regardless of the 5xx error type and on exhausting attempts deal with it somehow. I have the feeling that the HTTP specification or the REST architecture pattern is missing a piece of standardization in this aspect, though.
    – edalorzo
    Feb 3, 2017 at 21:33
  • 1
    @edalorzo: well, it depends on the service, and how it was implemented. In general (in my experience), 500, 502, 503, and 504 can usually be retried at least once with some hope that a good response will come back. I've never worked with a service that responded with 501, but I imagine that response would imply "do not retry now - retry in the distant (days/weeks/months) future when we implement that endpoint". 505 to 508 probably should not be retried (well you probably can but you probably won't get anything back), but the logs for my application never seem to get these responses. Feb 3, 2017 at 21:56
  • 1
    @edalorzo: Looking at this: en.wikipedia.org/wiki/List_of_HTTP_status_codes 510 and 511 are probably not retriable, as they imply the client needs to do something more to get a good response. 509 is a bandwidth limit error, probably no point in retrying that one. There are other unofficial status codes, that you can decide if they should be retried. Feb 3, 2017 at 22:01
  • 1
    Very interesting! It seems that defining a good contract for the service retryability will be a bit harder than what I was expecting with all this "possibly retryable" codes and the possibility of third-party services using customized codes. I suppose I can control those services we implement, but there is no way to assume anything about how third-party services will behave until they give us a contract which could possibly ignore this issue entirely. I find it interesting that a distributed architecture paradigm like this does not make of retryability a first class citizen. Don't you think?
    – edalorzo
    Feb 3, 2017 at 22:07
  • 1
    Very good point about having a congestion control strategy. Feb 5, 2017 at 15:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.