Microservices - Are retries enough?

Question

I have been asked to look at moving our current architecture to microservices. I am aware of the warning to always assume a request could fail.

So I am aware we should always be prepared to retry the request. However, when designing this, I am also assuming that the retry can fail.

So with that in mind we have been looking at a pattern where either all the processing in committed or it always rollback. This is achieved via message Outbox (and Inbox) Outbox pattern. The services stored the functional changes in their database, then within the same transaction stores the event messages in their database in an Outbox. A separate dispatcher service then dispatches the messages from the Outbox and sends it to a messaging system. It is detailed in this series of articles Life Beyond Distributed Transactions: An Apostate's Implementation

To me this is the safest option because if the dispatcher fails to send the message, it is available for a retry.

However one of my colleagues thinks that although we need to retry, the solution will be resilient enough that the message will always be successfully sent to the messaging system. E.g. the issue that causes the need for the retry will always be transient, and will be cleared by in time for one of our retries to succeed.

I'm looking for a opinions on whether I'm being over cautious and retries should be enough. Therefore I do not need the dispatcher or the outbox pattern.

I guess the main problem is not that a service I am calling cannot be reached, but the server my service is running on shuts down.

Answer 1

The outbox and retries strategy will work but adds a lot of complexity to each microservice. Because this strategy by necessity forces you into an asynchronous approach, consider using an event-driven architecture.

The microservices communicate through events published on highly available event channels (kafka, rabbitmq, ...). Each service pulls events from the channel, does whatever it should in respond to them, and produces new events as a consequence.

• These are events, not messages, meaning they describe only what a service did, not what it wants another service to do. Correlation id's are used to observe which actions happened as a consequence of broadcasting an event.
• The CQRS pattern is used to create views of necessary data from broadcasted events. In this way each service can handle all read requests without any synchronous call to another service. The consequences on other services of a write are not handled by the microservice itself, but by the other services which subscribe to the event it publishes when it is written to.

The benefits of this approach:

• Microservices need not have any awareness of other microservices. They are only aware of which events they subscribe to.
• The event channel is separate from the microservices, allowing a simpler microservice design while still guaranteeing event delivery.
• The pull instead of push design eliminates the need for retries. Either a service is up and pulls events from the channel, or it is down and does not pull events, which will remain in the channel until it reappears.

The downsides are that you will have to redesign everything in the Event Sourcing style (which can be difficult), and that the explosion of events can be hard to manage (solutions: schema stores, AsyncAPI / Cloudevents.io, correlation and causation id's, logging service that observes all events and records them).

Answer 2

Note that if a request succeeds but its response fails, you have a problem, because the client thinks the request failed whereas the server thinks it succeeded.

The word you're looking for is Idempotence.

Also useful information here: Fallacies of distributed computing -- read it, understand it, design your system based on it!

Answer 3

As juhist pointed out, the first fallacy of distributed computing is "The network is reliable."

I think your proposed outbox solution makes perfect sense for ensuring outgoing messages are always successfully delivered. There is no way to know in advance how long the failure's cause will exist. Do you retry 1 time? 5 times? Do you wait 1 second in between tries? 30 seconds? 15 minutes?

The recovery time could be anywhere from transient errors that succeed on the next retry to the more extreme like a physical disaster where it takes days to restore service. Even the most reliable networks and infrastructure still have things go wrong periodically.

Failures may also occur from updates gone wrong. My company has had cases where specific message handlers started completely failing after we deployed updates, both internal to the service and external dependencies. New messages in the system continued through as far as they could before failing. In this case, it took human intervention to fix the cause which again could vary from a few minutes to a few days to implement a fix. Yes, we test, but things still slip through the cracks from time to time.

A common approach to recoverability is multiple stages

1. Immediate - this handles transient errors by immediately retrying the failed message with no or little delay.

2. Delayed - this covers scenarios where immediate retries were not successful by waiting longer between retries. It might start with 10 seconds for the first retry, 20 seconds for the second, etc. The thought is it gives the receiving system more time to recover the longer the outage lasts, for example an overloaded web server might have a higher chance of recovering the longer you wait it out.

3. Manual - eventually the systematic retries should give up and move the failed messages out of the outbox/messaging system to somewhere else for system administrators (operations, pick your favorite title), to see and retry errors that require more manual intervention. Messages would be moved here after a number of delayed retries. This helps avoid blocking your outbox or queuing system with messages that are not likely to succeed until some manual intervention is done, and allows other messages to continue to flow unobstructed.

The worst thing you can do in a distributed, message-based system is to lose messages so no matter what, make sure you do not give up on retries without a way to continue them later.

This information is based on:

• https://particular.net/blog/the-network-is-reliable
• https://docs.particular.net/nservicebus/recoverability/

Answer 4

I guess it depends on how much resilience you need. You can never just assume, that the reciever will always be ready within a certain amount of retries.

An outbox pattern is deffinitely a viable solution (and probably the best), but maybe it would be enough with some sort of log mechanic, so you can trace failed messages?

In the end, it depends on your needs, but with my knowledge, I think an outbox pattern would be my go-to as well.

Answer 5

There's a couple patterns designed for reliability:

• Retry
• Circuit Breaker

For large deployments, it is common for one or more of your microservices to have multiple instances running at the same time. That means a retry will help with the majority of the Quality of Service (QoS) needs in normal operations. Typically there is a timeout associated with terminating an existing request and retrying.

Sometimes a microservice simply gets stuck, and won't terminate the connection properly. In these cases the Circuit Breaker will cut off the request and the retry logic and return a default response (can be an error code). The circuit breaker is designed to prevent cascading failures.

The two patterns are not mutually exclusive.

---

For when a deployment goes wrong...

Combined with these architectural patterns is the concept of DevOps (and DevSecOps). The basic principle here is one of continual deployments. There are times where things go horribly wrong and you need to recover from that. DevOps addresses that by ensuring:

• Good Configuration Management (CM)
• Automated Blue-Green Deployments
• Automated Testing
• (for DevSecOps) Automated Security Testing
• Separate test and production deployments
• And most importantly Automated Roll-back

As good as every team attempts to be, sometimes something goes to production that the system as a whole was not quite ready for. There is a limited amount of time to get production back to a known stable build before the impact to users is completely unacceptable.

If your deployment system knows what the last version was, then it is simply a matter of deploying that version again, and then doing deeper testing and triage on the test network. When you get the kinks worked out, you can redeploy the fixed version.