How do replicas of an app instance assure the task was done only once when the main instance fails?

Question

Let's assume i app have with basic flow:

1. Client send request to server
2. Server do some business logic
3. Server puts message into some message queue
4. Server returns success to the client
5. microservice A reads the message from message queue and process it - be it some email

microservice that generates and sends email to the client , or some audit/logging microservice that process the message and log the data somewher - be it whatever

Now my question is as follow: what happens when our microservice A fails, but we want to have as much uptime as possible and dont want to process data more than once?

The most basic solution ( but without much uptime ) is to have queue that requires confirmation from client before it deletes the message from queue. In that case the flow for microservice would be

1. Reads message
2. Proces it
3. send email/log the data whatever
4. confirm to the queue to delete the message

This is however flow without error, if this flow fails in the 2. step, the microservice just restarts and just do whole flow again, since message in the message queue stayed there and wasnt deleted.

Now what happens when the microservice fails after 4th step? Everything is done, email is sent/audit is logged but after that microservice fails and message in queue remains, thus if the microservice restarts, it sees message in queue and process same message twice, if we add some additional DB layer to the flow ( like storing what was processed and what not ) it can still fail before the microservice stores the data into the DB.

Another solution that has better uptime is to have load balancer with multiple instances of microservice A behind it, and these microservices shares the state between them, but as far as im aware, these microservices have to exchange heartbeat and confirm they processed the request from client - but since "processing" here means sending data somewhere, it can process it and then fail , so the heartbeat wont go on, replicated instances gets promoted to the main, reads the message again and process it again and now we are processing it multiple times, which may not be wanted at all in many cases.

I may be out of the loop and my solutions i wrote here may be completely wrong, but how are things like this dealed with? The more i read about distributed systems and so on the more questions i have.

This is not problem i am facing and the system does not have to use message queues/ or the microservices doesnt have to neccessary send email or log data - this is more of general problem question.

Thanks for help and answers!

Answer 1

ACID property of databases can be used for this. Database is shared, multi-instance, and manages its own uptime. This approach increases the number of components in the system, but if "email only once" is important, one would do something like this. Email protocol itself has problems, as we will see later.

1. Request comes to process item A
2. It is put into a database with timestamp. If item A itself has uniqueness, that is the primary key, otherwise the inbound API gateway generates a unique key for each incoming request. Assume the unique key is A.
3. N instances are running, instance i picks up the processing of item A. So it inserts an entry in a database table, "processingInstance = i, itemId = A". This table has a unique constraint on these itemId, so no other instance can add another constraint. If instance j tries, it will get a unique constraint violation.
4. After processing, instance i makes another database entry in another table : "processedBy = i, itemId = A, result = XYZ, timestamp = t". Another type of application picks up results from here, and sends email.

Points to be noted :

i. In step 3, instances pick up items not picked by any other instance. 2 instances might TRY to pick up the same item, but only one will succeed due to the unique constraint.

ii. A background application keeps monitoring results table. If any entry in incoming table that is 1 hour old does not have any entry in results table, do the following :

a. Ring alarm bells, let devops check. This is manual, but I'll explain how it will be kept to manageable levels.

b. If it was not picked up by any instance, instances are overloaded, or there is a bug. Number of instances can be increased, their CPU/ memory etc. can be increased, or bug can be fixed. Ideally and eventually, there should be no bug and instance scaling should be automatic, so this manual effort will go down as the application matures.

c. If it was picked up by instance i, go check i. It must be ill, or there should be a bug. Fix the bug. The illness should have been detected by instance probes, see why it was not done.

This step can be partially automated : if we are confident in the time limit of 1 hour, the database entry giving ownership of item A to instance i should be removed automatically. Another instance j will then pick up the item A.

These are the general directions, I might have missed some steps. Since we are closing loopholes, some more due diligence will have to be done. But ACID works well for this problem, and databases are able to scale well for this purpose generally. Shard them using uniqueness of A, if they don't scale.

Answer 2

If you want to learn about serious DS problems then read about Raft. It solves Distributed Consensus just as well as Lamport's awesome Paxos, but in a deliberately easier-to-understand way. Ok, back to the client transaction you proposed.

Technologies like email and SMS are inherently non-idempotent, and thus not a good match for what you're driving at. Let's set the controls of the Way Back machine to a gentler simpler time, when people sometimes communicated (flamed?) via NNTP Usenet messages. They look a lot like email.

Usenet gossip messages are idempotent. A client could send one message or N copies of it, with the same effect. How does that work? Recipients would de-dup on the Message-ID header field. Dups are silently discarded, so only the initial message counts. In your scenario, Server (or even Client) should specify a Message-ID.

Email systems can offer similar semantics, but there's no strong specification for MTAs or MUAs to offer it. So it is "nice to have" behavior, not solid enough to rely on.

---

An email sender can append a unique message ID to a persistent datastore before sending a message, and append a "success" record after it's sent.

Attempted resends can suppress dup sends by noticing the ID already appears in the datastore in a "success" record. If we were to log the ID, and then something were to happen during Send and we never log success, it's probably best to let some future invocation attempt a retry.

It's worth noting that SMTP verbs have the same ambiguity. After sending a DATA command, either we receive a {good, bad} status reply or we don't -- not a strong foundation on which to build idempotency.

---

Your question about message delivery boils down to these concerns:

• lost message is delivered zero times
• exactly once delivery
• multiple deliveries must be de-dup'd

The Kafka folks have done some excellent work on exactly-once delivery. You could do worse than to build atop Kafka.

Answer 3

1. Reads message
2. Proces it
3. send email/log the data whatever
4. confirm to the queue to delete the message

Just use a progress bar alike implementation by recording the failure/success of each step and additionally implement a failure recovery system that tracks failures, identifies failing cause by exchanging information with each system involved in the process and retries the process from the failed step after the removal of failure cause.

There was a joke about two teams of researchers trying to solve the problem of writing in imponderability, where one team invented a gravity proof pen to write with ink in outer space while the other team used a pencil.

Answer 4

This question, or more accurately any given answer to it, is prone to "but you could invest more effort and cover the gap this way". That's a recursive response, you can always invest more effort, but this isn't necessarily the more pragmatic approach and therefore I'm drawing a line at what I find meaningfully productive.

At the end of the day, when dealing with a distributed model, you either accept that your message doesn't get processed, or you accept that it might get processed more than once. Which one you should prefer depends on context. A bank will prefer to not process a transaction instead of doing it twice. An life-saving service will prefer sending two alerts instead of none.

The reasoning behind that sentiment is that you can't meaningfully confirm with 100% certainty that your observed state (as to whether the previous attempt succeeded) exactly matches reality.
As you pointed out, it's perfectly possible that the service processes the message but then drops dead before it relays the confirmation. There are many scenarios you can imagine here, and while none of them are likely to happen frequently, they're always possible.

The sanest way to tackle this problem is through message versioning and idempotency.

Message versioning ensure that you don't process messages out of order. It can, if you're so inclined, also be tailored to only allow currentversion+1 (and not currentversion+2), which is an optional safeguard against dropped messages (what if currentversion+1 was never processed?), but it comes at the risk of grinding the update process to a halt when a message has been dropped, as the system will outright reject any further updates on the queue.

Idempotency ensures that even if you end up processing the message more than once, it doesn't really matter, because a repeat doesn't meaningfully change the system. Idempotency is a really good goal to have, but it's contextual whether this works for you or not. If you deal in absolute values (e.g. "the new stock level is 150 items"). Processing that twice in a row makes no difference to the outcome.

But if your messages indicate a difference in value (e.g. "add 10 items to the current stock level"), it doesn't work because two of those messages will stack (adding 20 items in total). In this case, you'll have to hope that your versioning protects you from processing the message twice.

Can you come up with more complex solutions that try to cover the gap a bit more? Sure, depending on your context I can't exclude that. But in my 5ish years of experience in working with distributed systems, it tends not to be worth it on a cost-benefit scale, where the effort, complexity, and possibility of making mistakes becomes more of a pain than a solution.