Microservices: Handling eventual consistency

Question

Assume we have a function that updates a User's password.

Once the 'Update Password' button is clicked, an UpdatePasswordEvent is sent to a topic where 3 other services are subscribed:

1. A service that actually updates the User's password
2. A service that updates the password history of the user
3. A service that sends out an e-mail informing the user that his password has been changed.

Based from what I've understood about eventual consistency, all these services (consumers) will receive the event at the same time and process them separately which, in a good scenario, will lead to data being consistent.

However, what if a service fails to process the event? e.g. sudden disconnect, database error, etc... What is a good pattern/practice to handle these transaction failures?

I was thinking of creating a RollbackTopic where if any event fails to be processed, a RollbackEvent will be created in a topic where "rollback services" will do it's job and revert the data back

Answer 1

Based on what I've understood about eventual consistency, all these services (consumers) will receive the event at the same time and process them separately which, in a good scenario, will lead to data being consistent.

Yes, but as I commented, we can't undo an email notification so we still need a sort of "sequence". Event-driven data management is not exempt of some sort of orchestration ¹.

For instance, the email should not be sent unless the previous transactions finish successfully and the email service gets proof of it.³

However, what if a service fails to process the event? e.g. sudden disconnect, database error, etc... What is a good pattern/practice to handle these transaction failures?

Say Hello! to the fallacies of the distributed computing. They are what make things complicated and, as usual, there're no silver bullets to deal with them.

Before starting our particular search of the Lost Ark, we have to consider asking the organization first. Often, the solution is in how the organization deal with these problems in the real world.

What do we (the company) do when certain data is missing or incomplete?

We'll come to realise that different departments have different ways to handle the situation. These ways guide the final solution.

Here some practices that could help.

Eventual consistency

Instead of ensuring that the system is in a consistent state all the time, we can accept that the system will be at some point in the future. This approach is especially useful for long-living business operations.

The way for the system to reach consistency varies from system to system. It might involve automated processes or some kind of human intervention. For instance, the typical trying It again later or the contact with Customer Service.

Abort all the operations

Put the system back into a consistent state via compensating transactions. However, we have to take into account that, these transactions can fail too, which could lead us to a point where the inconsistency is even harder to get solved. And, again, we can not undo a sent email.

For a low number of transactions, this approach is feasible, because the number of compensating transactions is low too. If there were several business transactions involved in the IPC, handling one compensating transaction for each of them would be challenging.

If we go for compensating transactions, we'll find circuit breaker design pattern to be very useful.

Distributed transactions

The idea is to span multiple transactions within a single transaction, through an overall governing process known as Transaction Manager. A common algorithm for handling distributed transactions is Two-phase commit.

The main concern is that transactions here rely on locking resources during the transaction lifetime, and as we know, things can go wrong for the Transaction Manager too. If the Transaction Manager gets compromised, we could end up with several locks all across the different bounded contexts, resulting in unexpected behaviours across the whole system. ²

Decomposing operations. Why?

If you're decomposing an existing system, and find a collection of concepts that really want to be within a single transaction boundary, perhaps leave them till last.

Sam Newman

In the line with the above arguments, Sam -in his book Building Microservices- states that, if we really, really can not afford the eventual consistency, we should avoid splitting the operation now.

If we can not afford to split certain operations into two or more transactions, it might come to say that -probably- these transactions belong to the same bounded context, or -at least- to an emergent and cross-cutting context.

For example, in our case, we come to realise that transactions #1 and #2 are tightly related to one another and probably both could belong to the same bounded context Accounts, Users, Register, ...

Consider placing both operations within the boundaries of the same transaction. It would make the whole operation easier to handle. Also, weigh the level of criticality of each transaction. Probably, if transaction #2 fails, it should not compromise the whole operation. In case of doubts ask the organization.

---

^{1: I'm not talking about ESB's orchestration. I'm talking about making services react to the proper event. It's rather a choreography.}

^{2: You might find interesting Sam Newman's opinions regarding distributed transactions.}

^{3: See Andy's answer regarding this subject.}

Answer 2

In your case you cannot just process all three things at once. What you need is a process. Here is an extremely simplified example:

It is important to know that state altering operations MUST be always made on a consistent entity. Unless you can guarantee strong consistency, it has to be made on a master record.

Your system should guarantee that before any event is raised in your system changes MUST be persisted with transactional safety first. This is to ensure that a raised event really is a confirmation of what really happened.

There are several tricky parts of the process as is and I am going to ignore the obvious ones - such as: What if your database server dies when persisting a user with changed password? You simply issue the UpdatePassword again. However, some parts need to be taken care of by you, and these are:

• handling message duplication,
• handling e-mail sending.

In a system, process orchestrator (PO) is nothing else but another entity, which contains internal state - in the literal term as well - and allows transitions between the states, effectively acting as some sort of state machine. Thanks to the internal state you can remove message duplication processing.

When the PO is in a New state and processes UserPasswordHasBeenUpdated, it changes its state to UserPasswordHasBeenUpdated (or whichever state name works for you). Should the PO still be in a UserPasswordHasBeenUpdated and another UserPasswordHasBeenUpdated would arrive, the PO would completely ignore the message, knowing it's a duplication. Similar mechanism would be implemented for other states as well.

Handling the actual sending of the e-mail is a little bit trickier. Here you have two options:

1. send it at most once,
2. send it at least once.

Send it at most once

With this option, when the PO reached UserPasswordHistoryHasBeenSaved state a command to send an e-mail is dispatched as a reaction to the state change. Your system would ensure the UserPasswordHistoryHasBeenSaved state would be persisted before sending the e-mail, i.e. duplicated message would not trigger the e-mail sending again. With this approach you ensure that the correct state is saved for the PO but cannot guarantee any following operation.

Send it at least once

_{This is what I would go for.}

Instead of saving UserPasswordHistoryHasBeenSaved and sending out the e-mail as a reaction to it, you try to send the e-mail first. If the sending operation fails the state of the PO is never changed to UserPasswordHistoryHasBeenSaved and another message of the same type is still processed. Should sending of the e-mail actually succeed but your system would fail during persisting of the PO with its new UserPasswordHistoryHasBeenSaved state, another message of the UserPasswordHistoryHasBeenSaved would once again trigger the command to send out the e-mail and the user would have received it multiple times.

In your case you want to make sure the user actually receives the e-mail. That's why I would choose the second options over the first.

Answer 3

What you're facing here is the two generals problem. In essence: how can you be sure a message is received and a response to that message occurs? In many cases, a perfect solution does not exist. In fact, in a distributed system it is often impossible to get an exactly-once delivery of messages.

A first obvious remark is that the service that changes the password should be sending out the password-change event. This way the password history and mail sending services are only triggered when the password actually changes, regardless of why it changed.

To actually solve your problem I would not consider distributed transactions, but instead look in the direction of at-least-once message delivery and idempotent processing.

• At Least Once

  To make sure the password-change event is actually seen by all consumers you need to use a durable communication channel where messages can be consumed in an "at least once" style. The consumers only acknowledge a message as consumed when they have fully processed it. If, for example, the password history service crashes while writing a history entry, it will reread the same password-change event after restart and try again, acknowledging that event as read-only after it has itself written to the history. You should choose a message queue solution based on its ability to resend messages until they are acknowledged.

• Idempotence

  After achieving at-least-once delivery there is the problem of duplicate actions occurring when a message was partially processed prior to the consumer being interrupted and then reprocessed later on. That should be solved by designing each service so it is idempotent. Either the writes it performs can occur multiple times without adverse effects, or it keeps its own store of which actions it took and avoids performing an action more than once. In the case of mail sending, you'll find it probably isn't worth trying to get it to behave idempotently and just be fine with occasionally a mail being sent twice.

In any case, be careful how micro you make your services. Does your password history service really need to be independent from the password change service?

Answer 4

Queuing systems are not quite as fragile as you might think.

If we were writing all three processes to a relational db, we might use a transaction to handle a mid processes failure.

Without the final commit the partial work would be discarded.

In a queue bases system you will have a similar options when you read a message from the queue to handle mid process failures.

Amazon SQS for example simply hides messages which are read. unless a final Delete command is sent the message will reappear or be put in a dead letter queue.

You can implement similar 'transactions' in various ways, essentially holding a copy of the message untill you receive confirmation of successful processing. If the confirmation is not received in time. you can send the message again or keep it for manual attention.

Potentially you could create a 'rollback service' which monitored these erred messages, knew about related messages and past state and performed a rollback.

However! It is usually better just to resend the erred messages. After all these tend to be edge cases. Either a server catastrophically failed or there was a bug in the handling a particular message type.

Once alerted to the error the service can be repaired and the messages processed successfully. Bringing the system as a whole back to a consistent state.

Answer 5

I disagree with a lot of the answers.

1. Send the e-mail now “Someone has changed your password. If it was you then you don't need to do anything. If not panic.” This will arrive when it arrives.
2. Change the password. Though you have eventual consistency. You want to ensure that this session sees changes made by the user.

There are other consistency promises that you can add.

• Ensure that changes happen in time order.
• Ensure that a user never sees a roll-back, but other users may still not see the change.
• There are others

These additional consistencies will need to be implemented depending on the deeds of the application.

---

I have no idea what you mean by “updates the history“ but please never change history. If you are just extending the DAG, then this should cause the change in the current state. They are not independent. If they are then you can not rely on history reflecting what happened. (and last but not least, don't store passwords see how not to store passwords )

Answer 6

This case can be simply resolved by the topic itself. You only need to configure the topic to not commit the offset until it receives the consumer acknowledgment. If your broker doesn't support that you should consider other technologies like Kafka or RabbitMQ. https://www.rabbitmq.com/confirms.html#acknowledgement-modes.

Regarding consistency, in this case, eventual consistency still applies. If your requirements don't tolerate that level of consistency you can implement a distributed transaction using the two phases commit.

Also, you should review the decision of separating all these services or at least the services which host the data.