Based on what I've understood about eventual consistency, all these
services (consumers) will receive the event at the same time and
process them separately which, in a good scenario, will lead to data
Yes, but as I commented, we can't undo an email notification so we still need a sort of "sequence". Event-driven data management is not exempt of some sort of orchestration 1.
For instance, the email should not be sent unless the previous transactions finish successfully and the email service gets proof of it.3
However, what if a service fails to process the event? e.g. sudden
disconnect, database error, etc... What is a good pattern/practice to
handle these transaction failures?
Say Hello! to the fallacies of the distributed computing. They are what make things complicated and, as usual, there're no silver bullets to deal with them.
Before starting our particular search of the Lost Ark, we have to consider asking the organization first. Often, the solution is in how the organization deal with these problems in the real world.
What do we (the company) do when certain data is missing or incomplete?
We'll come to realise that different departments have different ways to handle the situation. These ways guide the final solution.
Here some practices that could help.
Instead of ensuring that the system is in a consistent state all the time, we can accept that the system will be at some point in the future. This approach is especially useful for long-living business operations.
The way for the system to reach consistency varies from system to system. It might involve automated processes or some kind of human intervention. For instance, the typical trying It again later or the contact with Customer Service.
Abort all the operations
Put the system back into a consistent state via compensating transactions. However, we have to take into account that, these transactions can fail too, which could lead us to a point where the inconsistency is even harder to get solved. And, again, we can not undo a sent email.
For a low number of transactions, this approach is feasible, because the number of compensating transactions is low too. If there were several business transactions involved in the IPC, handling one compensating transaction for each of them would be challenging.
If we go for compensating transactions, we'll find circuit breaker design pattern to be very useful.
The idea is to span multiple transactions within a single transaction, through an overall governing process known as Transaction Manager. A common algorithm for handling distributed transactions is Two-phase commit.
The main concern is that transactions here rely on locking resources during the transaction lifetime, and as we know, things can go wrong for the Transaction Manager too. If the Transaction Manager gets compromised, we could end up with several locks all across the different bounded contexts, resulting in unexpected behaviours across the whole system. 2
Decomposing operations. Why?
If you're decomposing an existing system, and find a collection of
concepts that really want to be within a single transaction boundary,
perhaps leave them till last.
In the line with the above arguments, Sam -in his book Building Microservices- states that, if we really, really can not afford the eventual consistency, we should avoid splitting the operation now.
If we can not afford to split certain operations into two or more transactions, it might come to say that -probably- these transactions belong to the same bounded context, or -at least- to an emergent and cross-cutting context.
For example, in our case, we come to realise that transactions #1 and #2 are tightly related to one another and probably both could belong to the same bounded context Accounts, Users, Register, ...
Consider placing both operations within the boundaries of the same transaction. It would make the whole operation easier to handle. Also, weigh the level of criticality of each transaction. Probably, if transaction #2 fails, it should not compromise the whole operation. In case of doubts ask the organization.
1: I'm not talking about ESB's orchestration. I'm talking about making services react to the proper event. It's rather a choreography.
2: You might find interesting Sam Newman's opinions regarding distributed transactions.
3: See Andy's answer regarding this subject.