Based from what I've understood about eventual consistency, all these
services (consumers) will receive the event at the same time and
process them separately which, in a good scenario, will lead to data
No, not necessarily. As I commented, we can't undo a sent email, so we still need a sort of "sequence". IPC over event-driven data management is not exempt of orchestation 1.
For instance, the email should not be sent unless the previous transactions finish successfully and the email service gets a proof of it. 3
However, what if a service fails to process the event? e.g. sudden
disconnect, database error, etc... What is a good pattern/practice to
handle these transaction failures?
Say hello to the fallacies of the distributed computing. They are what make things complicated and, as usual, there're not silver-bullets to deal with them.
Before starting our journey in search of the Lost Ark, we have to consider asking the organization first. Often, the solution is in how the organization faces these problems in the real world.
What everyone (departments) does when certain data is missing or incomplete?
We'll come to realise that different departments have different solutions whom, altogether, comprise the solution to be implemented.
Anyway, here some practices that could help us out with the strategy to follow.
Rather than ensuring that the system is in a consistent state all the time, instead we can accept that the system will get it at some point in the future. This approach is especially useful for long-living business operations.
The way for the system to reach the consistency varies from system to system. It might involve from automated processes to some kind of human intervention. For instance, the typical trying It again later or the contact with Customer Service.
Abort all the operations
Put the system back into a consistent state via compensating transactions. However, we have to take in account that, these transactions can fail too, what could lead us to a point where the inconsistency is even harder to get solved. And, again, we can not undo a sent email.
For a low number of transactions, this approach is feasible, because the number of compensating transactions is low too. If there were several business transactions involved in the IPC, handling one compensating transaction for each of them would be challenging.
If we go for compensating transactions, we'll find circuit breaker design pattern to be very useful -and mandatory I would dare to say-
The idea is to span multiple transactions within a single transaction, through an overall governing process known as Transaction Manager. A common algorithm for handling distributed transactions is Two-phase commit.
The distributed transactions's main concern is that they rely on locking the resources during its life-time, and as we know, things can go wrong for the Transaction Manager too.
If the Transaction Managers gets compromised, we can end up with several locks all across the differents bounded contexts, resulting in unexpected behaviors due to the enqueueing of the messages. 2
Decomposing operations. Why?
If you're decomposing an existing system, and find a collection of
concepts that really want to be within a single transaction boundary,
perhaps leave them till last.
In the line with the above arguments, Sam -in his book Building Microservices- state that, if we really, really can not afford the eventual consistency, we should avoid to split the operation now.
If we can not afford splitting certain operations into two or more transactions, it might comes to say that -probably- these transactions belong to the same bounded context, or -at least- to a cross-cutting context that remains to be modeled.
For example, in our case, we come to realise that transactions #1 and #2 are tightly related one another and probably both could belong to the same bounded context Accounts, Users, Register, whatever...
Consider placing both operations within the boundaries of the same transaction. It would make the whole operation easier to handle. Also weigth the level of criticality of each transaction. Probably, if transaction #2 fails, it should not compromise the whole operation. In case of doubts ask to the organization.
1: Not the kind of orchestration you think. I'm not talking about ESB's orchestation. I'm talking about making the services react to the proper event.
2: You might find interesting Sam Newman's opinions regarding distributed transactions.
3: Do check out David Parker's answer regarding this subject.