22

Assume we have a function that updates a User's password.

Once the 'Update Password' button is clicked, an UpdatePasswordEvent is sent to a topic where 3 other services are subscribed:

  1. A service that actually updates the User's password
  2. A service that updates the password history of the user
  3. A service that sends out an e-mail informing the user that his password has been changed.

Based from what I've understood about eventual consistency, all these services (consumers) will receive the event at the same time and process them separately which, in a good scenario, will lead to data being consistent.

However, what if a service fails to process the event? e.g. sudden disconnect, database error, etc... What is a good pattern/practice to handle these transaction failures?

I was thinking of creating a RollbackTopic where if any event fails to be processed, a RollbackEvent will be created in a topic where "rollback services" will do it's job and revert the data back

  • 11
    You can't undo a sent email :-) – Laiv Aug 1 '17 at 17:49
  • 1
    Because all of them should be part of the same service. Micro service are opposed to monoliths, it doesn't mean you have to design them as little as "physically" possible. Though this isn't directly related, you should read this question and the two top answers : softwareengineering.stackexchange.com/questions/339230/… – Walfrat Aug 2 '17 at 7:43
  • 1
    You might like to consider updating the user's password in the database synchronously, so that you provide immediate feedback to the user, and triggering other services asynchronously by emitting a message that the password changed on a topic, so that your message doesn't have to contain the password. – cr3 Aug 3 '17 at 16:24
  • Is the e-mail to tell the user that the transaction has completed, or is it there to tell the user that someone (hopefully them) has changed the password. “If it was not you, then you need to act”. If the 2nd then just send e-mail now, as best you can. – ctrl-alt-delor Feb 1 '18 at 11:05
29

Based from what I've understood about eventual consistency, all these services (consumers) will receive the event at the same time and process them separately which, in a good scenario, will lead to data being consistent.

No, not necessarily. As I commented, we can't undo a sent email, so we still need a sort of "sequence". IPC over event-driven data management is not exempt of orchestation 1.

For instance, the email should not be sent unless the previous transactions finish successfully and the email service gets a proof of it. 3

However, what if a service fails to process the event? e.g. sudden disconnect, database error, etc... What is a good pattern/practice to handle these transaction failures?

Say hello to the fallacies of the distributed computing. They are what make things complicated and, as usual, there're not silver-bullets to deal with them.

Before starting our journey in search of the Lost Ark, we have to consider asking the organization first. Often, the solution is in how the organization faces these problems in the real world.

What everyone (departments) does when certain data is missing or incomplete?

We'll come to realise that different departments have different solutions whom, altogether, comprise the solution to be implemented.

Anyway, here some practices that could help us out with the strategy to follow.

Eventual consistency

Rather than ensuring that the system is in a consistent state all the time, instead we can accept that the system will get it at some point in the future. This approach is especially useful for long-living business operations.

The way for the system to reach the consistency varies from system to system. It might involve from automated processes to some kind of human intervention. For instance, the typical trying It again later or the contact with Customer Service.

Abort all the operations

Put the system back into a consistent state via compensating transactions. However, we have to take in account that, these transactions can fail too, what could lead us to a point where the inconsistency is even harder to get solved. And, again, we can not undo a sent email.

For a low number of transactions, this approach is feasible, because the number of compensating transactions is low too. If there were several business transactions involved in the IPC, handling one compensating transaction for each of them would be challenging.

If we go for compensating transactions, we'll find circuit breaker design pattern to be very useful -and mandatory I would dare to say-

Distributed transactions

The idea is to span multiple transactions within a single transaction, through an overall governing process known as Transaction Manager. A common algorithm for handling distributed transactions is Two-phase commit.

The distributed transactions's main concern is that they rely on locking the resources during its life-time, and as we know, things can go wrong for the Transaction Manager too.

If the Transaction Managers gets compromised, we can end up with several locks all across the differents bounded contexts, resulting in unexpected behaviors due to the enqueueing of the messages. 2

Decomposing operations. Why?

If you're decomposing an existing system, and find a collection of concepts that really want to be within a single transaction boundary, perhaps leave them till last.

Sam Newman

In the line with the above arguments, Sam -in his book Building Microservices- state that, if we really, really can not afford the eventual consistency, we should avoid to split the operation now.

If we can not afford splitting certain operations into two or more transactions, it might comes to say that -probably- these transactions belong to the same bounded context, or -at least- to a cross-cutting context that remains to be modeled.

For example, in our case, we come to realise that transactions #1 and #2 are tightly related one another and probably both could belong to the same bounded context Accounts, Users, Register, whatever...

Consider placing both operations within the boundaries of the same transaction. It would make the whole operation easier to handle. Also weigth the level of criticality of each transaction. Probably, if transaction #2 fails, it should not compromise the whole operation. In case of doubts ask to the organization.


1: Not the kind of orchestration you think. I'm not talking about ESB's orchestation. I'm talking about making the services react to the proper event.

2: You might find interesting Sam Newman's opinions regarding distributed transactions.

3: Do check out David Parker's answer regarding this subject.

  • 3
    Very good answer. I would only emphasize the importance of taking into account risks which come when using distributed transactions - mainly resource locking producing deadlocks and systems-halts. On an e-commerce product I worked on about 3 years ago we had to replace DTs with messaging system, because with the amount of users available in the systems, the system was very prone to errors. Problems with DTs mostly occur when a user base grows. – Andy Aug 2 '17 at 4:11
7

In your case you cannot just process all three things at once. What you need is a process. Here is an extremely simplified example:

Command and event orchestration

It is important to know that state altering operations MUST be always made on a consistent entity. Unless you can guarantee strong consistency, it has to be made on a master record.

Your system should guarantee that before any event is raised in your system changes MUST be persisted with transactional safety first. This is to ensure that a raised event really is a confirmation of what really happened.

There are several tricky parts of the process as is and I am going to ignore the obvious ones - such as: What if your database server dies when persisting a user with changed password? You simply issue the UpdatePassword again. However, some parts need to be taken care of by you, and these are:

  • handling message duplication,
  • handling e-mail sending.

In a system, process orchestrator (PO) is nothing else but another entity, which contains internal state - in the literal term as well - and allows transitions between the states, effectively acting as some sort of state machine. Thanks to the internal state you can remove message duplication processing.

When the PO is in a New state and processes UserPasswordHasBeenUpdated, it changes its state to UserPasswordHasBeenUpdated (or whichever state name works for you). Should the PO still be in a UserPasswordHasBeenUpdated and another UserPasswordHasBeenUpdated would arrive, the PO would completely ignore the message, knowing it's a duplication. Similar mechanism would be implemented for other states as well.

Handling the actual sending of the e-mail is a little bit trickier. Here you have two options:

  1. send it at most once,
  2. send it at least once.

Send it at most once

With this option, when the PO reached UserPasswordHistoryHasBeenSaved state a command to send an e-mail is dispatched as a reaction to the state change. Your system would ensure the UserPasswordHistoryHasBeenSaved state would be persisted before sending the e-mail, i.e. duplicated message would not trigger the e-mail sending again. With this approach you ensure that the correct state is saved for the PO but cannot guarantee any following operation.

Send it at least once

This is what I would go for.

Instead of saving UserPasswordHistoryHasBeenSaved and sending out the e-mail as a reaction to it, you try to send the e-mail first. If the sending operation fails the state of the PO is never changed to UserPasswordHistoryHasBeenSaved and another message of the same type is still processed. Should sending of the e-mail actually succeed but your system would fail during persisting of the PO with its new UserPasswordHistoryHasBeenSaved state, another message of the UserPasswordHistoryHasBeenSaved would once again trigger the command to send out the e-mail and the user would have received it multiple times.

In your case you want to make sure the user actually receives the e-mail. That's why I would choose the second options over the first.

2

Queuing systems are not quite as fragile as you might think.

If we were writing all three processes to a relational db, we might use a transaction to handle a mid processes failure.

Without the final commit the partial work would be discarded.

In a queue bases system you will have a similar options when you read a message from the queue to handle mid process failures.

Amazon SQS for example simply hides messages which are read. unless a final Delete command is sent the message will reappear or be put in a dead letter queue.

You can implement similar 'transactions' in various ways, essentially holding a copy of the message untill you receive confirmation of successful processing. If the confirmation is not received in time. you can send the message again or keep it for manual attention.

Potentially you could create a 'rollback service' which monitored these erred messages, knew about related messages and past state and performed a rollback.

However! It is usually better just to resend the erred messages. After all these tend to be edge cases. Either a server catastrophically failed or there was a bug in the handling a particular message type.

Once alerted to the error the service can be repaired and the messages processed successfully. Bringing the system as a whole back to a consistent state.

2

What you're facing here is the two generals problem. In essence: how can you be sure a message is received and a response to that message occurs? In many cases, a perfect solution does not exist. In fact, in a distributed system it is often impossible to get an exactly-once delivery of messages.

A first obvious remark is that the service that changes the password should be sending out the password-change event. This way the password history and mail sending services are only triggered when the password actually changes, regardless of why it changed.

To actually solve your problem I would not consider distributed transactions, but instead look in the direction of at-least-once message delivery and idempotent processing.

  • At Least Once

    To make sure the password-change event is actually seen by all consumers you need to use a durable communication channel where messages can be consumed in an "at least once" style. The consumers only acknowledge a message as consumed when they have fully processed it. If, for example, the password history service crashes while writing a history entry, it will reread the same password-change event after restart and try again, acknowledging that event as read-only after it has itself written to the history. You should choose a message queue solution based on its ability to resend messages until they are acknowledged.

  • Idempotence

    After achieving at-least-once delivery there is the problem of duplicate actions occurring when a message was partially processed prior to the consumer being interrupted and then reprocessed later on. That should be solved by designing each service so it is idempotent. Either the writes it performs can occur multiple times without adverse effects, or it keeps its own store of which actions it took and avoids performing an action more than once. In the case of mail sending, you'll find it probably isn't worth trying to get it to behave idempotently and just be fine with occasionally a mail being sent twice.

In any case, be careful how micro you make your services. Does your password history service really need to be independent from the password change service?

1

I disagree with a lot of the answers.

  1. Send the e-mail now “Someone has changed your password. If it was you then you don't need to do anything. If not panic.” This will arrive when it arrives.
  2. Change the password. Though you have eventual consistency. You want to ensure that this session sees changes made by the user.

There are other consistency promises that you can add.

  • Ensure that changes happen in time order.
  • Ensure that a user never sees a roll-back, but other users may still not see the change.
  • There are others

These additional consistencies will need to be implemented depending on the deeds of the application.


I have no idea what you mean by “updates the history“ but please never change history. If you are just extending the DAG, then this should cause the change in the current state. They are not independent. If they are then you can not rely on history reflecting what happened. (and last but not least, don't store passwords see how not to store passwords )

  • If you can send the email at the beginning then your approach is fine. If you have to send something alongside with the email. Maybe a sort of link/data that only can be obtained after the consistency is achieved, then you can not send the email first. That is what I commented as consider asking the organization first.. You are likely right. However, I have found to be important to condition those events we cannot be undo. For example notifications to the end-user. Notification lying on the real state of the user's data might case a bad impression. – Laiv Feb 1 '18 at 14:00
  • That said, for this specific scenario (password change notification), I agreed with this approach. As soon as it suites the requirements. – Laiv Feb 1 '18 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.