6

What is the best way of preventing of events from being handled more than once?

Imagine a system where customers can place orders and when successful an OrderCreated event is raised. There are some separate processes which listen to events on a message bus.

One process is to send an email to the customer, assume there is a requirement that the customer MUST be sent an email.

I imagine the email sending process would work in the following way:

  1. Send email
  2. Persist handled id
  3. Acknowledge the message bus

But if persisting the handled id fails the customer would get multiple emails.

The only other solution I thought of was to do the following:

  1. Persist attempt to handle event id
  2. Send email
  3. Persist handled event id
  4. Acknowledge the message bus

This would allow me to check whether the event was attempted to be handled before, and if so raise some error and ask a human whether to try and resend.

Are there any better ways to handle problems like these? I feel like this sort of thing would be a common problem but I can't find a robust solution.

  • Something like you describe is already done for you by ACID databases. – Kevin Krumwiede Dec 18 '16 at 21:16
  • 1
    How can ACID databases help here? What does the transaction commit? There is no way to rollback sending the email if the dB transaction is not committed. – Nick Williams Dec 18 '16 at 21:38
  • It will allow you to detect the failure to persist the event ID and avoid sending multiple emails. – Kevin Krumwiede Dec 18 '16 at 22:17
  • Sorry Kevin I am not following, how does it do that? You could begin a transaction, persist the id (insert/update a table), send the email, attempt to commit the transaction but the commit fails. For instance if the application server lost connection the database server. Given persisting the last handled id is a single statement it is already atomic, consistant and durable. The message bus would deliver events serially so the action is isolated. I think I might be misunderstanding your solution? – Nick Williams Dec 18 '16 at 22:27
  • I get that I can determine that the transaction failed. But if the event was redelivered, the event handler would not know that this event already created an email. bit.ly/1xiuGAb – Nick Williams Dec 18 '16 at 22:33
5

What is the best way of preventing of events from being handled more than once?

The best way: write your domain model such that at most once handling is built into the business logic.

Marc de Graauw, from Nobody Needs Reliable Messaging:

Stating that it is important on the business level that every message is received exactly once, like it is in order processing, means that every message constitutes a unique business transaction.... if every message is a unique business transaction, clearly there must be a unique id on the business level.... if we need such a unique token on the business level, the business level should assert it's uniqueness. Uniqueness on the business level should not be dependent on the transient uniqueness on the message level, but must be a persistent feature of the business message, and business semantics must guarantee it.

But that's not the sort of example you are considering;

One process is to send an email to the customer, assume there is a requirement that the customer MUST be sent an email.

In a situation where MUST; I'd normally look for at-least-once handling of the event. The process keeps track of both which emails are required, and which have been acknowledged.

For example, the process subscribing to OrderCreated, EmailSent, and EmailTimedOut. On "order created", we dispatch a command to send the email, and another to publish the timeout event. Upon receiving the timeout, we check to see if the EmailSent message has appeared -- if not, we resend.

Are there any better ways to handle problems like these?

Review in detail the cost to the business of the different failure modes. If you MUST send the email, then you need the ack to know you have done so. If the ack can get lost, then you need to resend -- what's the cost to the business of sending that email more than once? Don't sign up for expensive perfection when failure is rare and low cost.

I feel like this sort of thing would be a common problem but I can't find a robust solution.

Jonathan Oliver: Messaging: At Least Once Delivery

One of the primary reasons, if not the primary reason, that messaging systems offer at-least-once delivery rather than exactly-once delivery is that we run into limitations as expressed in the CAP theorem—all “nodes” cannot possibly have a globally consistent snapshot and be available and partition tolerant. There is no unfailing “global brain”.

See also: https://stackoverflow.com/q/416551/54734

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.