Prevent duplicate event handling

Question

What is the best way of preventing of events from being handled more than once?

Imagine a system where customers can place orders and when successful an OrderCreated event is raised. There are some separate processes which listen to events on a message bus.

One process is to send an email to the customer, assume there is a requirement that the customer MUST be sent an email.

I imagine the email sending process would work in the following way:

1. Send email
2. Persist handled id
3. Acknowledge the message bus

But if persisting the handled id fails the customer would get multiple emails.

The only other solution I thought of was to do the following:

1. Persist attempt to handle event id
2. Send email
3. Persist handled event id
4. Acknowledge the message bus

This would allow me to check whether the event was attempted to be handled before, and if so raise some error and ask a human whether to try and resend.

Are there any better ways to handle problems like these? I feel like this sort of thing would be a common problem but I can't find a robust solution.

Answer 1

What is the best way of preventing of events from being handled more than once?

The best way: write your domain model such that at most once handling is built into the business logic.

Marc de Graauw, from Nobody Needs Reliable Messaging:

Stating that it is important on the business level that every message is received exactly once, like it is in order processing, means that every message constitutes a unique business transaction.... if every message is a unique business transaction, clearly there must be a unique id on the business level.... if we need such a unique token on the business level, the business level should assert it's uniqueness. Uniqueness on the business level should not be dependent on the transient uniqueness on the message level, but must be a persistent feature of the business message, and business semantics must guarantee it.

But that's not the sort of example you are considering;

One process is to send an email to the customer, assume there is a requirement that the customer MUST be sent an email.

In a situation where MUST; I'd normally look for at-least-once handling of the event. The process keeps track of both which emails are required, and which have been acknowledged.

For example, the process subscribing to OrderCreated, EmailSent, and EmailTimedOut. On "order created", we dispatch a command to send the email, and another to publish the timeout event. Upon receiving the timeout, we check to see if the EmailSent message has appeared -- if not, we resend.

Are there any better ways to handle problems like these?

Review in detail the cost to the business of the different failure modes. If you MUST send the email, then you need the ack to know you have done so. If the ack can get lost, then you need to resend -- what's the cost to the business of sending that email more than once? Don't sign up for expensive perfection when failure is rare and low cost.

I feel like this sort of thing would be a common problem but I can't find a robust solution.

Jonathan Oliver: Messaging: At Least Once Delivery

One of the primary reasons, if not the primary reason, that messaging systems offer at-least-once delivery rather than exactly-once delivery is that we run into limitations as expressed in the CAP theorem—all “nodes” cannot possibly have a globally consistent snapshot and be available and partition tolerant. There is no unfailing “global brain”.

See also: https://stackoverflow.com/q/416551/54734