6

If I have a service that consumes events of type A and emits events of type B, how can I make sure that event B is only emitted once for every event A? Should the service also subscribe to events of type B to see that an event B with a same correlation ID/with a reference to the A event wasn't written yet? Seems like a lot of extra responsibilities for the service.

5
  • In general, exactly-once message-passing is hard. It's easier to make the end consumer of events (i.e. the consumer of event B) idempotent.
    – MikeFHay
    Sep 18, 2018 at 13:39
  • In the question title you use the term "idempotent", but you are asking about processing events only one - which is something quite different. Processing events in an idempotent way is actually lifting that restriction that an event must be processed only once. You do that by ensuring that processing the event twice has the same result as processing it only once, so you can make delivery reliable at the expense of duplicates in cases of trouble. Sep 19, 2018 at 6:20
  • @Hans-PeterStörr I'm looking to idempotently process my duplicate event of type A to not publish a duplicate event of type B. Sep 22, 2018 at 20:24
  • @MihailMalostanidis I still think this is misleading. A proper use of idempotency would be to make your distributed system indifferent against duplicated events B. See my answer, and MikeFHay's comment. Sep 25, 2018 at 19:26
  • It's indifferent in every aspect other than performance. Sep 26, 2018 at 7:58

4 Answers 4

5

Making sure events are processed once and only once in a distributed system is really really hard.

But, we are talking network failure and program crashes. In a normal message queue system, if you emit B when you get A and set your queues up correctly and have no major bugs you shouldnt worry overly.

Having said that, when you have outages it will be a possibility. So you should have systems in place to pick up on significant duplicate events, say taking payment twice, after the fact and correct them.

4
  • Would some sort of journaling transaction system be effective for implementing this? For example, if you keep a record of having emitted event B for a given event A (assuming it has some sort of unique transaction ID)? Then you can check the journal to see if you've already emitted the event after a crash has occurred. Sep 13, 2018 at 1:52
  • sure, but then your system is checking that central log all the time and you have lost the advantages of a distributed system
    – Ewan
    Sep 13, 2018 at 6:10
  • Well, you haven't lost all of the advantages. Many distributed systems have a central gateway that distributes messages; you could put your journaling there. See microservices.io/patterns/apigateway.html Sep 13, 2018 at 15:01
  • @RH hmm, not so sure about that
    – Ewan
    Sep 13, 2018 at 15:42
1

This is possible with the aid of Activities. Pat Helland discusses about Activities in his paper Life beyond Distributed Transactions: an Apostate’s Opinion. Vaughn Vernon also talks about this in this video.

The idea is that a receiver of a message should store in it's localstorage the IDs of all the messages that it has seen. If it receive a message that it has already seen then it ignores it. If the received has not seen the message then it processes it and then emit its new own messages.

In Event sourcing this means that the new messages should contain in their body the ID of the message that was the cause of those new messages.

So, in your case, message B should contain the ID of message A. So, when the message A is retried, the service replays all the previous messages that it has emitted, including the message B and then it sees that it has already processed the message A and ignores it.

This technique, combined with at least one delivery, gives you what you want, in the most scalable way.

11
  • But why would a sender ever send two messages with the same ID? Or for that matter, why would any other sender ever send a message having the same ID as some other message in the system? Sep 13, 2018 at 14:58
  • @RobertHarvey when it retries it Sep 13, 2018 at 14:59
  • @RobertHarvey Or for that matter, why would any other sender ever send a message having the same ID as some other message in the system? - I don't get this question. Sep 13, 2018 at 15:02
  • @RobertHarvey the OP whats to know how to not "re-do" a message Sep 13, 2018 at 15:09
  • Yes, I understood his question. I got confused when you said that " If it receive a message that it has already seen then it ignores it," which suggests the presence of two messages with the same ID. But then you clarified by saying that the retry message actually has its own ID, and that you're storing the origination ID in the body of the new message. Sounds like that might actually work, though it does complicate the messaging infrastructure a bit. Sep 13, 2018 at 15:18
1

It seems you are mixing up two quite different ideas for event processing here. You can either

  1. try hard to both transmit/process each event at least once, and at most once, even under the most exotic failures. These are two hard problems which both are only solvable to some extend in distributed systems.
  2. try hard to transmit/process each event at least once, and design your events so that it doesn't matter when you receive several events for one domain change. For this, idempotency of the event processing is a good approach.

The second solution is often easier and (sometimes much) more performant, at the expense of having to think hard about the structure and meaning of your events. You'll probably also have to design your distributed system so that the order, in which the events are received, doesn't matter, too. And that additional events triggered as the result of the second arrival of an event are OK, too.

One way to do this would be to keep all event IDs received in a reasonable timeframe, and discard duplicates. But you could also stucture your events so that it doesn't matter if they are received twice. That's where idempotency comes in: an operation (the event processing) doesn't change anything, anymore, when it is repeated more than once. For example, if you receive an "user adds street address" event, and the user already has that address, you can naturally just discard that event.

In your example, this means that you could make sure that the service emits a type B event for each type A event, but you can omit any special precautions that it never ever emits two type B events in the case of failures, since you have designed your events so that this wouldn't do any harm (besides a little more data being transmitted). You arrange things that events normally get received once, but in the case of failures it can happen that it's received two or more times.

0

If I have a service that consumes events of type A and emits events of type B, how can I make sure that event B is only emitted once for every event A?

The only way I know to do that is to ensure that all consumers of event B are part of the same transaction as event A.

In a distributed world? no chance.

At most once delivery is trivial -- just swallow event B without ever emitting it.

At least once delivery is manageable.

Exactly once delivery is beyond economically viable.

In most cases, the important points are (a) to make best effort to ensure that consumers recognize events that have been successfully processed, and (b) to have protocols in place to handle circumstances when events have been processed more than once.

(Example: on airplanes, we are legally limited to one passenger per seat. But it may sometimes happen that two passengers are assigned to the same seat. Even in this crisis, the plane usually manages to reach its destination....)

With just at least once delivery, it seems like this would cause an amplification of duplicate messages?

It certainly could - I think that depends on what synchronization points follow, and what strategies are applied by the handlers when they recognize a duplicate message.

For instance, if a receiver is processing messages in batches, then it might receive two copies of a message, and recognize the duplication.

On the other hand, the duplicated message may be an indication that an acknowledgement was lost in transit, in which case that acknowledgement should be resent, so that the upstream participant can know that the at-least-once requirement has been met.

A good starting point for this sort of thing is Udi Dahan's Reliable Messaging Without Distributed Transactions. You may recognize some similarities to the Pat Helland paper referenced by Constantin

2
  • The problem, as I see it, is not the idempotency of messages (most messaging systems like this can be made pretty durable), but rather the idempotency of the database transactions. In your airline reservation system, the need to consult a central database before assigning a seat seems unavoidable. Sep 13, 2018 at 15:24
  • With just at least once delivery, it seems like this would cause an amplification of duplicate messages? Two As are emitted before the emitter of As understands it's been heard. Then, for each A two Bs are emitted. Then for each B two Cs. Now we already have 8 Cs corresponding to the same real world event coursing through the system, all causing checks and computations. And who says they will be treated the same each time? We might get one successful reservation and 7 rejections. How will we know to ignore so many rejections and learn we were actually successful, if unordered? Sep 14, 2018 at 21:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.