If I have a service that consumes events of type A and emits events of type B, how can I make sure that event B is only emitted once for every event A? Should the service also subscribe to events of type B to see that an event B with a same correlation ID/with a reference to the A event wasn't written yet? Seems like a lot of extra responsibilities for the service.
Making sure events are processed once and only once in a distributed system is really really hard.
But, we are talking network failure and program crashes. In a normal message queue system, if you emit B when you get A and set your queues up correctly and have no major bugs you shouldnt worry overly.
Having said that, when you have outages it will be a possibility. So you should have systems in place to pick up on significant duplicate events, say taking payment twice, after the fact and correct them.
This is possible with the aid of Activities. Pat Helland discusses about Activities in his paper Life beyond Distributed Transactions: an Apostate’s Opinion. Vaughn Vernon also talks about this in this video.
The idea is that a receiver of a message should store in it's localstorage the IDs of all the messages that it has seen. If it receive a message that it has already seen then it ignores it. If the received has not seen the message then it processes it and then emit its new own messages.
In Event sourcing this means that the new messages should contain in their body the ID of the message that was the cause of those new messages.
So, in your case, message B should contain the ID of message A. So, when the message A is retried, the service replays all the previous messages that it has emitted, including the message B and then it sees that it has already processed the message A and ignores it.
This technique, combined with at least one delivery, gives you what you want, in the most scalable way.
It seems you are mixing up two quite different ideas for event processing here. You can either
- try hard to both transmit/process each event at least once, and at most once, even under the most exotic failures. These are two hard problems which both are only solvable to some extend in distributed systems.
- try hard to transmit/process each event at least once, and design your events so that it doesn't matter when you receive several events for one domain change. For this, idempotency of the event processing is a good approach.
The second solution is often easier and (sometimes much) more performant, at the expense of having to think hard about the structure and meaning of your events. You'll probably also have to design your distributed system so that the order, in which the events are received, doesn't matter, too. And that additional events triggered as the result of the second arrival of an event are OK, too.
One way to do this would be to keep all event IDs received in a reasonable timeframe, and discard duplicates. But you could also stucture your events so that it doesn't matter if they are received twice. That's where idempotency comes in: an operation (the event processing) doesn't change anything, anymore, when it is repeated more than once. For example, if you receive an "user adds street address" event, and the user already has that address, you can naturally just discard that event.
In your example, this means that you could make sure that the service emits a type B event for each type A event, but you can omit any special precautions that it never ever emits two type B events in the case of failures, since you have designed your events so that this wouldn't do any harm (besides a little more data being transmitted). You arrange things that events normally get received once, but in the case of failures it can happen that it's received two or more times.
If I have a service that consumes events of type A and emits events of type B, how can I make sure that event B is only emitted once for every event A?
The only way I know to do that is to ensure that all consumers of event B are part of the same transaction as event A.
In a distributed world? no chance.
At most once delivery is trivial -- just swallow event B without ever emitting it.
At least once delivery is manageable.
Exactly once delivery is beyond economically viable.
In most cases, the important points are (a) to make best effort to ensure that consumers recognize events that have been successfully processed, and (b) to have protocols in place to handle circumstances when events have been processed more than once.
(Example: on airplanes, we are legally limited to one passenger per seat. But it may sometimes happen that two passengers are assigned to the same seat. Even in this crisis, the plane usually manages to reach its destination....)
With just at least once delivery, it seems like this would cause an amplification of duplicate messages?
It certainly could - I think that depends on what synchronization points follow, and what strategies are applied by the handlers when they recognize a duplicate message.
For instance, if a receiver is processing messages in batches, then it might receive two copies of a message, and recognize the duplication.
On the other hand, the duplicated message may be an indication that an acknowledgement was lost in transit, in which case that acknowledgement should be resent, so that the upstream participant can know that the at-least-once requirement has been met.
A good starting point for this sort of thing is Udi Dahan's Reliable Messaging Without Distributed Transactions. You may recognize some similarities to the Pat Helland paper referenced by Constantin