How to manage event dependencies in event-driven architectures?

Question

I'm trying to design an event-driven architecture based on Kafka for messaging but I'm struggling in managing event dependencies. Let's suppose the following service choreography:

• Service A: emits A-created
• Service B: emits B-created when processing A-created
• Service C: emits C-created when processing A-created and B-created

I have several times this situation, Service C that needs to join information from two events (A and B) to emit event C. This is quite complex in term of coding because it needs to be stateful processing since events can arrive in any order.

I don't understand if I'm reasoning in a wrong way or there is some particular suggestion in these cases. Every article I read about event-driven doesn't present such cases so I don't have a clear idea on how to approach this.

Answer 1

Having such strong interrelations, dependencies or coupling, however you want to call it may be an indication in itself that the architecture is not optimal. I will however put that aside for a moment and assume all of this is warranted by requirements, and services are not just crud-databases of things.

If your events are so interrelated and everything seem to be based them anyway, you can go full event-stream. If you are using Kafka already, one easy option would be Kafka Streams. With this, your services would basically become processing nodes in a graph of streams. Such Kafka Streams nodes can easily join two streams, like the stream for A and B, or aggregate A and B in another topic explicitly, or keep a (persistent Kafka-backed) state in the node itself. It can do all of this distributed, with exactly-once processing guarantees.

I would in any case suggest you not re-invent joining, synchronizing or aggregating event streams in a potentially distributed environment. It will be complex as you say, and you can easily have it off-the-shelf nowadays.

Answer 2

This is quite complex in term of coding because it needs to be stateful processing since events can arrive in any order.

Yes, so think state machine

START + Created(A) = WAITING_FOR_B(A)
START + Created(B) = WAITING_FOR_A(B)
WAITING_FOR_B(A) + Created(B) = READY(A,B)
WAITING_FOR_A(B) + Created(A) = READY(A,B)

You'll need something analogous to a correlation identifier so that the matching events are consumed by the same "instance" of the state machine.

Each time we get an event that we haven't seen, we copy information from the event into our own state (aka, we cache the event), so that when all of the information is available we are ready to go.

Rinat Abdullin wrote a good overview of process managers that is worth reviewing. The basic pattern being to think through how you would do it if a human being makes the decisions.