1

I'm trying to design an event-driven architecture based on Kafka for messaging but I'm struggling in managing event dependencies. Let's suppose the following service choreography:

  • Service A: emits A-created
  • Service B: emits B-created when processing A-created
  • Service C: emits C-created when processing A-created and B-created

I have several times this situation, Service C that needs to join information from two events (A and B) to emit event C. This is quite complex in term of coding because it needs to be stateful processing since events can arrive in any order.

I don't understand if I'm reasoning in a wrong way or there is some particular suggestion in these cases. Every article I read about event-driven doesn't present such cases so I don't have a clear idea on how to approach this.

2

Having such strong interrelations, dependencies or coupling, however you want to call it may be an indication in itself that the architecture is not optimal. I will however put that aside for a moment and assume all of this is warranted by requirements, and services are not just crud-databases of things.

If your events are so interrelated and everything seem to be based them anyway, you can go full event-stream. If you are using Kafka already, one easy option would be Kafka Streams. With this, your services would basically become processing nodes in a graph of streams. Such Kafka Streams nodes can easily join two streams, like the stream for A and B, or aggregate A and B in another topic explicitly, or keep a (persistent Kafka-backed) state in the node itself. It can do all of this distributed, with exactly-once processing guarantees.

I would in any case suggest you not re-invent joining, synchronizing or aggregating event streams in a potentially distributed environment. It will be complex as you say, and you can easily have it off-the-shelf nowadays.

3
  • That's what I've begun doing but it seems impracticable. The problem is that implementing a streaming application for joining two events is not so lightweight, considering that I could have to do the same for other events in the same microservice (I'm using Spring Boot for implementing microservices). – Alessandro Dionisi Oct 4 '19 at 12:13
  • Doing multiple joins, or any combinations of processing is really not that big a deal using Kafka Streams. It is as easy as calling map(), join(), etc. functions on a stream (or in this case a KStream). It only works though if you are sort-of all-in on events and messages. – Robert Bräutigam Oct 4 '19 at 12:37
  • I'm not talking about code complexity but cpu/memory footprint for the microservice – Alessandro Dionisi Oct 4 '19 at 12:39
0

This is quite complex in term of coding because it needs to be stateful processing since events can arrive in any order.

Yes, so think state machine

START + Created(A) = WAITING_FOR_B(A)
START + Created(B) = WAITING_FOR_A(B)
WAITING_FOR_B(A) + Created(B) = READY(A,B)
WAITING_FOR_A(B) + Created(A) = READY(A,B)

You'll need something analogous to a correlation identifier so that the matching events are consumed by the same "instance" of the state machine.

Each time we get an event that we haven't seen, we copy information from the event into our own state (aka, we cache the event), so that when all of the information is available we are ready to go.

Rinat Abdullin wrote a good overview of process managers that is worth reviewing. The basic pattern being to think through how you would do it if a human being makes the decisions.

1
  • On my opinion, BPMs are a good fit in long-running processes. But in my case I'm processing short-running choreographies (like creating an account for a user and creating all the necessary resources in other microservices). It seems a bit heavyweight... – Alessandro Dionisi Oct 4 '19 at 12:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.