How can we keep sight of business flows in event driven architectures?

Question

I am planning to set up an event driven architecture using Spring Boot apps that publish and read messages from a Kafka broker.

Let's suppose it were an e-commerce application with the usual events (order placed, payment processed/failed, item-reserved, no-inventory-availability, order-shipped, and so on).

In my business context I fear this could happen:

The problem is that it can be hard to see such a flow as it's not explicit in any program text. Often the only way to figure out this flow is from monitoring a live system. This can make it hard to debug and modify such a flow. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. The pattern is still very useful, but you have to be careful of the trap.

From Martin Fowler's article What do you mean by event driven

The question is how can I keep a global view of the business flow happening in such a decoupled and event rich architecture?

Answer 1

The problem is that it can be hard to see such a flow as it's not explicit in any program text. Often the only way to figure out this flow is from monitoring a live system.

There are two separate aspects to this: how do you centralize a flow's logic, and how do you identify flows when monitoring.

For monitoring a possible solution is the use of correlation and causation ID's. Every event is labeled with a random and unique ID. The correlation ID of a flow is the unique ID of the originating event, and it is copied over to a separate field of each successive event that is part of that flow. The causation ID is the unique ID of the previous event in the flow that directly led into the current event, which is useful for determining the causative relationship between events. Convenient in this is that there is no central component that must do bookkeeping of flows, as only log aggregation after the fact is needed.

For centralizing the flow's logic a solution lies in the use of sagas and process managers. The saga is a long-running flow in an evented system that spans multiple parts and possibly multiple bounded contexts. The process manager is a piece of software that connects the parts together. It does not itself implement any business logic, but merely maps events onto commands to other parts of the system.

Incidentally, there are many more problems in evented systems than identifying flows. A useful resource is Microsoft's CQRS Journey book. It illustrates through a real system what these are and how to address them.

Answer 2

You can rely on the Context map that gives you a high level view of the Bounded contexts and their relationships.

Then, inside a Bounded context, you can rely on the code to give you a detailed view of how the system should work. Here you have Aggregates, that process command and emit events. Then you have Sagas/Process managers that coordinates higher level business processes by reacting to events and sending commands to the Aggregates.

Then you can have a live view of how the messages are traveling through the system. I have done something like this in the past.