TL;DR event-driven system seems to focus on a highlevel view of the system ("error rate is 0.5%"). How are IT operations supposed to locate and remedy individual issues in such systems?
In today's mainstream push towards distributed systems the even-driven architecture is often regarded very well. Among the benefits of this naturally async architecture are often quoted:
- ability to achieve loose coupling
- clean design of one-to-many calls
- ability to apply backpressure/feedback
- often natural horizontal scalability.
As drawbacks are mentioned things like:
- lack of transactional processing
- no promise of event order
- only-once event delivery being hard.
All this totally makes sense to me. However, I can't really figure out the operations side of this.
In a non-trivial business application there are often dozens of services connected with dozens of queues/topics/? Each event travels through a subset of application's services and queues/topics/? during the processing. The ops typically need to know and have ability to react when there has been an error in the processing of an event.
A typical approach seems to be employing event observability - each even carries a unique ID as a correlation ID through the processing. This way one can log the lineage of an event and obtain KPIs like error ratio, average throughput, etc. But this is a very highlevel view for typical ops tasks.
Ops have to handle classic tasks like: "What's the status of invoice no. X?" or "Why wasn't user John Doe able to order product Y?". How to find and remedy those cases in even driven application?
The first issue is to somehow locate the right event(s). From the observability logs ops have to be able to find the right event ID - so one have to log virtually every single attribute of any event. That doesn't sound right.
Then they need to locate the event in the system - it might be in almost any dead-letter-queue (DLQ), it might be stuck in any broken/slow queue/topic/?, etc. Is there a common ability to somehow query such systems?
Lastly, after the fix is in place ops need to replay the event. It doesn't seem to be common to manually pick events from queues/topics/? and placing or rerouting them somewhere else. Is this widely supported?