We're building a service using DDD, CQRS, & Hexagonal Architecture that allows the user to upload a CSV feed for which every row will be transformed and sent to a third party over HTTP.
We have two aggregates: Feed & Delivery. Feed contains original entries from CSV and Delivery contains payload, status, and fail count.
Both Feed & Delivery generate domain events which we then insert to a DB in the same transaction as the aggregate that emitted it ("transactional outbox" pattern). This way we keep events consistent with their aggregate states and handle eventual consistency well, i.e. aggregates are always consistent, but overall application state will be eventually consistent because the whole processing chain might take a while.
Finally, we have an out-of-process listener ("transaction log tailer") that reads inserted events and asynchronously feeds them back to our app so that we can build complex workflows. Once event is processed, its
processed value is updated in DB (to prevent it from being processed again in case our listener delivers it more than once). This allows us to keep all our use cases small and let them fail as much as they want: every event will be retried independently. Our application layer acts as a "router" for asynchronous domain events, and also handles commands.
Our events are
Our commands are
The flow is as follows (format is event->command->event):
- [User] ->
DeliverySucceeded(on success) ||
All of this works pretty well so far, but with that come few questions.
Let's say we add a new command
RegisterAttempt which writes delivery information to another aggregate, and we want to call it every time
DeliveryScheduled happens. Our options are:
- Do this in the same handler (and same transaction) where we already handle
ExecuteDelivery). Cons: we'll violate SRP because now our handler runs two commands (
RegisterAttempt), so failure from one of them will fail both.
- Make 2 separate handlers. Cons: one of them will mark the event record as "processed" and thus prevent second handler from commiting on completion.
- Insert one copy of each event for every handler (aggregate) interested in processing it. Cons: this will require the publisher (use case) to know about all stakeholders ahead of time and will polute DB with duplicate events which doesn't sound right since only one event happened, so we're mixing concepts of topic & subscriptions here.
- Make event handler behave like a multiplexer: upon receiving 1 event, create N "EventInstance" records (one for each handler) and put them on a bus so that they can then be processed by actual handlers (aggregates) independently. Cons: we now have events and event instances, which sounds confusing.
- Store a list of "succeeded" handlers within the event record, populating it after every successful handler so that if one fails, we retry the whole event, but skip handlers that previously succeeded. Cons: this again violates SRP because our handler is operating on multiple aggregates.
- Have only one handler for any event and do only one thing (command) there. This is probably the best and easiest solution. It also respects SRP really well at several levels and makes handlers (and surrounding code) very simple. Cons: this requires us to unwind flows like
Event1->Command1->Event2->Command2->Event3->Command3which increases overall latency, makes parallelization harder and introduces surrogate events that might make no sense from the domain's point of view.
TL;DR: When using Transactional Outbox pattern in DDD, what if we want to have multiple handlers (i.e. multiple aggregates that need to react to some event) for the same domain event that need to be retried independently, but there's only a single instance of the event? Where to store the state of each handler?
Even though our architecture works well so far (because we currently only have 1 handler for any given event), I feel like I may be getting some concepts fundamentally wrong. Any thoughts & advices would be greatly appreciated!