How to store Sagas with Event Sourcing

Question

I have a long running process that collects together the IDs of Aggregates that have been edited/changed, so that a later command can 'publish' them to a legacy model.

I understand that I should use a kind of Saga for this which listens to certain events, then stores additional IDs for later processing.

The question I have, given I am in an event sourcing environment (using an event store for the aggregates), is how should I be expecting to store the actual saga data?

• Should I be emitting an event from the saga to be recorded in the event store?
• Should I have a separate persistence engine to store the data, like a MySQL instance?

I am confused as to how complicated it would be for option 1, but also, to have to create another data schema with a single table for option two seems overkill also.

What are others in the community doing in this kind of scenario?

Answer 1

You have two options to run a Saga:

1. Synchronously/in-process, at each request, after the Event store persists the events, you sent them to the subscribed Sagas. This has the disadvantage that when something wrong will happen (i.e. server is restarted) you will lose events.

2. Asynchronously, as a long running standalone process that tails/polls the Event store, process the new events, acknowledges them in a persistent storage and then tails/polls again. Unlike the first option, when something wrong happens you just restart the process from the last acknowledged event. This is the preferred way, when possible.

From what I understand from your comments, you "have no mechanism to have a continually running process" so you are left with option no.1.

A Saga models a long running process; in Event sourcing this means a process that spans multiple events. In your case the Saga must collect the Aggregate IDs and persist them as if it were an Entity. For this you can use a flat persistence (as opposed to an event-sourced one), a simple NoSQL database. You should take care about concurrent modification of the Saga entity as events could come at the same time; for this you could use low level database operations like MongoDB's addToSet.

From the DDD point of view, this Saga entity is in fact a new Aggregate type, i.e. PublishSomeAggregatesToLegacySystemAggregate, that lives maybe in another Bounded context. The Saga would just react to events and send commands to ProcessSomeIDsAggregate, just like any other Saga. This new Aggregate should be non-event-sourced. So, when an Aggregate is edited/changed, the Saga sends a AddAggregateToBePublishedToLegacySystem command to the PublishSomeAggregatesToLegacySystemAggregate. When the final event is received by the Saga, it will send the PublishTheAggregates command.

You should be careful to what kind of logic you put inside the Saga. It should contain only coordination logic, not low level like SQL or legacy system access code.

Answer 2

"Saga", in this context, is more commonly referred to as a "process manager", in part because the original meaning of saga was different, and in part because we suck at naming things.

My recommended starting point on process managers was written by Rinat Abdullin.

Process managers are, essentially, command recommenders; given that I know about events A, B, and C, I recommend commands Y and Z. The are a book keeping device to keep track of work that should be done in the system.

They often have an internal expression as a state machine.

state = fsm.start()
           .andThen(A)
           .andThen(B)
           .andThen(C)
           .state()

pendingCommands = state.commands()

As a book keeping device, they are a lot like "views", in the cqrs sense. Like views, if you have a history of all of the events available already, you can rediscover what state a process is in.

For instance, you might imagine a stand along service, that has the code that defines all of your processes. On start, it pulls from your event store the entire history of the world, computes in memory the state of each managed process, and sends the command messages to your domain model. On each clock tick after that, it checks for new messages, reads them in, and sends new command messages.

If the service panics, no problem -- just launch a new copy that starts all over again.

So in a sense, you don't need to write anything down at all.

(Which isn't to say that you shouldn't -- there are advantages to knowing when things happened, for instance, or being able to reproduce the behavior of the production system in a lab environment.)

Answer 3

So the Saga should be stored as events in your eventstore. And as VoiceofUnreason answered about process managers, a process manager will do the orchestration. You know when the orchestration is complete when the final event is published.. the process manager should post commands to the DDD aggregate which will publish the events.

There is a great platform for event sourcing... www.eventstore.org