Event sourced persistence and data model refactoring

Question

I've been working on a system that is using event-sourced persistence for it's data model. It is using a standard model of sequence of events with aggregates being rehydrated from those events and projections used for more complex queries.

One thing that has been bothering me lately is the question of refactoring. As we follow an agile, incremental way of building the system, it becomes inevitable that new features might result in changes to the data model. In this case, it could be that events need to change. One specific example is when we made an improvement to the domain model, which resulted in splitting one event into two separate ones, with properties of the original being split into the two events.

Currently, the system is using sub-optimal model of using only single event. This results in usage of the aggregate being inconsistent, as it is forced to create events later instead of creating one event first and later creating a second event with additional data. I'm thinking of refactoring this, but because we keep history of all events, it would result in having both single-event model and double-event model in history, which needs to be supported in both aggregate rehydration and projections. This would not reduce complexity, but just move it from the point of creation of the events to the point of consumption of the events.

So I'm thinking about chaning the events history, but as we are using sequence numbers and version history, it is impossible to insert events into the past and in between two other events of an aggregate. The only option that comes to mind is to create a full copy of the events store and transform the events from one into another. But that feels like a really complex operation, as it needs two persistence stores to run at the same time, and software to support reading from both.

So I wonder if there is simpler way of doing this kind of refactoring?

Answer 1

Short answer: the book that you want to read is Versioning in an Event Sourced System by Greg Young.

Somewhat longer answer: there are simple solutions, but the simple solutions are not easy.

After all, what you are asking about is analogous to discovering that a table in your relational database needs to be turned into two tables, or a column in your table needs to be turned into two columns.

To some degree, the answer is "don't have that problem", which is to say invest in designing your schema correctly in the first place - which means both getting it right, and also leaving room to extend it later.

Events are, after all "just" messages; that is to say a data structure that decouples the aggregate from past/future versions of itself. So all of the ideas of future proofing the message schema apply.

But if mistakes are going to happen, then you need a path to recover from them.

A simple approach is to treat the old events as "legacy"; you stop writing the old events, but leave in the capability to read and interpret them. In other words, you act as though your event schema has a lifecycle, and treat it as such.

This often means that you need to come up with more names for things; or more namespaces; so that code can correctly interpret the old and new message schema without getting confused. Several of Rich Hickey's talks center this idea - start with Speculation.