Is event sourcing only for when writes are rare?

Question

I'm reading up on event sourcing and can't stop asking myself if it only makes sense in exotic situations where writes are very rare or military-grade auditing is required.

A non-exceptional system with any significant usage might produce between hundreds and thousands of writes per day, translating to, say, a million or 2 writes (thus events) per year of operation. Merging millions of objects (events) just to get the current state sounds like a ludicrous proposition, when compared to a straight-up read from a traditional storage. Yet, event sourcing is behind some of the most performant systems (think LMAX).

So, what am I missing? Is restoring state from the event stream even commonly done? Or is the idea to rarely have to do this and instead use a different storage altogether for normal operations (i.e. use the Query storage from CQRS), and restore from events only in exceptional cases (like replication, auditing etc.)?

Answer 1

So, what am I missing?

Taking a guess.

The first thing that you may be missing is that you only need to reload the events for the state you are rebuilding. If you can model your transaction boundaries cleanly, each object can write out events tagged with its own id, and then read back in only those events. Using a relational database for event storage, there would be an indexed id column to speed that query. Using EventStore, each object would have its own stream.

It takes some care in your model to do this cleanly, as you want to be sure that you are only modifying a single object in each transaction, and therefore you need to take some care that you are correctly isolating each invariant that you are trying to enforce.

In cases where that isn't fast enough, you still have the possibility of creating snapshots of your state (memoization) , and persisting that in "traditional storage". Each snapshot gets tagged with the sequence number of the last event used to build the snapshot; on reload, the repository grabs that snapshot first, then applies newer events to it. (This implies some reasonable way to grabbing the more recent snapshots -- either the events are also tagged with the sequence number, or you have some efficient way to read the event stream backwards until you get to your starting point.)

There's still an advantage over the usual approach here, being that your snapshots can be built in parallel to your writes, rather than being merged with them: you just put an event listener in some other thread/process, and let it merrily along writing out to the snapshot store on whatever schedule seems reasonable. After all, the snapshot doesn't need to be particularly timely -- just often enough that the work of re-applying the newer events doesn't blow your SLA.

(Snapshotting does complicate migration; any changes to the serialization of the model will invalidate the snapshot cache. Of course, you can rebuild snapshots using the new serialization as part of the migration, and then "catch up" when the changes go live.)

Is restoring state from the event stream even commonly done?

Yes, it is. What normally is shown in CQRS examples is that the Application layer, after ensuring that the submitted command is well formed, the application layer will load the domain object from a repository, where the load is a default constructor followed by a replay of the event stream (or equivalently, a call to a factory with a list of events).

Two other, contradictory thoughts.

1. There could be a cache behind the repository interface
2. Cache invalidation is one of the two hard problems.