12

So I've been flirting with Event Sourcing and CQRS for a while now, though I've never had the opportunity to apply the patterns on a real project.

I understand the benefits of separating your read and write concerns, and I appreciate how Event Sourcing makes it easy to project state changes to "Read Model" databases which are different from your Event Store.

What I'm not super clear on is why you would ever rehydrate your Aggregates from the Event Store itself.

If projecting changes to "read" databases is so easy, why not always project changes to a "write" database whose schema perfectly matches your domain model? This would effectively be a snapshot database.

I imagine this must be pretty common in ES+CQRS applications in the wild.

If this is the case, is the Event Store only useful when rebuilding your "write" database as a result of schema changes? Or am I missing something bigger?

  • There is nothing wrong with asynchronously writing to a write model state store and using that exclusively to load entities. The same exact consistency issues are present whether you do that or not. The key to fixing those consistency issues are to model your entities differently. There is nothing magic about Event Sourcing that resolves those consistency issues. The magic lies within modeling and not caring. There are specific applications that do require consistency at that level that have highly contentious entities no matter how you model them and will require special attention regardless. – Andrew Larsson Aug 1 '18 at 23:33
  • As long as you can guarantee delivery of events. To do this, your application simply needs to synchronously publish an event onto a durable event bus. After publishing, the job of the application is complete. The bus will then deliver it to various event handlers: one to update the event store, one to update the state store, and any others needed to update the read stores. The reason you're using Event Sourcing is because you don't care about immediate consistency anymore. Embrace it. – Andrew Larsson Aug 3 '18 at 17:02
  • There is no reason you should be constantly loading your entities from the event store. That's not its purpose. Its purpose is to provide a raw, permanent ledger of everything that has occurred in the system. Entity state stores and denormalized read models are for loading and reading. – Andrew Larsson Aug 3 '18 at 17:06
12

What I'm not super clear on is why you would ever rehydrate your Aggregates from the Event Store itself.

Because the "events" are the book of record.

If projecting changes to "read" databases is so easy, why not always project changes to a "write" database whose schema perfectly matches your domain model? This would effectively be a snapshot database.

Yes; it's sometimes a useful performance optimization to use a cached copy of aggregate state, rather than regenerating that state from scratch every time. Remember: the first rule of performance optimization is "Don't". It adds extra complexity to the solution, and you'd prefer to avoid that until you have a compelling business motivation.

If this is the case, is the Event Store only useful when rebuilding your "write" database as a result of schema changes? Or am I missing something bigger?

You are missing something bigger.

First point is that if you are considering an event sourced solution, it is because you expect there to be value in preserving the history of what has happened, which is to say that you want to be making non destructive changes.

So that's why we are writing to the event store at all.

In particular, this means that every change needs to be written to the event store.

Competing writers could potentially either destroy each other's writes, or drive the system to an unintended state, if they aren't aware of each other's edits. So the usual approach when you need consistency is to address your writes to a specific position in the journal (analogous to a conditional PUT in an HTTP api). A failed write tells the writer that their current understanding of the journal is out of sync, and that they should recover.

Returning to a known good position then replaying any additional events since that point is a common recovery strategy. That known good position can be a copy of what is in the local cache, or a representation in your snapshot store.

In the happy path, you can keep a snapshot of the aggregate in memory; you only need to reach out to an external store when there is no local copy available.

Furthermore, you don't need to be completely caught up, if you have access to the book of record.

So the usual approach (if using a snapshot repository) is to maintain it asynchronously. That way, if you do need to recover, you can do so without reloading and replaying the entire history of the aggregate.

There are many cases where this complexity is not of interest, because fine grained aggregates with scoped lifetimes don't usually collect enough events for the benefits to exceed the costs of maintaining a snapshot cache.

But when it is the right tool for the problem, loading a stale representation of the aggregate into the write model, then updating it with additional events, is a perfectly reasonable thing to do.

  • This all sounds reasonable. When I played around with a dummy implementation of ES a while ago I used the EventStore to guarantee consistency but I also synchronously wrote to what you call a "snapshot repository". This meant the current state was always ready to read without having to replay events. I suspected this wouldn't scale well, but since it was just an exercise I didn't mind. – MetaFight Jan 10 '18 at 11:31
6

Since you don't specify what would the purpose of the "write" database would be, I will assume here that what you mean is this: when registering a new update to an aggregate, instead of rebuilding the aggregate from the event store, you lift it from the "write" database, validate the change, and issue an event.

If this is what you mean, then this strategy will create a condition for inconsistency: if a new update happens before the last one had a chance to make it into the "write" database, the new update will end up validated against outdated data, thus potentially issuing an "impossible" (i.e. "disallowed") event and corrupting the system state.

For example, consider a standing example of booking seats in a theater. To prevent double booking, you need to make sure that the seat that is being booked is not already taken - this is what you call "validation". To do that, you store a list of already booked seats in the "write" database. Then, when a booking request comes in, you check if the requested seat is in the list, and if not, issue a "booked" event, otherwise respond with an error message. Then you run a projection process, where you listen to the "booked" events and add the booked seats to the list in the "write" database.

Normally, the system would function like this:

1. Request to book seat #1
2. Check in the "already booked" list: the list is empty.
3. Issue a "booked seat #1" event.
4. Projection process catches the event, adds seat #1 to the "already booked" list.
5. Another request to book seat #1.
6. Check in the list: the list contains seat #1
7. Respond with an error message.

However, what if requests come in too quickly, and step 5 happens before step 4?

1. Request to book seat #1
2. Check in the "already booked" list: the list is empty.
3. Issue a "booked seat #1" event.
4. Another request to book seat #1.
5. Check in the list: the list is still empty.
6. Issue another "booked seat #1" event.

Now you have two events for booking the same seat. The system state is corrupted.

To prevent this from happening, you should never validate updates against a projection. To validate an update, you rebuild the aggregate from the event store, then validate the update against it. After that, you issue an event, but use timestamp guard to ensure that no new events have been issued since you last read from the store. If this fails, you just retry.

Rebuilding aggregates from the event store might carry a performance penalty. To mitigate this, you can store aggregate snapshots right in the event stream, tagged with ID of the event from which the snapshot was created. This way, you can rebuild the aggregate by loading the most recent snapshot and replaying only events that came after it, as opposed to always replaying the whole event stream from the beginning of time.

  • Thanks for your answer (and sorry for taking so long to respond). What you say about validating against the write database isn't necessarily true. As I mentioned in another comment, in an example ES implementation I was playing with I made sure to update my write database synchronously (and store the concurrencyId/timestamp). This allowed me to detect optimistic concurrency violations without needed to ready from the EventStore. Granted, synchronous writes alone don't safeguard against data corruption, but I was also doing single-access (single-threaded) writes. – MetaFight Jan 10 '18 at 11:38
  • So, I had my consistency issues sorted out. Though, I assumed this was at the expense of scalability. – MetaFight Jan 10 '18 at 11:38
  • Synchronously writing to the write database still carries a danger of corruption: what happens if your write to event store succeeds, but your write to the write database fails? – Fyodor Soikin Jan 10 '18 at 13:15
  • 1
    If read projection fails, it will just retry until it succeeds. If it crashes outright, it will wake up and continue from where it crashed - in other words, also retry. The outside observable effect would be no different from it just running a bit slow. If the projection keeps failing and failing consistently, that would mean there is a bug in it, and that will have to be fixed. After fixing, it will resume running from the last good state. If the whole read database becomes corrupted as a result of the bug, I will just rebuild the database from scratch using the event history. – Fyodor Soikin Jan 10 '18 at 15:45
  • 1
    No data is ever lost, that's the big point. Data can be stuck in an inconvenient (for reading) shape for a while, but it's never lost. – Fyodor Soikin Jan 10 '18 at 15:46
3

The main reason is performance. You could store a snapshot for every commit (commit = the events that are generated by a single command, usually only one event) but this is costly. Along the snapshot you need to store also the commit, otherwise it would not be Event sourcing. And all this must be done atomically, all or nothing. Your question is valid only if separate databases/tables/collections are used (otherwise would be exactly Event sourcing) so you are forced to use transactions in order to guarantee consistency. Transactions are not scalable. An append-only event stream (the Event store) is the mother of scalability.

The second reason is Aggregate encapsulation. You need to protect it. This means that the Aggregate should be free to change its internal representation at any time. If you store it and heavily depend on it then you'll have a very hard time with versioning, which will happen for sure. In the situation when you use the snapshot only as an optimisation, when schema changes you simply ignore those snapshots (simply? I don't really think so; good luck determining that Aggregate's schema changes - including all nested entities and value objects - in a efficient way and managing that).

  • When my Aggregate schema changes, wouldn't it be a simple matter of replaying my events to generate an updated "write" database? – MetaFight Jan 10 '18 at 11:43
  • The problem is detecting that change. An Aggregate could be very large, with many files/classes. – Constantin Galbenu Jan 10 '18 at 11:46
  • I don't understand. The change would happen with a software release. The release would probably come with a database script to regenerate the "write" database. – MetaFight Jan 10 '18 at 11:47
  • It's a lot of work to do for a migration script. While it runs the app must be down. – Constantin Galbenu Jan 10 '18 at 11:58
  • @MetaFight if the stream is very large it will take a lot of time to rebuild the new aggregate schema... I am thinking now of an snapshot that is a state of a live projection that could be running before the release of the new aggregate schema – Narvalex May 15 '18 at 19:03

protected by gnat Dec 26 '17 at 12:09

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.