Event sourcing and persistence

Question

I'm reading up on event sourcing and have a question regarding persistence.

I can still have a DB with all entities, right? Or should the events be replayed every time the application is started to get the latest version of each entity in the memory? Seems like a waste on larger systems (as in large amount of data)?

The point with event sourcing is that I can can replay the events to populate a data store if required? (or analyze the data)

Answer 1

You will benefit the most from the event sourcing when you decide to change your system architecture also. Going towards a CQRS style architecture combined with DDD will bring up the true benefits of an event sourcing, at least in my opinion.

Building an event store that behaves well in large systems is not an easy task indeed. Replaying all the data might be expensive indeed, depends a lot on the amount of data that needs to be replayed. But there are techniques that might help you with this, one of them being the concept of a snapshot. The replay is done only from a certain point forward. The advantages that an event store bring into your system are invaluable. Having everything that happened in your system replay-able, all the data in every moment is a great thing. Think about analysis, about bug reproduction, about statistics.

There are a lot of great event stores, the last one was just released yesterday Event Store and it seems like a really good one.

The traditional database can be kept for the query part of your system to build up DTO's with the requested data. This database can be organized and optimized considering the query needs of your application and clients.

I wrote a detailed article about what are the benefits and how does a CQRS architecture combined with event sourcing really looks like. You can check it out CQRS, Domain Events and DDD review.

Answer 2

With Event Sourcing the main question is "what is your book of record".

If your book of record is your event stream then you will have no problems. If your book of record is your "entity model" then problems will start happening all over the place. Part of this is that you can say "if I lost my entity model could I rebuild it from my event stream". If you are in the positive on this question then your Event Log is your book of record.

Its also important to remember that most people that use event sourcing use a read model. This model is used for querying data. This is more likely to look something like a 1nf model than a 3nf entity model though. They only replay events to get back the states of aggregates to determine if writes should be allowed.

Answer 3

I can still have a DB with all entities, right? Or should the events be replayed every time the application is started to get the latest version of each entity in the memory?

The answer depends on your application's requirements. I have seen it done both ways.

One extremely successful software package for small accounting firms reads its CQRS log every time on start-up. The raw amount of data was relatively small, so the start-up time was under a minute even on slower computers. They have been doing CQRS for more than a decade before the practice became popular. They knew they were on to something good when they realized that they can upgrade their client data again and again without running into troubles that they see with their larger systems.

In systems with larger volumes of data and/or systems that rely on RDBMS functionality for implementing the query side you have a database for the "current view" of the event-sourced data (you can even have multiple such views). The advantage of this approach is that it lets you build the query side using the familiar technologies.

Answer 4

When I've had to implement CQRS/ES in real life, I've used a mongoDB data store for perisisting read models (aka projections) so my hands-on experience is there, but let's do some reasoning:

I can still have a DB with all entities, right? Or should the events be replayed every time the application is started to get the latest version of each entity in the memory? Seems like a waste on larger systems (as in large amount of data)?

In theory you can follow both approaches and still have a full fledged, correct implementation of CQRS/ES. The key of Event Sourcing is to use the operational data store to represent behaviours (sequence of domain events) over data (entity state) and, as postulate, events become the contract that tie together the aggregate[s] (read side) with their representations (read side). Nothing will constrain read-side implementation, except that it should behave consinstently with the events

Let's go through tradeoffs of both choices

• One solution for the read side is to attach event listeners that are used to update projections (what you've called "entities" but it's a misleading nomenclature here, since entities are on the other side of your implementation) and perform tasks that create/modify the projection and persist it in a DB storage. Here you can benefit of data durability and get rid of replaying events everytime the service is bootstrapped. Anyway, everytime you introduce breaking changes on your event you have to take care to develop mappers that let new event version

This approach fits well with situations where you have a large amount of events to process in your store and you don't want to replay them everytime to not hinder deploy time and to not cause datastore load spikes. Eventually though, your codebase will be full of patch mappers and you may want to get rid of them, so a replay will be still needed at some point

• In The second approach you should asses how big is the catalog of projections you are building. If you have 10M bills and each bills takes 100B of RAM, you've already made your memory footprint pretty big. Therefore, according with involved numbers, a countermeasure could be introducing an in-memory datastore (eg. Redis) which lets scale RAM avaliability and offer parallel access to muliple instances. Still, you have to replay events at every deployment or whenever data is not available in your datastore. This can be affordable only if you keep the event store pretty small, perhaps through continous snapshotting. The advantage is that you don't need to be aware of event versioning

so TLDR

• if you have few events / high development velocity -> in memory projection
• if you have a lot of events / high velocity -> DB projection with event mapping
• if you have a lot of events / low velocity -> DB projection with replay
• BONUS: if you need to make queries/aggregations on the projections -> DB