How do event sourcing services recreate the state of an entity?

Question

I'm reading about event sourcing and have a question regarding the recreation of an entity state.

For a simple scenario, lets say we have Order events such as OrderCreated / OrderUpdated / OrderDeleted.

Now I understand that if I use event sourcing, than these event will first end up in an event store,let's say a kafka topic named OrderEvents. If our application handles 10000+ orders a day with changing state, it will be full of events. Let's say the OrderService can be used to ask for a given order state. In this case, how would a real service recreate the current entity state for a given order?

My concern:

• It might take a lot of time to check all the events and find what is relevant for a given orderId, and then apply those events to eventually create the current entity state, and return that to the client. How is this solved?

Answer 1

So the usual answer is that instead of using an event delivery system, like Kafka, the "book of record" is a database (could be relational, could be a document store, could be an "event" store).

To answer questions like "what color is entity 12345?", you would load the sequence of events for entity 12345, then compute the color from the history.

Depending on what color is, this might mean paging from the most recent event backwards until you find the most recent event that sets the color, or "replaying" all of the events from the beginning of time to recreate a local copy of the current state of the entity, and then asking your local copy of the entity what color it is.

In other words, for queries, the implementation of the query can have a number of different forms, depending on how events are stored and how they are retrieved from the permanent storage.

It might take a lot of time to check all the events and find what is relevant for a given orderId

Yup, it might. Usual design is to arrange that (copies of) all of the events for a given entity are stored in the same history. In other words, instead of having a single history for an entire topic, we normally will have individual histories for each entity in the topic.

(Reading up on "domain driven design" and the "aggregate" life cycle pattern might help here -- typically each aggregate has a history, where an aggregate is a cluster of closely related entities).

In cases where query latency is a significant concern: we can cache copies of event state, and use those to answer questions. So you get faster, correct answers that reflect the state of the entity at some point in the recent past.

For more on these ideas, see CQRS (command requiry responsibility segregation).

For making updates to the entity state, you need to be a little bit careful with caches, because lost edits are a bummer. We want to arrange our history store so that it has "first writer wins" semantics - either we arrange to lock the history against concurrent modification, or we use a conditional write mechanism to ensure that the location where events are written into the history exactly matches where it was when we started the calculation.

Answer 2

If you follow the philosophy it doesn't take that long to resolve.

You load maybe 10 rows instead of one, but its on the same order of timescale and then its simple in memory operations to get your object up to date.

The problem starts when you have hundreds or even thousands of changes, 90% of which will be overwritten by later changes. One solution to this is to create "snapshot" records which roll up all the changes before a certain date. But to my eyes this breaks the whole point of doing event sourcing in the first place.

However, its not so much a length of time problem as a complexity issue. Lets say you have an event which is fired on OrderConfirmation. When the order is confirmed an email gets sent to the customer "thanks for your order" or whatever and the state is updated to "Confirmed"

Now when you load that event from the event source database obviously you don't want to resend the email, but you DO want to update the state. So now you have to divide the event triggers into repayable and non repayable. Which again, to my mind at least, goes against the philosophy, but also makes your code so much more complex than if you are just loading the object state.

The other problem is even if you can do the operations in the same sort of timescale, you are paying for that cpu!! If you are trying to lower your cloud compute bills and some non technical person says "why do we load and process 10 rows instead of 1?" its a hard question to answer