Event Carried State Transfer and initial state synchronization

Question

I'm exploring Event Carried State Transfer patter, I understand how it works, but there is one problem I don't know how to solve.

So, in ECST, a service can listen to events and then build a private state from those events. As an example - let's say I have a User service and my service consumer CRUD-like user-related events to build a private state about the users I need in my service. I can nicely update that state when I receive an event - but how can I bootstrap the state of my service with data that already existed before I released my service?

The simplest idea is to have an API that the service can query to fetch all users, but that sounds nor right as it couples the consumer with the producer.

Answer 1

The usual mechanism is a combination of snapshots and replays. This means that you store all of the events that lead to the current state for a given number of events, and then periodically you "flatten" them into a snapshot of the state at that time, and store it. The next time you need to bootstrap an instance, you load the latest snapshot and replay the remaining (unflattened) events that have occurred since that snapshot.

Answer 2

It really depends on the way you store and handle your event data.

From the question, I'm assuming you are not using an event sourcing strategy or keeping you history of events stored somewhere.

If the history of events is stored in a persistent storage, the strategy that @asthasr recommends is a good choice.

If you only have the final state of the users in some storage, like a database, and your system is capable of understanding events with a "change" in all properties, you can build a replay tool that fetches users from the storage or source service and publishes events with the current state of users, including all properties in a single event for each of them.

On the other hand, if you have more granular events that represent changes in different groups of user's data, the replay tool will have to generate all those granular events and publish them in order.

Answer 3

In broader terms, you have the same problem of schema updates, which use "migrations" as a solution.

But just as how to structure a migration is done on a case-by-casis basis, this can be the same.

If you add a new feature requiring the downstream service to have a full constructed view for all upstream entities, then you should do something to retrigger all older event messages. This is usually called a backfill.

It helps here if the event update logic is idempotent, because you can then play fast and loose with how you backfill.

Sometimes there's no need to backfill, and you have to live with partial data.