How to synchronize databases using event streams?

Question

Say you have a bunch of microservices each accessing dedicated databases. Due to business logic reasons, data from some DBs need to be replicated to other DBs (e.g. to serve as local cache).

One way to handle replication is to stream changes done to DB 1 (e.g. using my Debezium) into a queue (e.g. Kafka). Queue consumer then applies those changes to DB 2.

Problem is: what if queue fails temporarily? Or if DB 2 needs to be recreated from scratch due to a catastrophic failure?

The only solution I can think of is to make periodic snapshots of DB 1 and publish these snapshots in the same queue. DB 2 can be rebuilt from that snapshots plus event that happened after it. Does that make sense?

Answer 1

Throwing complete snapshots down the line can be overwhelming and a bit wasteful if none of the downstream systems are desynchronised or need it.

Ways to mitigate/handle it.

1. Dribble feed correct records. Essentially send any record that hasn't had a recent update, as a verification message periodically/slowly. If the downstream system detects a discrepancy it can correct it, and perhaps request a full synchronisation if too many discrepancies have been recently detected.

2. Feed through thumb-prints representing sets of records. If the downstream system cannot verify the thumb print it knows that something is amiss. It can request that set of records be sent, or a full synchronisation if too many discrepancies have been detected.

3. In a side channel allow the downstream system to request full or partial copies of the data outside of the update stream. This allows a full synch without requiring all the other systems to hear it too.

4. Use a transactional append only queue for events + a base line view. Downstream systems that get behind or go down can revive from the last known read point, or if that is too old can restore the view and play forward. Update the view as the queue content ages/fills. If you integrate thumbprints of the view into the data the downstream systems can verify the validity of their views at that point in time.