Context: A typical corporate landscape with many systems located in different data centers, with firewalls, without uniformised interfacing.
Some data X is being replicated for availability and independence reasons. Say System A is the master system for X.
X is being replicated on-update (event notifications) using various technological means (from file interface to API calls) to systems B, C, and D. In this replication the slave systems have access to the master's change timestamps.
For the customer-facing applications to function correctly, we want to ensure that A, B, C, and D are in sync with each other.
Is there a robust mechanism/pattern that can report on whether these systems' versions of the data are in sync and highlight any differences?
I am aware of challenges regarding a benchmark in an eventually consistent setting. Maybe the comparison should take the change timestamps in account?
The primary use cases for ensuring synchronicity are:
- Periodic synchronicity report on the entire customer base to cover any unexpected issues;
- Incidental report for a batch of new customers before we open up the applications to them.
Of particular concern is that I would like the sync report to be as simple as possible to avoid false-positives or (worse) false-negatives. Since re-sending update notifications is easy, it would suffice to have an overview of the differences; then we can work from there.
I could constrain the question further but I am also open to reconsidering other parts of the architecture if this is needed (based on suggestions/premises/assumptions in the answers). However the distributed nature will remain and should be addressed.
NB. As this is my first post on SE.SE, please point out any mistakes in etiquette/tagging/title; apologies for that. I did search for similar questions on this site but did not manage to find any. This came close however does not have the answers I need.
