Context: A typical corporate landscape with many systems located in different data centers, with firewalls, without uniformised interfacing.

Some data X is being replicated for availability and independence reasons. Say System A is the master system for X.

X is being replicated on-update (event notifications) using various technological means (from file interface to API calls) to systems B, C, and D. In this replication the slave systems have access to the master's change timestamps.

For the customer-facing applications to function correctly, we want to ensure that A, B, C, and D are in sync with each other.

Is there a robust mechanism/pattern that can report on whether these systems' versions of the data are in sync and highlight any differences?

I am aware of challenges regarding a benchmark in an eventually consistent setting. Maybe the comparison should take the change timestamps in account?

The primary use cases for ensuring synchronicity are:

  • Periodic synchronicity report on the entire customer base to cover any unexpected issues;
  • Incidental report for a batch of new customers before we open up the applications to them.

Of particular concern is that I would like the sync report to be as simple as possible to avoid false-positives or (worse) false-negatives. Since re-sending update notifications is easy, it would suffice to have an overview of the differences; then we can work from there.

I could constrain the question further but I am also open to reconsidering other parts of the architecture if this is needed (based on suggestions/premises/assumptions in the answers). However the distributed nature will remain and should be addressed.

NB. As this is my first post on SE.SE, please point out any mistakes in etiquette/tagging/title; apologies for that. I did search for similar questions on this site but did not manage to find any. This came close however does not have the answers I need.

  • Eventual consistency and guarantees of real-time consistency are of course two incompatible qualities. It's impossible to answer your question about what approaches can be used to generate a consistent report without knowing exactly what you have to work with.
    – Steve
    May 31, 2019 at 11:02
  • @Steve I am aware of the challenge in that aspect. You can assume the availability of last-changed timestamps. I would hope there could be some abstract inroads to the problem without referring to the specific technologies (for they are many...)
    – Lord_Farin
    May 31, 2019 at 11:08
  • 1
    Abstractly, the problem is one of establishing the latest time at which all systems were consistent with one another, generating output based on the state of each system at that time, and ensuring that the effect of any subsequent updates to some (but not all) of the systems is disregarded.
    – Steve
    May 31, 2019 at 12:05


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.