0

We have a master system storing customer data. Data is replicated to client systems (channels) at night. During the day data can be updated by users/customers on the master as well as on the clients.

I need all data to be kept in sync between master and clients. Preferably "real-time" (within minutes).

What pattern should I be looking at? CQRS/ES?

Some notes:

  • I don't control the master, so I can't implement a broadcast soltution on that end.
  • Master data can be set real-time using web-services, but can only be bulk-read once a day.
  • When updates are coming from certain clients, I need to do some processing on other clients.
  • <1000 messages a day.
  • MS/.NET environment.
7
  • 2
    if you don't control the master, or some central system, you'll just have to poll every second or so. not a good solution
    – Ewan
    Commented Jun 24, 2017 at 16:54
  • 1
    He can also make sure, that no user send data to master. That way he will know all data on clients and keep master as his-client-write-only bucket.
    – Marqin
    Commented Jun 24, 2017 at 17:08
  • like here: i.sstatic.net/qmkmG.png
    – Marqin
    Commented Jun 24, 2017 at 17:16
  • 1
    Plain old MS SQL server can do master-slave replication, and did for many years. Many other databases can do this, too. If you considered DB-level replication, what were the reasons that made you reject it?
    – 9000
    Commented Jun 24, 2017 at 17:40
  • not sure i would recommend every client having a slave replicated db
    – Ewan
    Commented Jun 24, 2017 at 18:06

1 Answer 1

3

There is no standard method or pattern for synchronization of multiple (mutating) masters, when you cannot make any design changes to one of the masters (unless it already happens to have all the features you want).

You need to be able to detect changes since last time you checked. Bulk difference is out because you can only do that once per day, plus it probably would be unusable for real-time change detection anyway.

That means using some version numbering scheme, and that's a change to the master schema as well as to the approach to modifying data since the version number scheme requires maintenance during updates.

Ideally, you'd get push notification on changes to the master, but you are saying that is out of the question.

Still, you want real-time bidirectional synchronization. So, what you're looking for is known as magic, and does not exist.


Now if you also don't have control over the clients, then it's game over.

So, assuming you do have some control over the client, you will have to modify the problem statement & requirements to something manageable. With your system constraints, there is no way to keep everything in sync near real time.

However, a client might be able function by pulling a small amount of data from the server before making a change. So, the client isn't 100% in sync, but has enough to make some minor yet accurate changes.

Further, the client should consider these changes as tentative until they are replicated to and confirmed by the server.

This goes to a different client user experience than having a single centralized master store (which is what you're trying to emulate by having real-time synchronization with multiple masters).

There's no way to hide that the changes are tentative until confirmed, and that some changes having conflicts will have to be backed out and/or redone later. So, the user experience has no choice but to address that.

1
  • "I have an impossbile-to-solve question." "Then you need magic." Perfection.
    – moodboom
    Commented Sep 8, 2020 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.