Is caching data between independent but interacting systems okay?

Question

We have two products let call them X and Y. Product-X is established and has numerous clients over continents whereas Product-Y's development was started a few years ago and it has yet to enjoy any client.

We came across a scenario where we have to sync data and actions between both systems. This can be done at any stage e.g. clients may use systems on their own and on a later date would like to sync them.

Now we will always host Product-X on our server farms. However clients may or may not do that for Product-Y and host it locally on there own server farms. I am very new to designing systems so I am calling these systems distributed systems which may not be the right term.

Both products are different and they will always have their own database and structure regardless of whether they are in same environment or in client's server farm.

Example

• Synching data based on events, e.g. if someone creates something in Product-X then it needs to be created in Product-Y too but following different logical steps and same for when something is created or updated in Product-Y.

Possible Solution

• Any operation happening in any system will push the data to the other system using generic services and the other system will decide what to do with it.

• Create a cache of data on both ends e.g. in SQL Tables. This means both system will never ask each other for data or if they need something they will only send a flag and get a callback with data.

• Both systems will have endpoints (WCF services) to connect to each other.

Question

• Is it okay to keep a cache of data if not then what would be the right design for this scenario?

Answer 1

Your requirements are:

• you have two independent products, that manage overlapping data (according to your two first paragraphs)
• the two products shall remain independent with their own databases and would in general be run on distinct servers with their own database (according to your third and fourth paragraph)
• you want to find a data synchronization mechanism between the two systems

It appears that these requirements share many commonalities with the microservice architecture, even if each of your system could in reality be a monolith of its own. The key is loosely coupled and interacting.

Therefore you may be interested in the common microservice patterns, which offer solutions to these issues. In particular, it's quite common in this world to have a database per service (per system in your case) which remain in synch using an event driven architecture.

Now your question is basically looking at the best way to communicate the events; should you:

• connect the endpoints via remote invocation,
• or use messaging with either a message queue (e.g. RabbitMQ) or event streams (e.g. Kafka),
• or use domain-specific approaches

Basically, all three have their own speicific forces (see links provided). Personnally, and intuitively, I'd opt for an event stream, and organize future developments on this paradigm. But I don't know enough on your context, so don't bet on it without making first your own analysis.

The last point with the cache, is a little different. Here it's not about communication, but about whether to replicate data or not. If the overlapping data is only there to trigger some actions, not need to replicate (this would even more be in favour of event streams). If the overlapping data is complementary and is often used together, you can replicate, according to the motto "one replication is worth a millions of additional querries". But again, it's up to you to analyse how to best ballance time vs. space.

Answer 2

The way I see it, there's nothing wrong with what you trying to achieve. Anyhow I'd try to look for a bottleneck first, do you know if you're going to have latency/performance issues? Take into account that you're transferring the workload from the data access process to the data modification process, you may be just creating a problem while trying to solve one you still don't know you have.

Evaluate carefully the time required for data writes and the time require for data reads. Also evaluate what data is a good candidate for caching; this usually means low write frequency an high read frequency.