Data Replication Across Microservice Domains

Question

One of the architectural challenges we are facing on a project is ensuring data consistency over our microservice domains. We have two rules that we are trying hard to enforce: 1. Services cannot directly communicate with one another (primarily to reduce latency and prevent deadlocks) and 2. Each service only has direct access to its own database. The challenge is that there's a lot of data that we need from service to service.

For example, Users are associated with Customers, which live in the Customer Domain. However, our Jobs domain service needs to know what customers a user has access to. Ensuring that an update to the Customer Association in the Customer Domain flows into the Jobs domain is a key need.

Our current design has these updates flowing on a message queue. Basically, when Customer Domain updates a Customer Association, it drops a message on the queue and anything that cares about that change can read off that queue and update its database where relevant. This feels like a lot of stuff to maintain, though, as each domain now has to have code to listen to the MQ and process data where appropriate (and also code to push messages into the MQ).

An earlier design provided by a contractor included "Read Only Copies" of each relevant domain's database (so Jobs would have a readonly copy of the Customer Domain database), but because we're on MS SQL Server, we could not figure out a good way to create readonly slaves for those services that would be updated as the master was updated.

Are we missing something obvious here?

Answer 1

You're doing it the right way, for your architecture. It should arguably be split more into more granular chunks

Keeping updates in an asynchronous queue is good, since it lets you add and remove services without issue - both instances of a service, as well as new services. Your boilerplate for pulling from and pushing to the queue should be mostly generic and part of a base project or similar, outside of the explicit "I listen to this kind of message and do this with it" code (effectively, you should almost be able to write a ListenToThen("Message", codeToCall) declaration outside your boilerplate).

One thing you may be missing is that you should only be duplicating the data you need to duplicate. If Jobs need to know which Customers a User can access, that sounds like the Jobs only needs a mapping of User:Customer, and no other data about either. Which means the only data Jobs should have on Users or Customers is their unique ID's and the one-to-many mapping of User:Customer(s) by ID. Any more data and you're bloating your memory/database/bandwidth usage with no payoff.

Another is that you don't actually need to enforce services not talking with each other. The big goal is to not allow one service to force another to do some work that isn't public. They may still talk through public-facing API's - in fact, this is sort of encouraged. The Message on the queue says "User 24601 has updated!", so when the Jobs service picks up that message it hits the public API for /CustomerAssociation/getAssociation/24601 to get the data it needs. The important part is that Jobs doesn't care who or what the CustomerAssociation is. It might be a service similar to itself, it might be a monolithic app outside the cluster it works in, it could even be a function in your favorite cloud provider that didn't spin up until requested. The key isn't to avoid all dependencies, the key is to avoid hard dependencies. If you get slaves of databases, you're now requiring that every service you create must have a database, and must be using a database technology that allows slaves, and you must be using a technology yourself that can talk to those slaves correctly - if you decide to build a service that doesn't need any data of its own (like a Monitor service that gets statistics for "how's the system performing right now" from other services)... it must have a database because we're not allowing direct communication between services, which means it must use technology which can talk to compatible databases, etc. If you're using an async queue to exchange "anytime" messages and then public APIs to communicate directly when needed, your Monitor could be an AWS lambda, or an Elixir service, or any number of other things. You gain the ability to use the right tool for the job without constraining yourself.

Answer 2

Looks like you created a Microservices architecture based on your monolith data model. :-(

Customers are not supposed to be in the 'Customers Domain' (I would probably argue that Customers is not a domain).

There's probably one place where customers data are entered/updated - Registration, maybe? - and several places where a portion of customer information are needed for a specific purpose. That's where you may want to have a local copy, maybe updated listening to events coming from Registration.

The 'obvious' thing that you seem to be missing is Bounded Contexts, the DDD trick to allow for multiple cooperating models. Bounded Contexts are independent models tailored against a specific purpose, not around portions of data.