0

One of the architectural challenges we are facing on a project is ensuring data consistency over our microservice domains. We have two rules that we are trying hard to enforce: 1. Services cannot directly communicate with one another (primarily to reduce latency and prevent deadlocks) and 2. Each service only has direct access to its own database. The challenge is that there's a lot of data that we need from service to service.

For example, Users are associated with Customers, which live in the Customer Domain. However, our Jobs domain service needs to know what customers a user has access to. Ensuring that an update to the Customer Association in the Customer Domain flows into the Jobs domain is a key need.

Our current design has these updates flowing on a message queue. Basically, when Customer Domain updates a Customer Association, it drops a message on the queue and anything that cares about that change can read off that queue and update its database where relevant. This feels like a lot of stuff to maintain, though, as each domain now has to have code to listen to the MQ and process data where appropriate (and also code to push messages into the MQ).

An earlier design provided by a contractor included "Read Only Copies" of each relevant domain's database (so Jobs would have a readonly copy of the Customer Domain database), but because we're on MS SQL Server, we could not figure out a good way to create readonly slaves for those services that would be updated as the master was updated.

Are we missing something obvious here?

  • I think the obvious question here is why are you using microservices? The very concept is predicated on the idea that there are units of your application that are completely independent of one another that can therefore be developed, maintained, and deployed separately. The cost of a distributed architecture is tremendous, so I really hope you have a concrete problem you can prove this architecture is solving. Your specific issue is simply poorly drawn boundaries. – king-side-slide Feb 14 at 21:50
  • @king-side-slide the bad answer is "a contractor architect told us to." We've kept the design because of our deadlines not allowing us to change it. We will be doing an architecture review and reorientation in a few months, which is what I'm trying to prepare for with this question and other research. The reason we've gone with domain-level architecture the way we have is because we've been given an artificial limit of a small number of databases for our project, and with our "one database per service" rule, we've drawn large boundaries. – Marshall Tigerus Feb 15 at 14:16
1

Looks like you created a Microservices architecture based on your monolith data model. :-(

Customers are not supposed to be in the 'Customers Domain' (I would probably argue that Customers is not a domain).

There's probably one place where customers data are entered/updated - Registration, maybe? - and several places where a portion of customer information are needed for a specific purpose. That's where you may want to have a local copy, maybe updated listening to events coming from Registration.

The 'obvious' thing that you seem to be missing is Bounded Contexts, the DDD trick to allow for multiple cooperating models. Bounded Contexts are independent models tailored against a specific purpose, not around portions of data.

  • You are most likely correct, which is what we get for not having an in-house architect and blindly trusting the architecture of a contractor who seemed to only throw around buzzwords and not do any real work. Unfortunately, we're significantly built into what we have now, which means we're stuck until we can put forth the effort of a redesign. – Marshall Tigerus Feb 13 at 20:13
  • are there any resources you can recommend that go over how to properly architect a microservice architecture? – Marshall Tigerus Feb 13 at 20:15
  • @MarshallTigerus despite recommending resources and books is off topic here, I would suggest read Nginx' blog and Sam Newman's book Building Microservices. Both are worth readings. Additionally, It makes a big deal if you get familiar with enterprise architecture patterns. – Laiv Feb 13 at 22:04
  • @Laiv TY, I will be sure to do both. – Marshall Tigerus Feb 14 at 14:50
  • @MarshallTigerus just for your info, while blogs and books use to provide us with the "insights", they leak on details of implementation (obviously). However, once you start getting familiar with the "ideas" around the MS architectural style you start to identify the "problems", the common ones. It's then when you have to start getting familiar with architectural styles and patterns to look for "approaches to the solutions". – Laiv Feb 14 at 15:30
1

You're doing it the right way, for your architecture. It should arguably be split more into more granular chunks

Keeping updates in an asynchronous queue is good, since it lets you add and remove services without issue - both instances of a service, as well as new services. Your boilerplate for pulling from and pushing to the queue should be mostly generic and part of a base project or similar, outside of the explicit "I listen to this kind of message and do this with it" code (effectively, you should almost be able to write a ListenToThen("Message", codeToCall) declaration outside your boilerplate).

One thing you may be missing is that you should only be duplicating the data you need to duplicate. If Jobs need to know which Customers a User can access, that sounds like the Jobs only needs a mapping of User:Customer, and no other data about either. Which means the only data Jobs should have on Users or Customers is their unique ID's and the one-to-many mapping of User:Customer(s) by ID. Any more data and you're bloating your memory/database/bandwidth usage with no payoff.

Another is that you don't actually need to enforce services not talking with each other. The big goal is to not allow one service to force another to do some work that isn't public. They may still talk through public-facing API's - in fact, this is sort of encouraged. The Message on the queue says "User 24601 has updated!", so when the Jobs service picks up that message it hits the public API for /CustomerAssociation/getAssociation/24601 to get the data it needs. The important part is that Jobs doesn't care who or what the CustomerAssociation is. It might be a service similar to itself, it might be a monolithic app outside the cluster it works in, it could even be a function in your favorite cloud provider that didn't spin up until requested. The key isn't to avoid all dependencies, the key is to avoid hard dependencies. If you get slaves of databases, you're now requiring that every service you create must have a database, and must be using a database technology that allows slaves, and you must be using a technology yourself that can talk to those slaves correctly - if you decide to build a service that doesn't need any data of its own (like a Monitor service that gets statistics for "how's the system performing right now" from other services)... it must have a database because we're not allowing direct communication between services, which means it must use technology which can talk to compatible databases, etc. If you're using an async queue to exchange "anytime" messages and then public APIs to communicate directly when needed, your Monitor could be an AWS lambda, or an Elixir service, or any number of other things. You gain the ability to use the right tool for the job without constraining yourself.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.