0

At my work we have a typical microservice architecture, but one of the issues we are running into is sharing data across multiple services. We have certain data that is used by multiple services and so we have to share the data across services. Our databases are separated out by service, for reference.

To give an example, we have a user service, communication service, and reward service. The user service stores all the user data including language preference and reward tier. The communication service and reward service need that data. The data is stored in the user service because it's fundamentally a property of the user model. This causes us to have to access the data outside of the microservice it's stored in. Even if it were stored in the appropriate service, running a GET to the user model would need to return this data and so we'd run into the same problem.

Right now the data is shared by making GET queries to the services for that data. However, this seems really naive, and it was largely done as an "easiest solution" answer.

On top of this, we have certain requirements on some of the data that require us to maintain consistency. We have a legal obligation to communicate to users in their preferred language. Similarly, rewards being applied from the wrong tier sounds like a problem (though perhaps that's not a critical problem).

How do we properly handle sharing this data?

One thought process is to share a read connection to the database. Reads scale very well and only the user service needs to update the data. It would be faster than running a GET between services that's for sure. I'm not overly sure I'm a fan of sharing database connections, but perhaps just for reads it's really not that bad.

Another is to make a shared service that stores data and publish to it, but having this data incur propagation delay seems problematic. We have obligations to have some data in sync, and any problems with the propagation can be huge issues. It's possible that if our data is only out of sync for a few seconds then it's likely not a big deal. Like if someone happens to get the wrong reward because of a timing issue it may not be that bad, or the communication is in the wrong language if it's basically right as it was sent out. I'm not sure if there are any that are critical that it absolutely has to be shared.

A third is to use a cache service that's shared. We have redis for our caching so we could potentially store data in there and only query services on cache misses. This causes us to have significantly faster lookup for data we're querying multiple times, but I don't know if we need that much data in our cache, and if we don't store enough we may miss too often.

Another is to store the data in the microservice where said data is critical. The problem with this is that the truth store is implied to be the user model. So if the data gets updated by an API call, it would update the microservice but then GETs to the data would temporarily return outdated data.

I'm sure this isn't a new problem but we're not sure about the best way to approach this. I'm fairly certain our current solution isn't ideal and won't scale, but I'm not sure which approach is the best.

  • Sounds to me like there is another service sitting in there waiting to break out. Either each piece of that shared data is different (its a perspective) in which case its not shared, Or each piece of data is identical and copied to each service verbatim in which case listen to the data and break it out. Take a hard look at why it is fundamentally a member of each service and see how it is used. If you could redraw the boundaries between services would it naturally arise? You might surprise yourself. – Kain0_0 Feb 5 at 23:28
  • The data belongs to a user, it's a property of them. It makes sense for the get and set for that data to be hitting the user model. A user has a reward tier, that just naturally makes sense. The data is used for calculating rewards by the reward service. The data is definitely identical. You can't join them together without creating a monolithic service. Making more services is just option 2. If querying the data has to hit a middle service then it's no better than what we already have. – Michael Yousef Feb 6 at 16:24
  • Does the User Service actually do anything with the data. Besides providing CRUD operations? If it does not actually do anything, why could that data not be referred too? Even if it does do something, are those actions almost identical/similar to the rewards service? Also natural sense, who's natural sense? Yours? How did you get this natural sense? Take a step back, look at the data and behaviours of your system. This duplication is telling you that the current services are wrong and do not fit. So treat it like a monolith, then figure out how to decompose it back into good services. – Kain0_0 Feb 6 at 22:28
  • No, it doesn't do anything other than CRUD. The issue comes from a concern of how CRUD accesses that data. Let's suppose for a minute that I put it entirely in the rewards service, how do I access it? By hitting /rewards/<user_id>/something? Does that make more sense than doing /users/<user_id>/update and giving it a new reward tier? From the perspective of an API, you end up with too many endpoints doing far too little if you take this approach. – Michael Yousef Feb 7 at 13:55
  • Hmm, I think that is the problem. You are looking at how you wish to present state to a downstream system. In this context I would agree that rewards appear to model along user lines. However Micro Services are not about this. They are about containing state and behaviour together in a tightly coupled module, which is only vaguely coupled to other services. You can obtain the logical grouping on the API by routing various sub-paths to different service behind the api. eg /user/X/reward can route to the rewards service, while /user/X/profile routes to the user profile service. – Kain0_0 Feb 10 at 10:56

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.