At my work we have a typical microservice architecture, but one of the issues we are running into is sharing data across multiple services. We have certain data that is used by multiple services and so we have to share the data across services. Our databases are separated out by service, for reference.
To give an example, we have a user service, communication service, and reward service. The user service stores all the user data including language preference and reward tier. The communication service and reward service need that data. The data is stored in the user service because it's fundamentally a property of the user model. This causes us to have to access the data outside of the microservice it's stored in. Even if it were stored in the appropriate service, running a GET to the user model would need to return this data and so we'd run into the same problem.
Right now the data is shared by making GET queries to the services for that data. However, this seems really naive, and it was largely done as an "easiest solution" answer.
On top of this, we have certain requirements on some of the data that require us to maintain consistency. We have a legal obligation to communicate to users in their preferred language. Similarly, rewards being applied from the wrong tier sounds like a problem (though perhaps that's not a critical problem).
How do we properly handle sharing this data?
One thought process is to share a read connection to the database. Reads scale very well and only the user service needs to update the data. It would be faster than running a GET between services that's for sure. I'm not overly sure I'm a fan of sharing database connections, but perhaps just for reads it's really not that bad.
Another is to make a shared service that stores data and publish to it, but having this data incur propagation delay seems problematic. We have obligations to have some data in sync, and any problems with the propagation can be huge issues. It's possible that if our data is only out of sync for a few seconds then it's likely not a big deal. Like if someone happens to get the wrong reward because of a timing issue it may not be that bad, or the communication is in the wrong language if it's basically right as it was sent out. I'm not sure if there are any that are critical that it absolutely has to be shared.
A third is to use a cache service that's shared. We have redis for our caching so we could potentially store data in there and only query services on cache misses. This causes us to have significantly faster lookup for data we're querying multiple times, but I don't know if we need that much data in our cache, and if we don't store enough we may miss too often.
Another is to store the data in the microservice where said data is critical. The problem with this is that the truth store is implied to be the user model. So if the data gets updated by an API call, it would update the microservice but then GETs to the data would temporarily return outdated data.
I'm sure this isn't a new problem but we're not sure about the best way to approach this. I'm fairly certain our current solution isn't ideal and won't scale, but I'm not sure which approach is the best.