We have 2 to 3 dozen microservices that serve our customers. These services are deployed in a Kubernetes cluster, and they're only accessible to the outside world through 3 or 4 API gateways.

We found that sometimes the same data is needed by two or more microservices, so we evaluated a couple of strategies to solve this problem and have implemented a solution in pieces. Like any design, we are not 100% sure if we're using the right approach, and whether we're missing potential pitfalls in the design.

Case 1: When a service of lesser business importance ServiceL needs data from a service of higher business importance ServiceH, then ServiceL calls ServiceH directly to get the necessary data.

Case 2: When a service of lesser business importance ServiceL needs data from many important services ServiceH1, ServiceH2, etc), then ServiceH1, ServiceH2 publish messages with that data to RabbitMQ. The publishing of messages is fire-and-forget, so the services are not blocked. ServiceL subscribes to these messages and stores the data in its own data store. We are okay with the delay in the data becoming available to ServiceL.

Case 3: When a service of higher business importance ServiceH needs data from a less-important service ServiceL, then ServiceL publishes a message with that data to RabbitMQ via a fire-and-forget or blocking mechanism, depending on urgency of syncing the data. ServiceH consumes the message and stores it in its data store. Often the data is needed by ServiceH for reports and summary, and we are okay with the summary not being perfectly up to date at all times (eventually consistent) .

Case 4: When data is needed by two services and both of them not only read data but also modify it, then we believe the domain identification is wrong in which case we redesign them, often merging these two microservices into one.

Additional Info for Case 2 & 3: Now when we use a messaging framework like RabbitMQ for syncing data across services, over a period of time we observed data is getting out of sync between services. When data gets out of sync, we could see the statistics from RabbitMQ and replay messages, but we believe this brings in unnecessary complexity. We've ended up running jobs once a day to sync the data from the source service to the destination service, where the data is retrieved from the applicable services and not directly from their data stores.

Is this a good approach to sync data between microservices? Are there any pitfalls?

  • Could you clarify Case 4 through more detail? And what do you mean by this in "this brings unnecessary complexity"? Commented Apr 18, 2019 at 9:34
  • @MilanVelebit, I have edited the question. unnecessary complexity is not for the case 4. It's for Case 2 & 3 where RabbitMq is used. RabbitMQ supports broadcast. While broadcast, what if a message is not delivered to one of the recipient. Should be replay the message? If we replay then some recipients get that message twice. So as per our design. Sending messages is to communicate data, and in this medium of communication there is a probability of failure. And we acknowledge this probability of failure Commented Apr 18, 2019 at 12:45
  • 1
    Having a dedicated process for sync is as I would do it too, however the daily scheduling concerns me. When It comes to synch datasources, the less changes to keep in synch the better. Daily synchs are likely to perform many operations. Have you considered making the synch reactive, so that you can perform the synch based on a operations log length instead of a fixed time rate?
    – Laiv
    Commented Apr 19, 2019 at 11:03
  • Hey @Laiv, I really like your idea of "the less changes to keep in sync the better". We have a reactive mechanism to sync data. Ideally the data is synced within seconds the data is changed. This is by messages that are sent via RabbitMQ. The mechanism is however now fool proof. This mechanism could fail due to sender not being able to reach RabbitMQ, Due to a RabbitMQ going down, the receiver crashing. (It could event be that the service is newly introduced and it needs to get some data from other services) Commented Apr 20, 2019 at 2:42
  • Does RabbitMQ has HA (high availbility) set up? Additionally, in case of failure, you can Queue messages locally until RMQ is up, then you flush the messages into It.
    – Laiv
    Commented Apr 20, 2019 at 6:11

2 Answers 2


I would take a look at one of Microsoft's newer projects code named "Ambrosia" (link will take you to their Github page where the project is being developed open source) which focuses on providing a solution to this exact problem and several other major data consistency problems when developing distributed services.

The cliff-notes version is that they provide Virtual Resiliency which they desribe as the holding the following meaning:

Virtual Resiliency is a mechanism in a (possibly distributed) programming and execution environment, typically employing a log, which exploits the replayably deterministic nature and serializability of an application to automatically mask failure.

With one of the key benefits of utilizing the Ambrosia project being, that you are then provided with a layer of abstraction over the top of all the Transient Fault handling and Data Consistency problems, which are encountered thanks to the transport layer's reliability! This means that your developers DO NOT have to write any fault handling or data consistency into your code base as the underlying Ambrosia framework manages all of those cross cutting issues, as well as handling reconnecting any disconnected connections (tunnels, ssh, etc).

All of the information below, is taken straight from the project's Github page, and you can thus find this information and much, much more detailed sample use cases, etc. by following the link in the first paragraph above! I hope that this helps you guys out! It has been working great for the projects I currently am running in cloud native context!

How it works

The figure below outlines the basic architecture of an AMBROSIA application, showing two communicating AMBROSIA services, called Immortals. Each inner box in the figure represents a separate process running as part of the Immortal. Each instance of an Immortal exists as a software object and thread of control running inside of an application process. An Immortal instance communicates with other Immortal instances through an Immortal Coordinator process, which durably logs the instance's RPCs and encapsulates the low-level networking required to send RPCs. The position of requests in the log determines the order in which they are submitted to the application process for execution and then re-execution upon recovery. The Ambrosia System Architecture

Ambrosia Architecture

In addition, the language specific AMBROSIA binding provides a state serializer. To avoid replaying from the start of the service during recovery, the Immortal Coordinator occasionally checkpoints the state of the Immortal, which includes the application state. The way this serialization is provided can vary from language to language, or even amongst bindings for the same language.


I'd avoid the separation into "high-business importance" and "low-business importance". Seems like an arbitrary distinction, and one which could change from outside reasons - new business focus, better understanding of the domain etc. - leaving you with some hard-to-change architectural decisions. I'm curious to why this separation exists and what are it's implications on the development side for you?

In general, you should prefer RPCs/direct calls between services whenever possible, and leave queues and other communication mechanisms for special cases - which usually involve performance. And even in those cases, I'd much rather have some backend job doing some eventually consistent update of derived data, than have some services listening on a queue do the same thing in a "streaming" fashion. Many/all of the cases I've encountered in practice of having queues, could have been replaced by a more robust "backend job". It doesn't have to be Hadoop or anything fancy though - just a periodically running binary / cron can do it. There's always a lot of ceremony around queues and services. Many times there's a write to the db and a write to the queue. Which needs special care - tracking of which write was sent, reconciliation jobs for failed writes, retries etc. Not to mention that it makes reasoning about the system harder, backfilling super-hard, local testing grosser etc.

  • There are lot of benefits by separating services as "high business importance" and "low business importance". Let me give you an example to make it relatable. If my business is retailing online, then ProductListingService and OrderingService are very important. However SuggestionService or a jobs that crawl other competing websites are relatively less important. This classification is not arbitrary but comes from a deep understanding of the business. When services are classified like this, it helps me to prioritize bug fixes, resource allocation and scaling of services Commented Apr 20, 2019 at 3:04
  • Could you expand your thought process to say that using non-direct communication between services should only exist for special cases or performance issues ? I feel like it’s the opposite actually especially a micro service architecture as it lets every service be self sufficient and independent of other services. It also makes testing very easy as you only have to test that the right event is sent at the right time. I agree about the high/low importance part though. Commented Apr 27, 2019 at 23:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.