I am relatively new to microservice architecture. We have a moderately sized web application and I am weighing the pros and cons of breaking it out into microservices instead of a monolithic system we have now moving forward.

As far as I understand it, consider microservices A and B each of which rely on a subset of data that the other has. If a message is posted by A saying that something has changed, B can consume that message and replicate a local copy of A's info and use that to do whatever B needs to do.

However, what if B goes down/fails and after a while, comes back up again. During that down time, A has published two more messages. How does B know how to update its local copy of A's info?

Granted, if B is the only consumer of A's queue, then it can start reading it once it comes back online but what if there are other consumers of that queue and those messages are consumed?

As a more concrete example, if a Users service has its email address updated while a Billing microservice is down, if the Billing microservice comes back up again, how does it know that the email has been updated?

When microservices come back up, does it do a broadcast saying "Hey I'm back up, give me all your current info?"

In general what would be the best industry practices for data synchronization?

  • 1
    To avoid it whenever possible. – Telastyn Jul 10 '18 at 20:59
  • 1
    Why does Orders need to know anything about Users? – kdgregory Jul 10 '18 at 21:01
  • It's just an example. Replace the two with whatever you want that makes sense. – noblerare Jul 11 '18 at 3:39
  • a fan out routing will solve your 'message is consumed by someone else' problem. but its really unclear what you are trying to achieve. – Ewan Jul 11 '18 at 9:57
  • @Ewan I've updated my original post to better explain what I'm trying to ask. – noblerare Jul 11 '18 at 12:45

After doing a bit more research, I stumbled upon this article from which I've pulled some quotes out that I think is helpful for what I want to accomplish (and for any future readers). This offers a way to adopt a reactive programming model over an imperative programming model.


The idea here is to represent every application’s state transition in a form of an immutable event. Events are then stored in a log or journal form as they occur (also referred to as ‘event store’). They can also be queried and stored indefinitely, aiming to represent how the application’s state, as a whole, evolved over time.

What this helps accomplish is that if a microservice goes down yet other events pertinent to it are being published and events are consumed by, say, other instances of that microservice, when that microservice comes back up, it can refer to this event store to retrieve all the events that it missed during the period it went down.

Apache Kafka as Event Broker

Consider the use of Apache Kafka which can store and dispatch thousands of events per second and has built-in replication and fault-tolerance mechanisms. It has a persistent store of events which can be stored on disk indefinitely and consumed at any time (but not removed) from the Topic (Kafka's fancy queue) were delivered to.

The events are then assigned offsets that univocally identify them within the Topic — Kafka can manage the offsets itself, easily providing “at most once” or “at least once” delivery semantics, but they can also be negotiated when an event consumer joins a Topic, allowing microservices to start consuming events from any arbitrary place in time — usually from where the consumer left off. If the last consumed event offset is transactionally persisted in the services’s local storage when the usecases ‘successfully complete’, that offset can easily be used to achieve an “exactly once” event delivery semantics.

In fact, when consumers identify themselves to Kafka, Kafka will record which messages were delivered to which consumer so that it doesn't serve it up again.


For more complex usecases where the communication among different services is indeed necessary, the responsibility of finishing the usecase must be well recognized — the usecase is decentralized and only finishes when all the services involved acknowledge their task as successfully completed, otherwise the whole usecase must fail and corrective measures must be triggered to rollback any invalid local state.

This is when saga comes into play. A saga is a sequence of local transactions. Each local transaction updates the database and publishes a message or event to trigger the next local transaction in the saga. If a local transaction fails because it violates a business rule then the saga executes a series of compensating transactions that undo the changes that were made by the preceding local transactions. Read this for more info.

  • I still do not understand why you want to build such a complicated structure. It is usually much easier if each service just holds its own data and gives it to other services upon request. – J. Fabian Meier Jul 12 '18 at 18:38
  • ^But it will reduce availability of the system. The complicated structure might be warranted if high resilience is required. – avmohan Feb 7 at 7:43

I would challenge your whole idea of "pushing the data to all other microservices".

Usually, if a billing service needs an email address, it just asks the address service for the email address of the specific customer. It does not need to hold a copy of all the address data nor will it be informed if anything changes. It just asks and gets the answer from the newest data.

  • I think this answer is exactly right. It eliminates a lot of issues related to synchronization. In fact, I'm looking at code right now that has such issues because different services are keeping copies of information and have such synch issues. – DaveG Jul 11 '18 at 15:26
  • 1
    Thanks for your answer. So why then is there a need for a pub/sub model and message queues? If we're trying to "pull" instead of "push" data, we are worried about service latency. – noblerare Jul 11 '18 at 20:15
  • AFAIK, your service does not need to react immediately if something changes (as in a pub/sub), but occasionally needs data. Then I would just pull it. If you worry about latency, you can cache the data, but this again comes at the cost of not knowing if the data is up-to-date. If your files are large, you can also ask if anything changes before you pull something again. – J. Fabian Meier Jul 12 '18 at 7:07

You can replace a normal event queue with a publisher/subscriber model, where A service publish a new message of topic T and B type of microservices would subscribe to the same topic.

Ideally B would be a stateless service, and it would utilize a detached persistence service, such that a failed B service instance would be replaced by spawning one or more B service instances to continue its work, reading from the same shared persistence service.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.