What is the proper way to synchronize data across microservices?

Question

I am relatively new to microservice architecture. We have a moderately sized web application and I am weighing the pros and cons of breaking it out into microservices instead of a monolithic system we have now moving forward.

As far as I understand it, consider microservices A and B each of which rely on a subset of data that the other has. If a message is posted by A saying that something has changed, B can consume that message and replicate a local copy of A's info and use that to do whatever B needs to do.

However, what if B goes down/fails and after a while, comes back up again. During that down time, A has published two more messages. How does B know how to update its local copy of A's info?

Granted, if B is the only consumer of A's queue, then it can start reading it once it comes back online but what if there are other consumers of that queue and those messages are consumed?

As a more concrete example, if a Users service has its email address updated while a Billing microservice is down, if the Billing microservice comes back up again, how does it know that the email has been updated?

When microservices come back up, does it do a broadcast saying "Hey I'm back up, give me all your current info?"

In general what would be the best industry practices for data synchronization?

Answer 1

After doing a bit more research, I stumbled upon this article from which I've pulled some quotes out that I think is helpful for what I want to accomplish (and for any future readers). This offers a way to adopt a reactive programming model over an imperative programming model.

Event-sourcing

The idea here is to represent every application’s state transition in a form of an immutable event. Events are then stored in a log or journal form as they occur (also referred to as ‘event store’). They can also be queried and stored indefinitely, aiming to represent how the application’s state, as a whole, evolved over time.

What this helps accomplish is that if a microservice goes down yet other events pertinent to it are being published and events are consumed by, say, other instances of that microservice, when that microservice comes back up, it can refer to this event store to retrieve all the events that it missed during the period it went down.

Apache Kafka as Event Broker

Consider the use of Apache Kafka which can store and dispatch thousands of events per second and has built-in replication and fault-tolerance mechanisms. It has a persistent store of events which can be stored on disk indefinitely and consumed at any time (but not removed) from the Topic (Kafka's fancy queue) were delivered to.

The events are then assigned offsets that univocally identify them within the Topic — Kafka can manage the offsets itself, easily providing “at most once” or “at least once” delivery semantics, but they can also be negotiated when an event consumer joins a Topic, allowing microservices to start consuming events from any arbitrary place in time — usually from where the consumer left off. If the last consumed event offset is transactionally persisted in the services’s local storage when the usecases ‘successfully complete’, that offset can easily be used to achieve an “exactly once” event delivery semantics.

In fact, when consumers identify themselves to Kafka, Kafka will record which messages were delivered to which consumer so that it doesn't serve it up again.

Sagas

For more complex usecases where the communication among different services is indeed necessary, the responsibility of finishing the usecase must be well recognized — the usecase is decentralized and only finishes when all the services involved acknowledge their task as successfully completed, otherwise the whole usecase must fail and corrective measures must be triggered to rollback any invalid local state.

This is when saga comes into play. A saga is a sequence of local transactions. Each local transaction updates the database and publishes a message or event to trigger the next local transaction in the saga. If a local transaction fails because it violates a business rule then the saga executes a series of compensating transactions that undo the changes that were made by the preceding local transactions. Read this for more info.

Answer 2

Even if I'm late, I would to put my 2 cents on the argument because I think it's an important point when you want to evaluate e design an event-driven microservices architecture. Each microservice knows exactly which are the events that impact on its state and is able to wait for them. When the microservice is not avaliable, there should be a component that keeps messages that are needed from the failed microservice until it is not able to "consume" them. This is in fact a "producer/consumer" model and not a "publish/subscribe" one. Message brokers (like Kafka, RabbitMQ, ActiveMQ etc. ) are usually the best way to achieve this behaviour (unless you are not implementing something different such as event sourcing) providing persistent queues and ack/nack mechanism.

Now the microservice know that a message is eventually delivered but it is not enough: which is the way it expects the delivery of a single message? can it manage the delivery of multiple copies of the same event notification? This is matter of delivery semantic (at least once, exactly once)

Final thought(s):

1. When you add a microservice to your architecture that needs to consume events from others, you have to do the first sync

2. Even the broker can fail, in this case messages are lost

for both scenarios, it would be useful to have simple mechanisms to re-hydrate you microservice state. It could be A REST API or a script that sends messsages, but the most important thing is to have means to do some maintenance task

Answer 3

You can replace a normal event queue with a publisher/subscriber model, where A service publish a new message of topic T and B type of microservices would subscribe to the same topic.

Ideally B would be a stateless service, and it would utilize a detached persistence service, such that a failed B service instance would be replaced by spawning one or more B service instances to continue its work, reading from the same shared persistence service.

Answer 4

If a message is posted by A saying that something has changed, B can consume that message and replicate a local copy of A's info and use that to do whatever B needs to do.

If you wanted B to be able to access A's internal data, you would be better of to just give it access to A's internal databases.

However you should not do that, the whole point of a service oriented architecture is that service B cannot see service A's internal state and is limited to make requests through REST APIs (and vice versa).

In your case you could have a user data service, which has the responsibility of storing all user data. Other services that want to use that data only request it when they need it and don't keep a local copy (which btw. is really useful if you think about GDPR compliance). The User Data service can support simple CRUD operations like "Create new user", or "Change name for user_id 23" or it can have more complex operations, "Find all standard users with a birthday coming up in the next 2 weeks and give them premium trial status". Now when your Billing service needs to send an email to user 42, it will ask the User Data service "What is the email address for user_id 42", use its internal data with all the billing information to craft the email and then may pass the email address and body to a mail server.