4

In a microservice architecture, it's commonly admitted that each microservice should have its own replica from another "main" microservice acting as single source of truth. This keeps microservices autonomous and loosely coupled.

When the "main" microservice changes, it emits some events, so that interested microservices update their own replica and keep in sync. By using a Broker with persistent queues, we can "quite" ensure that no events get lost and replicas keep up to date.

But "quite" is not 100%, and there still are other ways of getting ouf of sync :

  • If a newly developed microservice joins the system, how does it build its replicate (since all events have already been emitted) ? Should we introduce again some kind of (anti-pattern) sync communication to query the whole data from the "main" microservice and build the replica ?
  • When should this process take place ? At the system startup (on a device) ? Periodically on a cloud architecture ?
  • Should we block the whole system until this synchronization is over (we don't want the "main" microservice to emit new events while we are synchronizing, because we get the risk to get out of sync again)

How did you solve these problems on your implementations ? I've seen somewhere the concept of "reconciliation" but I did not find any implementation of that concept.

Many thanks !

Improve this question

2 Answers 2

Reset to default
4

First off, no your microservice shouldn't have a copy of another microservices data. Each microservice should only have its own data and make calls out to other apis if required. Although the design of your system to avoid lots of those calls is key.

However, this need to replay past or missed events to catch up is a common problem in event driven systems.

Solution 1.

Add a Get past events API in addition to the push messages. This allows catch up, checking for missed messages and other scenarios. It's not really an anti pattern unless you are forced to use it so much that you are basically admitting the push messages are not trusted to work.

I think its fairly common to add such an interface and have some sort of check/sync job which audits your overall system on a schedule. say for example you have jobs which haven't moved on in the expected SLA, it might be because a message was missed or errored, you might want to poll for missed messages on these delayed jobs.

Solution 2.

Change from a queue based system to a streaming database like kafka. a streaming database will support replaying events from a point in time, giving you a method for catch-up and spotting missed messages.

Solution 3.

Add a separate replication of existing data process that you can 'manually' apply. This can be useful where you have a specific one off process that needs the full info, say deploying a new tennant. You might need a large amount of base data before subscribing to the push messages, too much for an API, but doable with an export/import flat file on a memory stick

Improve this answer
4
  • Thanks for your ideas :) Yes as I said "each microservice should have its own replica" I meant "subset of data" not a while copy of the database ;-) Solution 1 is was I was having in mind. I also think it's not possible to completely avoid synchronous communication. Since our application will be first deployed first on an embedded device (then later on a PC, cloud...), we could perform some sync check on device restart, disk failure, or SLA mismatch as you stated. Solution 2 is not conceivable because of limited memory on embedded device. Solution 3 is the way to go for new microservices
    – djflex68
    Sep 8, 2022 at 13:41
  • with some thought you might be able to completely avoid the sync requirement, and it might make the over all system 'cleaner' but... having a sync method is easy and understandable and covers so many things
    – Ewan
    Sep 8, 2022 at 14:52
  • "the design of your system to avoid lots of those calls is key" - one way to avoid it is replication
    – user253751
    Sep 8, 2022 at 18:22
  • effectively that is the same as not having microservices
    – Ewan
    Sep 8, 2022 at 18:34
0

Normally you would connect to a database that you choose to your specific needs. There are some which guarantee transactional consistency (which you should avoid at all cost), while still (somewhat) allowing scaling, there are a ton of databases with eventual consistency that will take care of all the stuff you described automatically, possibly making your services essentially stateless.

If you want to hand-roll "reconciliation" for some reason, you could use Kafka for example. It gives you access to this "reconciliation" process (for joining and exiting nodes), while taking care of all the details and making sure it occurs in a safe and predictable way. It is all based on the concept of assigning a single node to partitions of data.

Improve this answer
7
  • Yes my point was not really about the database choice, but either to ensure data kept sync in a distributed µs environment. To do that, you must emit messages, and each interested µs updates its own database subset. I think you can't 100% rely that the broker would never miss any message. Although it shouldn't, you can't be sure you'll never have some disk failure... Even if this is the case in 0.0001% of the time, it can just unsync the whole system if not handled correctly. So what I meant with reconciliation is what @Ewan told in its solution1. Some topics on this subject would be great
    – djflex68
    Sep 8, 2022 at 13:54
  • 2
    Oh, I misunderstood you. I thought you are speaking about microservice instances of the same kind. Sharing data between different microservices is definitely an anti-pattern. The whole point of microservices is to be independent. There is no "single source of truth" in this meaning. There can not be. It defeats the purpose of having microservices in the first place. "Getting" data from other services, as suggested by Ewan is also an anti-pattern for the same reason. It indicates that the boundaries between services are likely wrong. Rare exceptions may apply, depending on requirements.
    – Robert Bräutigam
    Sep 8, 2022 at 15:29
  • The way I understood it from its comment (and how I see it), wasn't to always get data directly from other services (by using direct communication for example). This would in fact defeat the purpose of microservices by having strong dependencies. Instead we would just have a kind of "internal" API or Event, for System consistency checking purpose. When it's raised, each microservice would check that it is in sync with the "main" microservice (by getting events since the last snapshot or via REST API call...). This should help to identify and repair the (very few) cases of a broker failure
    – djflex68
    Sep 9, 2022 at 7:32
  • @djflex68 That could be the case under special requirements. I'm just pointing out, that there should not be a "main" microservice. Again, depends heavily on why you have microservices in the first place. Normally it is to scale either operationally or organizationally. Having a "main" anything is a major problem for both cases, regardless of direct communication, events or queues.
    – Robert Bräutigam
    Sep 9, 2022 at 7:45
  • By "main" microservice I mean the service which holds the original data. Lets take an example using 2 bounded contexts (DDD): Orders and Customers. The Order microservice will probably have a (subset) replica of the customers (to remain independent). Whenever a customer changes, the Customer aggregate would emit an event, so that the Order aggregate updates its internal Customer replica. Here it's quite obvious that the "main" microservice which holds the truth about "Customers" is the Customer Microservice itself. In case of a failure, other microservices should resync based on it, or not ?
    – djflex68
    Sep 9, 2022 at 9:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.