4

I've been reading though Architecture for containerized .NET applications book, and I have a question about replicating data from from service into another:

And finally (and this is where most of the issues arise when building microservices), if your initial microservice needs data that’s originally owned by other microservices, do not rely on making synchronous requests for that data. Instead, replicate or propagate that data (only the attributes you need) into the initial service’s database by using eventual consistency (typically by using integration events, as explained in upcoming sections).

If we deploy two services at the same time, I understand how we can replicate a subset of the data from one service into another using events, but what happens if the second service (that needs to replicate data) is deployed at a later stage when the first service already has data in its database? How can the new service "catch up" and get in sync?

Is it reasonable for the new service to use a synchronous call the first time it starts, and seed its own database? Should there be some sort of script that runs and reads from database A and writes to database B?

Improve this question

1 Answer 1

Reset to default
9

From your quote

do not rely on making synchronous requests for that data. Instead, replicate or propagate that data (only the attributes you need) into the initial service’s database by using eventual consistency

So no, it does not sounds reasonable to do a synchronous big-bang import from one data store into another. At least not if we consider what we are learning from the book. The release of the new service should not depend on data it has no accountability. If the data is already in the data store, then good. If not, it should be confident that at some point it will have it. The whole point of MS is they can appear or disappear at any time within the system's lifespan.

Ideally, the new service gathers data as it arrives from the existing system and saves only the necessary one (besides the one generated by the service itself) even if this data already exists in someone else's database. That's the " data duplication ". Each service might hold data (sometimes only chunks of it) that already exists in other domains (e.g the customer name can be in many databases each of which is managed by a different service).

New services should not be born upon the premise that concrete services already exist. That will couple and drag'em forever at different levels, not only at the technical level.

The autonomy doesn't only involves IPCs (inter-process communication protocols), it also involves the ALM (application lifecycle management). Relying on synchronous calls might lead you to ask another team for a new feature they don't have. Imagine deploying tens or hundreds of services a year. It can impacts everyone's roadmap, making project management unbearable. Leave alone product management.

However, it's reasonable for new services to need (historical) data. This can be achieved in many ways. One could be broadcasting a sort of newServiceRegistered event so existing services can react to this one and push data. How to move this data without direct communication between services is another problem to solve. I guess that's what the book will introduce in sections explaining the integration events.

That's all the theory and it might or might not fit your current needs and expectations.

If you can provide the new service with the required data without involving coupling among services, say by scripts executed once during the release, ETLS, etc, then it's fine. But bear in mind that, as new services appear, more of those scripts are needed and how to generate the scripts for each data store and how to execute'em from environment to environment can be a lot of work and could end up impacting several teams. Releasing a new service is not just deploying something to the wild, there's a whole ceremony around that event. It's not rare that MS dev teams build their own tools to feed and operate the service. If the infrastructure doesn't provide any mechanism, then they have to create it.

The goal is always the same: _to be as autonomous and independent from other services/teams/management as possible.

Improve this answer
5
  • 1
    Thank you for this answer. Much appreciated! I suspect that if a new service has a dependency on the historical data of another, it could be a sign that it shouldn't be a new service at all, but rather a feature in the existing service.
    – Q-bertsuit
    Nov 17, 2022 at 14:29
  • 1
    Not necessarily. A SuggestionService can perfectly depend on data generated and gathered by other services. In that case, worth delving into the subject of integration events. How can the new SuggestionService ask (to the system as a whole) for data of all sorts so it can digest it and generate suggestions. Maybe it leads you to a new MS. Let's say DataFlowService whose main purpose is handling input requests with data typed as T, generate an output with format F and sends it to a remote location (file system, DB, service, queue, etc)
    – Laiv
    Nov 17, 2022 at 15:04
  • That’s an interesting thought. But wouldn’t you just move the problem to the new MS? Wouldn’t that MS need to perform a synchronous call into the services it was collecting data from?
    – Q-bertsuit
    Nov 18, 2022 at 6:30
  • Yes, but there's no absolute decoupling. That's an expensive and unaffordable quimera for most of us. As in source code, we should recognise when, where and what kind of coupling is acceptable and enough. Note that, your system (lets say your business services) operates within an infrastructure which provides the business with dedicated services. E.g service discovery, messaging, gateways, databases, etc. It's reasonable that your MS knows about them. But please, use the proper ways to do so (DNS, convention over configuration, etc.).
    – Laiv
    Nov 18, 2022 at 11:59
  • Decoupling your MS from the infrastructure is another monster to feed apart.
    – Laiv
    Nov 18, 2022 at 12:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.