1

Currently we have a MicroserviceA which has a database of objects that need to be hydrated with some KI calculated data. For this it queries MicroserviceB via REST in batches and inserts this into its own database. This works but is of course synchronous and sometimes blocking as batches are pretty large.

MicroServiceA --Insert requested data---> database
\
 \ request for data
  \
   > MicroserviceB

We want to pull this functionality out, but the data still needs to end up in database of MicroserviceA. A possible solution could be to spin up a worker, that is doing the update in the database, however this would be problematic as the database only belongs to MicroServiceA.

How is this best done? I imagine it is a common problem that a Microservice that is working as an API sometimes needs to do something resource intensive to change something in its own database. Without sharing databases only that Microservice could do updates.

4
  • If you wish to avoid blocking behavior, does this mean you are okay with fetching a half-hydrated dataset when someone queries while a batch it already running? Or do you want to hydrate in private, serve the old data until then; and "release" the newly hydrated data in a single go?
    – Flater
    Apr 7 at 9:37
  • The data before is in a "notReady" state and will be used only after being ready. After being ready the data will be updated regularly with new insights. With data I mean one integer style field which has to be calculated and might change after more historical data becomes available
    – rStorms
    Apr 7 at 9:52
  • So you would allow the fetching of data mid-batch, provided that the requested data itself was already processed by the batch, even if other batch data may still not be ready? Do you also re-hydrate existing (ready) data? Does this then get set to not ready at the start of the re-hydration, or is it then considered to be ready both before and after the re-hydration?
    – Flater
    Apr 7 at 9:56
  • What is the problem with blocking?
    – user253751
    Apr 7 at 10:41

3 Answers 3

4

Your question correctly identifies that this is not a problem of "how can I even do this?", but rather one of "how do I do this while conforming to microservice standards?". To that end, we must investigate how this conflicts with microservice standards.

It wouldn't be incorrect to consider my answer as an attempt to "lawyer" you into accepting your approach as a valid one. I've used a few different arguments here to support the notion that it may be valid to separate the api and worker without necessarily violating the core goal of a microservice architecture.


...the database only belongs to MicroServiceA

Correct, it is generally advised for microservices to be sole owner of their own datastore, as not doing so inherently infringes on the independent and scalable nature of the microservice.

a Microservice that is working as an API sometimes needs to do something resource intensive to change something in its own database.

A possible solution could be to spin up a worker

If MicroserviceA is the sole owner of DatabaseA, then any and all A-related work needs to be under the purview of MicroserviceA. Anything different would inherently mean that MicroserviceA is no longer the sole owner of that database.

That is the theory. However, in reality we often have to make compromises when presented with technical implementation challenges that stem from the purely theoretical (elegant) model.

There is no technical reason why you couldn't subdivide the API and worker into separate applications. But in order to retain the microservice way of doing things, this means having to accept some consequences from doing so

The two applications will have to share a life cycle, and should be versioned and deployed together. When one of the applications causes a change in the data model, the other application needs to inherently account for that.
In order to avoid these kinds of issues, I would err on the side of always versioning both, even when you know that a change in one does not impact the other, simply to ensure that these applications are always kept in line, easy to upgrade, and don't slowly grow apart.

In principle, this kind of "joined deployment" of logically separate modules is precisely what microservices aim to avoid. However, as long as the scope of your A-related application is small and very closely tied together; the impact of coupling them together is minimal. It would functionally be impossible to couple them any looser anyway; as the only alternative would be to merge them into a single application, which would be an even tighter coupling.

Remember when I said "the independent and scalable nature of the microservice"? If these applications had separate life cycles, you would be infringing on that. However, if you lump the two applications into a single life cycle, I can reasonably agree that you've maintained the independent nature of "the A service" (which is comprised of two deployed applications).

Really, what it all boils down to is that you architecturally speak of "the A service" as if its a single homogenous blob; whereas the technical implementation details will entail having two separately built applications to deploy.
The reason for splitting these two applications is purely for crossing an otherwise technical hurdle (runtime performance impact) and it should be architecturally irrelevant in your microservice domain.

At no point should any high level architectural decision need to account for, or be blocked by, the consequences of having to account for the separately deployed A-related applications. If you can accept these consequences, then I see no reason to consider your additional worker as something to avoid.

Or, to explain it using your words:

A possible solution could be to spin up a worker .. this would be problematic as the database only belongs to MicroServiceA

If you take this approach, you need to separate "the api" from "the microservice". Right now, you are thinking of the api as exactly being the microservice in and of itself, which is no longer true if you also have a worker.

At its core, "the microservice" is what you consider architecturally. It has a single lifecycle, and it is considered to be a homogenous blob for versioning and deployment purposes.

Concretely, "the mircroservice" actually consists of "the api" and "the worker", who could theoretically reside in a single runtime but for practical purposes are better served separately - but this separation is intentionally abstracted and hidden in the architecture. The api and worker are the silent vassals of the microservice. They make no demands of their own and cannot (and should not) override what "the microservice" should do in its lifecycle.


A more straightforward justification that I can use here is that your two applications would effectively represent the read/write components of your A service.

It is generally considered acceptable to split these out so they can be scaled independently, since a "write once, read many" approach tends to avoid concurrency issues while at the same time maximizing read capability which is commonly the only real bottleneck.

Would it be better to then also split out the underlying database in a separate read and write store? From a theoretical and elegant perspective, I agree; but here we again strike on the possibility of practical compromises (YAGNI, overengineering, no meaningful performance improvement based on current requirements).

As long as you keep it reasonably open to future maintenance whereby the read/write stores may be split off; I would consider this approach valid in the right context.

2

Your problem is that the requests to Service B might take a long time and are therefore not a good fit for “synchronous” HTTP/REST APIs. To address this, you are thinking about having a worker service that handles results by bypassing Service A and writing the results directly to Service A's database. As you have correctly noted, this is problematic. This is similar to the OOP antipattern of breaking encapsulations and changing a different object's private fields directly.

But your actual problem is that you want to handle results in an asynchronous manners, whenever they become available in the future. There are better solutions to that problem:

  • Polling: Service B could respond immediately when the long request is accepted, providing a token or URL with which the request result will eventually become available. Service A polls with this token/URL until the result is available. Example interaction:

    A>>>B POST /slow-request
    A<<<B 202 Accepted
    A<<<B Location: https://service-b.example.com/result/953c2296-fb3d-4940-8ec8-600d38de87f4
    
    A>>>B GET /result/953c2296-fb3d-4940-8ec8-600d38de87f4
    A<<<B 200 OK
    A<<<B
    A<<<B {"status": "processing"}
    
    (time passes)
    A>>>B GET /result/953c2296-fb3d-4940-8ec8-600d38de87f4
    A<<<B 200 OK
    A<<<B
    A<<<B {"status": "done", "data": {...}}
    
  • Callbacks: When Service A invokes Service B, A provides an URL with which B can signal that there is a result. Either A can then fetch the data from B as with the polling approach, or provide the data directly. Example interaction:

    A>>>B POST /slow-request
    A>>>B
    A>>>B {"on-result": "https://service-a.example.com/notify/a8593408-1a82-4ef6-a8a5-76b799409903"}
    A<<<B 200 OK
    
    (time passes)
    B>>>A POST /notify/a8593408-1a82-4ef6-a8a5-76b799409903
    B>>>A
    B>>>A {"data": {...}}
    B<<<A 200 OK
    
  • Message bus: maybe you have reached the limits of what can be reasonably handled with a REST API, and you should look into inherently asynchronous event-based systems instead that communicate over a message bus.

In any case, Service A would eventually discover that a result exists, and would then update its database itself.

1
  • Thanks for your great reply! Not sure yet which way I will go, but this helped a lot! Will comment later with what I did
    – rStorms
    Apr 7 at 9:53
1

I see two major options:

  • MicroserviceA could have a callback endpoint to be used by the worker process to insert the computed data. I suppose the actual database operation isn't complex or time-consuming.
  • But you may just as well spawn a thread within MicroserviceA to process the data and insert it asynchronously, most web service frameworks have some way of running asynchronous tasks.

Which option is best in your case depends on the specific processing to be done by the worker - if it would severely impact MicroserviceA's performance it makes sense to run this processing step in a separate process, maybe even on another machine.

Edit: After reading @amon's response there seems to be a possibility that MicroserviceA delays some response while waiting that data is processed by MicroserviceB. If that's the case, please consider changing it. Microservice interactions should be quick. If long-running operations are involved, have something like an "in progress" status on data that are not finalized, and communicate that status to client who request data that isn't ready yet, allowing them to retry.

1
  • Thanks for your great reply! Not sure yet which way I will go, but this helped a lot! Will comment later with what I did
    – rStorms
    Apr 7 at 9:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.