0

In microservices oriented architecture, service A receives a file and stores it. Later Service A has to pass the data in the file to Service B. We have a choice to send the data, as:

  1. batches of rows to Service B's Api or using message queue
  2. share the file URL with Service B.

Does number 2 seem a reasonable choice? It looks very similar to Integration database, which should generally be avoided.

EDIT: Answering @pjc50 We need to process the complete file in either case, so row by row control is not applicable for our scenario.

3
  • 2
    Can only be "it depends" : do you want fine-grained control over processing, or process the whole lot? The answer might be different depending on how large the file is, as well.
    – pjc50
    Jan 5, 2022 at 9:23
  • @pjc50 We need to process the complete file in either case, so row by row control is not applicable for our scenario.
    – Asad Iqbal
    Jan 5, 2022 at 9:32
  • It's not integration database because a file store doesn't need a schema, which is the coupling point between services that is considered bad. It may very well need some kind of lifetime policy, but not a schema. Consider for yourself: What exactly is the coupling that the file store (not the file content, but the file store itself) creates between services such that they can't be modified independently?
    – davidbak
    Jan 5, 2022 at 16:49

1 Answer 1

3

I'd say number 2 is perfectly fine in theory but you can improve on it, which would alleviate some of the concerns that are leading you to doubt if it's a valid approach. In simpler terms, your cookie needs to bake a bit more until it's a good cookie and not a half-baked one.

In essence, what you're arguing here is that your file storage server is a shared resources that other microservices have access to. In a way, that means that you're really just considering your file storage server to be a microservice in and of itself.

Edit: Upon rereading, it is ambiguous if the service A from your question already acts as the file server microservice whose job it is to receive and distribute the file. If that is the sole purpose of A, then A is the microservice that I'm referring to in the rest of this answer. If A has other responsibilities, I suggest that you separate the file server microservice from service A to keep things better manageable.

Personally, I would explicitly enshrine this approach by creating an actual proper microservice in which you can upload and access files; as opposed to giving all relevant microservices access to the file server itself; which would become a file system access rights challenge for your Operations team, on top of having to face the reality of conflicts when one microservice accesses another's storage; any errors and delays encountered when there are rights issues; a possibly hellish scenario of needing to do partial backup restores; ... I'm not saying the file server microservice will be flawless in comparison, but it will make it more manageable and shift the workload away from Operations.

Instead of passing around a file URL, you'd be passing around a file reference that makes sense to the file storage microservice. That could be a path, but it could just as well be a GUID or other. File access would then be regulated by whatever authentication/authorization you're likely to already be using with your microservice architecture.

Note that this also allows you to develop an API where consumers can fetch partial data from a file; if this is relevant for your scenario.

This is no different from your integration database example. Giving everyone direct access to the database is generally a bad idea (too many cooks spoil the broth). Letting everyone call a specific microservice with its own database is more acceptable (the specific microservice is the only cook making the broth).

You can apply the exact same line of thinking to your file server example. When many actors all need to share a resource, it's generally beneficial to shield that resource behind a "resource manager" who manages the resource and its many consumers.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.