Share files between microservices

Question

We have a service (lets say FileService) that provides API to store/download files into various stores (S3/local file etc.,)

We have a scenario where one microservice (Service A) writes a file to S3 via FileService and the same file needs to be read by another microservice (Service B). In an ideal scenario, Service A will have an API exposed to read/download this file; Service B can leverage this API and be able to read it.

However, due to the file(s) being huge, we wanted to see if its OK to have Service B read/download the file directly via FileService. (Given that Service A agrees that its fine to provide read access to Service B)

In this case as the data being shared across microservices is files, is this an acceptable pattern ? Do we foresee any issues with this approach ?

Answer 1

The goal in a microservice architecture is to reduce dependencies shared by multiple services, so that if, e.g. the FileService needs to change the file system or structure, other services are not affected.

Telling the consumer service "you can find your file in path x/y/z.dat" can harm that goal because other services now depend on a file system with a given structure, need to know how to connect and navigate the path, and so on.

But what you need to remember is that your service's API isn't limited to just the method calls on it. If your files are stored in an AWS bucket or any HTTPS-accessible file system, all your file service needs to do is supply the caller with an opaque URL, which the caller then retrieves. The caller doesn't know about file systems or paths - it gets a URL and retrieves it. End of story.

Answer 2

In most situations this is a Bad Idea™.

It breaks encapsulation, but much worse it opens you up to supply chain risk.

That FileService API isn't guaranteed to function the sameway tomorrow. It might get a bug fix, a new bug, or suddenly sprout new (and possibly) breaking features. Eitherway its change that you won't see coming, and you won't be able to do anything to ameliorate that risk till its already happened. (Even if you a subscribbed to their release schedule/maintenance blog/or other change notification channel).

That being said if your specific circumstances are that:

• Service A and Service B are services within your own company.
• The FileService API can be accessed securely from Service A and Service B without exposing it to external clients.
• You have a system for tracking interfaces between your services that can describe this scenario.
• And you've done your due diligence to ensure that this does not bypass business logic.

Then its a reasonable decision to make. Its no different to providing file-drops via a shared-drive. There is some additional complexity in securing the interface, and in ensuring that partial files aren't mistakenly processed, but this is what mainframes having been doing for decades.

As for downsides it certainly makes the api more complicated, as Service A essentially exposes the FileService as part of its interface. That's not much of a downside if Service A is merely providing extra orthogonal functionality (its a strict superset of FileService), or if this is a literal file-drop making the FileService API a communication channel not an internal storage mechanism.