How to deal with file events with microservice that exposes a REST API

Question

A bit of context.

The company where I work is starting to transition to a modular architecture (something similar to microservices) for our main product, currently monolithic. The first step is to build a service that is currently not present in the product, and we want to build it outside the monolith to stop increasing the size and complexity of it.

This component is a "connector", in the sense that is something that should take care of sending messages to external parties that all use different protocols, standards and formats. It's aim is to provide a simple way for the customer to interact with these external services via standardized API/service. Currently our main product generates the messages, but it does not take care of sending them.

Here is where I do struggle. We want this new connector to expose a REST api that will enable the customer to interact with it, in order to POST a message to send it, GET details on the message, GET the list of sent messages and so on. That is all well and good. The problem is that we also want the application to being able to pick up files from a file share and send them, then show the results via REST api. This is because this is a natural improvement over the current product: monolith place files in folder X, connector picks them up.

But how should it pick them up? I have the feeling that having a microservice being approachable both via REST and react to file events is a bad idea. The servlet container takes care of all the threading when calling it via REST, so what about the files that would be picked up via file share? We are mixing synchronous REST and event-based file share here.

Maybe a better idea is to have something that picks-up the files and POST the file to the connector APIs?

The language of choice is Java, but it's more a design problem!

Thanks in advance.

Answer 1

The best solution is to change whatever generates the files so that it sends them to your api rather than saving them to files.

This allows your api to be triggered on a new file being added and do whatever is appropriate, ie save its content and meta data to a database of some sort.

If you can't change the source of the files, then yes. A tertiary application which monitors the directory and posts the files in response to IO events is better than adding that logic to the api.

Generally speaking separation is good. Something to bear in mind when you are talking about a single "connector" api. Maybe you should be thinking about multiple apis with more specific purposes?

Answer 2

You could expose the files to your managed APIs.

By for example, store your files in a cdn. With a file object storage, like storage buckets you could store files with different permissions, so they could be accessed only by using a {key,client_id} pair, or even, set policies for a grainy ip filter. As an example, this resource shows you how to do that on S3.

Object storage could offer you other advantages of a content delivery network, like a distributed file system (very useful in case the files are access worldwide).

The way you would pick up this files, allows you several solutions:

One could be to share protected file resources with the APIs by adding keys to your APIs (client_id,key) so they can access protected urls of files in the shape of:

https://files.anycdn.com/path/to/file.extension?Policy=bnQiOldGF0ZW1lt7IlJl-SaeO27zJXgtYtk&KeyPairId=APKAQADZ7BCN2PFPW8DA

This way you'll treat your APIs as clients of the file cdn.

Another one could be to mask the url, so the clients, in order to get a file, could request:

GET https://rest.api/file/endpoint/filename

In the API side, internally, GET the file (located elsewhere) and send back a HTTP stream with the file content to the client.
Or in case you have an API acting as a request dispatcher, request a stream to the file service and then forward this stream as a response.

Keep it simple, generally gives us the best results.
It possible to make the file url's protected under controlled credentials which are shared with your APIs and set a way renew this credentials regularly.