A bit of context.

The company where I work is starting to transition to a modular architecture (something similar to microservices) for our main product, currently monolithic. The first step is to build a service that is currently not present in the product, and we want to build it outside the monolith to stop increasing the size and complexity of it.

This component is a "connector", in the sense that is something that should take care of sending messages to external parties that all use different protocols, standards and formats. It's aim is to provide a simple way for the customer to interact with these external services via standardized API/service. Currently our main product generates the messages, but it does not take care of sending them.

Here is where I do struggle. We want this new connector to expose a REST api that will enable the customer to interact with it, in order to POST a message to send it, GET details on the message, GET the list of sent messages and so on. That is all well and good. The problem is that we also want the application to being able to pick up files from a file share and send them, then show the results via REST api. This is because this is a natural improvement over the current product: monolith place files in folder X, connector picks them up.

But how should it pick them up? I have the feeling that having a microservice being approachable both via REST and react to file events is a bad idea. The servlet container takes care of all the threading when calling it via REST, so what about the files that would be picked up via file share? We are mixing synchronous REST and event-based file share here.

Maybe a better idea is to have something that picks-up the files and POST the file to the connector APIs?

The language of choice is Java, but it's more a design problem!

Thanks in advance.

  • I didn't see you mention storage requirements or data availability in your description. It sounds like you want to copy the files out of the monolithic program into your new micro service. Any particular reason for that? We've done similar things by just slapping an internal webdav server on top of the files that backend applications generate. The micro services in front of them effectively proxy requests to the webdav service with some additional business logic. – Jeremy May 10 at 16:57
  • The message delivery is a new service. Currently the monolith only produces the messages and places them somewhere, and the delivery is done by some other software. We now want to build that software as well, but without necessarily change the monolith. It should also work as stand alone thing, so picking up messages produced by software of another company. – GalacticDessert May 10 at 21:30
up vote 4 down vote accepted

The best solution is to change whatever generates the files so that it sends them to your api rather than saving them to files.

This allows your api to be triggered on a new file being added and do whatever is appropriate, ie save its content and meta data to a database of some sort.

If you can't change the source of the files, then yes. A tertiary application which monitors the directory and posts the files in response to IO events is better than adding that logic to the api.

Generally speaking separation is good. Something to bear in mind when you are talking about a single "connector" api. Maybe you should be thinking about multiple apis with more specific purposes?

  • Thank you for your suggestion Ewan. I am referring to a connector API because it will be the component (microservice) that takes care of interfacing with the market. So it will serve one single business value (market communication that is). Yes, we cannot change the monolith code at this stage, therefore probably a middleware is the best idea. – GalacticDessert May 9 at 10:21

You could expose the files to your managed APIs.

By for example, store your files in a cdn. With a file object storage, like storage buckets you could store files with different permissions, so they could be accessed only by using a {key,client_id} pair, or even, set policies for a grainy ip filter. As an example, this resource shows you how to do that on S3.

Object storage could offer you other advantages of a content delivery network, like a distributed file system (very useful in case the files are access worldwide).

The way you would pick up this files, allows you several solutions:

One could be to share protected file resources with the APIs by adding keys to your APIs (client_id,key) so they can access protected urls of files in the shape of:

https://files.anycdn.com/path/to/file.extension?Policy=bnQiOldGF0ZW1lt7IlJl-SaeO27zJXgtYtk&KeyPairId=APKAQADZ7BCN2PFPW8DA

This way you'll treat your APIs as clients of the file cdn.

Another one could be to mask the url, so the clients, in order to get a file, could request:

GET https://rest.api/file/endpoint/filename

In the API side, internally, GET the file (located elsewhere) and send back a HTTP stream with the file content to the client.
Or in case you have an API acting as a request dispatcher, request a stream to the file service and then forward this stream as a response.

Keep it simple, generally gives us the best results.
It possible to make the file url's protected under controlled credentials which are shared with your APIs and set a way renew this credentials regularly.

  • Thank you. Is it possible to have the s3 bucket post files automatically to the API? I need my apologies to react as fast as possible to new message files – GalacticDessert May 10 at 16:06
  • The S3 bucket just stores files, but you can POST files between APIs of from any client to the API, which has permissions (a write allowed key) to upstream files to the S3 bucket. – Evhz May 10 at 16:13
  • You can use S3 with a trigger to a lambda function to process the file or send it to the API. – rcdmk May 11 at 1:07

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.