Shared file storage for several web API projects

Question

I am designing a web application that is supposed to be an 'entry point' for a few file processing tools.

Lets call the main app the 'Connector' and the tools will be 'Tool1', 'Tool2' etc.

The tools will be installed on separate servers and expose a REST API (my design).

The files should be stored somewhere in a shared central location, so that they are not transferred between the services.

The idea is as follows. The user uploads a file to the 'Connector'. I want to take the file evaluate and use a file repository to store it on some network share. The storage is yet to be decided, so I will need a generic FileProvider class that will provide access to the files - regardless whether its an azure blob, FTP server or shared folder on a network.

I will have a database accessed with EF that will hold metadata information about the files (who uploaded it, when, file name, file guid etc). I will then need to send a POST request to Tool1, to process the file. I want to only send the file ID, not the actual path.

The Tool1 is supposed to pick the file from the shared location by ID and process it - then save the processed version of this file on the same shared location - and send a success callback to 'Connector'.

Connector will then evaluate the file and maybe send a similar REST request to the 'Tool2' - again, just with the file ID. Same story, basically.

The problem I am having is... In order for the files to be picked by ID, the 'Tools' WebApi projects would have to have access to the same database. Is that a correct approach?

I wanted the Tool APIs to be pretty much independent and separate from the Connector app, and it seems to be coupling them...

Answer 1

In addition to @eddyce answer.

As you said, you have to implement a new abstraction between the Connector and the File Storage System.

So, why not a 2nd REST API?.

Lets call it StorageService. The service could provide the next contract:

• POST /store (multipart/form-data)
• GET /file/{fileId}

It has three concerns:

• save files into the file store
• save file's metadata into the DB.
• search files

Note: Using nosqlDB like mongoDB you could do all three with only one system. Instead of two. Nevertheless, I'm sure should be already a product in the cloud that does so.

It's the only one accessing to the DB and to the file storage. In consequence, the protocol between Connector and Tools would be similar to the next:

1. The user uploads a file to the Connector
2. The Connector evaluate the file
3. The Connector send the file to the StorageService
4. The StorageService store the file and save the metadata.
5. The StorageService return back the ID. It could be a URI too
6. The Connector sends the response to the Tool.
7. The Tool retrieve the file from the StorageService (or from the URI) and process it

... and so on.

It may seem more complicated than coding direct accesses to the DB and to the file storage. But, it makes a big deal better the design.

Why? Because:

• The separation of concerns of each component.
• Loosely coupling between the components.
• Easier to scale out.

Finally, the StorageService could evolve in many ways.

• Adding new file storage systems
• Providing new actions: copy, move, remove, checksum, compress, backup, etc..
• Providing security: Permissions, Authorization, Authentication, ...
• Providing public access to the files.
• ...

Answer 2

If you only want to send the file ID (ID from the database), then that's all that Tool1 and Tool2 can work with; they'll require access to the database to get the rest of the information on the file's location etc. - because they don't have anything else to go on.

For Tool1 and Tool2 to be independent, they'll need one of three options I can think of at the moment:

Option 1: Receive the actual file to process.

Option 2: Be supplied with knowledge of where the file to be processed is located.

Option 3: Have a method to call that returns the file from a given Id.

Oh, and also: Be supplied with knowledge of where to store the processed file.

Option 1 would require that you pass the physical file to Tool1 and Tool2.

Option 2 would mean that you pass the file's location to Tool1 and Tool2.

Option 3 would mean providing a way for Tool1 and Tool2 to get the file's location.

Option Oh, and also would mean being passed that information as well when using Options 1 and 2, or being able to get it if using Option 3.

How to get the information:

You could use a delegate in Tool1 and Tool2 that can retrieve the file, and subsequently another delegate to store the result.

The Connector can pass the Id of the file to Tool1 and Tool2, as well as a (publicly accessable) method that retrieves a file from a given Id. This method could be in the Connector app (if referenced), or you could use IOC to locate it, or as a separate REST API you could provide/call directly.

Finally, they would use the other delegate to pass back the processed file to the calling code to store wherever it wishes (network share, database, etc).

Tool1 and Tool2 apps could then utilise these delegates, and internally they'll never know anything regarding the database (even that it exists), where the file came from, or where the processed file is stored.

Answer 3

The main constraint of your approach is when you say:

I want to only send the file ID, not the actual path.

this complicates the matter cause you have no mean of accessing file metadata from Tool1 without having Tool1 making some kind of request, either directly to your db or to an exposed REST method somewhere in the Connector. A slightly easier solution could come from what kind of storage for the file you will use: instead of using the file system you could have another db, a key-value store only for the files and the ID would be the key to access the file and metadata infos from this new db.

In this way your connector will use the ID to store the file and then will send that ID to the Tool1 or Tool2 that will process it and maybe move it somewhere.

Ideally I'd put another REST layer in front of this new db.