I'm in the process of designing a server responsible for serving files that are between 10MB and 50MB in size.
Initially we will run two instances of the server (lets call them
fs2), with future plans to switch to a micro-service architecture, where the server instances will grow or shrink depending on the load.
These two instances need to interact with a third server running a scheduler and a file management application, as well as a database (on another server) where some metadata will be saved for clients to use.
My initial thoughts where to use a rabbitmq to allow the fs1 and fs2 to communicate with each other and the management app. the process would work as follows:
- The management app uploads to fs1 server (could be either fs1 or fs2)
- fs1 notifies fs2 and the management app when upload is complete
- fs2 contacts fs1 and stores a copy of the file
- fs2 notifies the management app when upload is complete
- The management app saves metadata to the external database
- both fs1 and fs2 can now server the files when requested
This seems OK, if there are only two instances, but once you start adding more it doesn't work.
Our ops department are very much against the idea of using the database to store files. They are worried that it will slow down the system too much. I agree it might, which is why I want a separate database for the specific purpose of storing the files and metadata. I want to build something like the following:
My thinking is that the upload service can manage uploading of files and saving of metadata to the database.
When the scheduler schedules a new job, the upload service (badly named, I know, but I'm not making that image again :-) ) can notify the file server instances that they need to cache the required file(s) from the database, which they can access directly.
The file servers won't need to cache more than 5 or 6 files each at a time.
Also, in the diagram I missed that the file management service will receive download progress messages from both file servers.
So to my questions:
- Is this a reasonable way to store files of this size for serving?
- Is this the right way to be thinking when considering the move to microservices in the future?
- Are there advantages to storing the files on the file system of each fs instance instead of just caching?
- How can I convince our ops team that storing 50MB files in a database is the way to go? what are the pros and cons?
- Any other thoughts or comments appreciated.