Problem
I am currently investigating a solution to allow fast upload (and retrieval) of data that can be implemented via NoSQL/SQL or a file system on a Centos server with 64 core cpu with 529GB of ram.
The case is as follows:
The users have a need to keep a track of data uploaded onto a server, then at some point retrieve all the data stored on server ready for processing on a different system.
The operations on the data involve upload/import and download/export.
each record uploaded will range between 4mb-5mb
Options
To handle the large amount of data I believe the server we have is more than efficient enough to handle the load. However, we’re looking for a solution to make sure that scalability and backup isn’t going to be an issue in the future. Some options I have considered are:
- mongoDB(GridFS to chunkate the data).
- Oracle database (the LOB compression and datafield).
What are your thoughts on the best approach to take for this project? Has anyone had a similar dilemma and what was done to solve it?
Thanks, in advance for your help.
EDIT More Information
The estimated records we are expecting are 2400 as a worst case scenario 9600-12000mb in a day and will arrive in a burst. The users will need a sub select of data and will be used on corporate gigabit LAN network with standard Ethernet cables