I am working on an integration project, where my “app/web service” will sit in the middle serving documents.
Basically, a request is sent with the document id as part of the query string, I check if it exists and if it does, write the document (pdf) back to the response. Now, this is very simple to do and we have done this many times.
This is where it gets very tricky, there will be about 5200 new files added each week, that’s 3GB per week (so about 150GB of data per year). We are expected to keep 10 years of data.
What is the best way of storing these documents and searching has to be very quick.
Some options are:
If I choose to save all these files on a network share, initially it will be quick but over time it is going to be very slow.
If I choose SQL to store these files, I can’t really store all documents in a single table or utilize table partitioning, DB partitioning etc. If I go with DB partitioning that’s still 150GB DB file each year to backup and restore.
I was thinking implementing RBS for MS SQL but not sure how quick this is going to be.
Any suggestions or other options I might have....?