I am working on a project to save external credit bureau reports in database. These reports are typically big and could go from 0.5MB to 5.0 MB. Number of files will grow exponentially over the time based on incoming traffic. Estimated file size would be 1TB in 3 months. Files will be read immediately and multiple times by other services/applications once its retrieved from external bureau in real time and saved it in data store. These files will be associated with an uniqueId and file / credit report retrieval will be based on that uniqueId. Response time is one of the critical factor as online experience is associated with it.

We are searching couple of options like: 1. SQL blob store with uniqueId 2. NoSQL column or document store with uniqueId 3. Object store with a path mapping in SQL DB against an uniqueId.

Backup / Archive / overall maintenance would play a critical role down the line as our app should be up 24x7 with 99.9999 %.

Need some input and guidance here to finalize a potential database option based on above scenario.

  • Sharing your research helps everyone. Tell us what you've tried and why it didn't meet your needs. This demonstrates that you've taken the time to try to help yourself, it saves us from reiterating obvious answers, and most of all it helps you get a more specific and relevant answer. Also see How to Ask – gnat Apr 17 at 17:44
  • If the number of files will really grow "exponentially", as you wrote, what's your estimated file size after 12 months (1 exabyte?) – Doc Brown Apr 19 at 7:06

Backup / Archive / overall maintenance would play a critical role down the line as our app should be up 24x7 with 99.9999 %.

If the 99.9999 % figure is a serious one, there is no single solution, and the cost is significant (e.g. millions of US$ or €).

But consider a difficult mix of :

  • toplevel and costly software development, system administration and management (human) skills
  • code reviews (including by external experts)
  • careful choice of the software stack, including popular open source solution (seriously supported Linux distribution, well chosen and well configured PostGreSQL)
  • careful and independent evaluation of the software (including your Linux kernel), notably with static analysis tools (à la Frama-C)
  • careful system administrations
  • careful choice of the software tools, e.g. the CompCert compiler.
  • careful choice of the hardware

Good luck. You are making a scary bet.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.