0

I'm building a distributed storage system that works with different storage sizes. For instance, my storage devices have sizes of 50GB, 70GB, 150GB, 250GB, 1000GB, 5 storage systems in one system. My application will store any files to the storage system.

Question: How can I build a distributed storage with the idea of data redundancy and fail-over to store documents, videos, any type of files at the same time ensuring that should one of any storage devices fail, there would be another copy of these files on another storage device. However, the concern is, 50GB of storage can only store this maximum number of files as compared to 70GB, 150GB etc. With one storage in mind, bringing 5 storage systems like a cloud storage, is there any logical way to distribute or store the files through my application?

How do I ensure data redundancy through different storage sizes? Is there any algorithm to collate multiple blob files into a single file archive? What is the best solution for one cloud storage with multiple different storage sizes?

I open this topic with the objective of discussing the best way to implement this idea, assuming simplicity, what are the issues of this implementation, performance measurements and discussion of the limitations.

  • This is a fairly broad topic, which is not what this site is about. What we want is for questions to be discreet, and as narrow as possible. Would you please consider breaking this down into perhaps separate questions for each of the concerns. Of course, feel free to provide links back to the other posts if you feel the context from each adds to the other, and of course any links to research you have already done on the topic helps too. – JustinC Jun 29 '13 at 6:44
2

There are two classic, NP-hard problems that relate to your question.

As they are both NP-hard, there really isn't a "perfect" a priori algorithm to handle solving the problem. So that answers your 3rd question - there isn't a "one best system" to solve this. On the other hand, those articles provide quite a few approximations you can use to "cheat" the problem and come up with a reasonable solution.

From a higher level point of view, consider having some sort of meta-data storage layer (aka a database) in order to keep track of what files are in your backup system as well where the primary and backup copies are located. That will help with your first question. Having redundant copies is simply part of your business logic and you can write rules to verify that logic is met or to take corrective action by making copies.

For your second question, the knapsack problem directly applies -- multiple blobs in a single file is a pretty simple restatement of the classic problem.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.