I was brainstorming and I found quite an interesting topic I couldn't come up with a solution for myself.

How does storing excessively large amounts of data work? I mean physically, no matter how big the drive is, there will always be more data than there is space on a single disk. And I'm guessing this is more of a fundamental question, because I can see how it works with caching, since it is essentially the same problem, except it's RAM that you run out of. When a server runs out of RAM you start a second server and then query both servers to find the key you need, it's OK because RAM is fast, but how does that work with relational databases? You can't split a table between two drives, or can you? Even if you could, this still wouldn't work because querying multiple databases would be slow? Apparently there is some kind of a solution to this problem, since Google, Facebook and others are functioning and they surely possess insane amounts of data.

  • I don't have the time for a full answer, but the keyword to search for is sharding.
    – Telastyn
    Sep 19, 2015 at 20:54
  • And for SQL, you want to look into "filegroups".
    – user22815
    Sep 19, 2015 at 20:55
  • 3
    Your question is very broad, there are full books written about it. Why not start with studying some examples like Google File System and Big Table?
    – Doc Brown
    Sep 20, 2015 at 9:26

3 Answers 3


The most generalized term for this is "distributed index", and it is indeed a quite complex topic where a lot of active research is done.

The first step is a dedicated index server, i.e. one that contains no data itself but for every key it knows which server has the data for that key and forwards the request there.

When even the index becomes too big for one server, you can split up the index, most naively by making each index server responsible for an equal range of the key space, e.g. with string keys one is responsible for A-M and the other for N-Z. But data could be unevenly distributed, and the distribution can change, so you need a way for the ranges to be dynamic.

The next step is then a hierarchy of index servers, where the top one only determines which other server is responsible for the range in which the requested key is, possibly with multiple such "hops".

The really tricky thing about this is how to coordinate changes in the index (additions, deletions) between the servers, especially when you replicate parts of the index on multiple servers to handle a large amount of requests. You run into all sorts of very tricky concurrency problems - deadlocks, lock contention, distributed locks being unacceptably slow, dealing with network delays and outages... As I wrote, active research topic!


File systems can span multiple physical disks. You may be familiar with the concept of “partitions”, where one physical medium is split into multiple logical volumes, e.g. a hard disk in a typical Windows installation might have a C: and D: partition. However, this doesn't have to be a 1:n mapping and some file systems can handle n:m mappings between drives and volumes/file systems.

The main motivation for an abstraction level between logical volumes and physical disks is not the potential larger storage capacity, but more flexibility and better fault tolerance, especially in a server setting: when a hard drive fails, I want the server to continue running without interruption or downtime, even when I replace the failed disk. This fault tolerance implies that the same data is replicated on multiple hard disks (e.g. in a RAID-6 setup); the ability to hot-swap a drive requires the applications to be unaware of physical disks (i.e. only use the file system, not access the device directly).

This functionality can be provided by hardware disk controllers that present themselves to the OS as a single disk, but actually contain multiple drives. However, there are software approaches as well. On Linux, you can use the Logical Volume Manager to implement multi-drive partitions, move partitions between drives, or to hot-swap drives. The ZFS file system was created to support massively big amounts of data, and includes sophisticated management of large pools of physical drives. A ZFS file system is spread over all disks in its pool, and can support hot-swapping if in a suitable RAID configuration.

For example, I have configured a server with a ~1TB ZFS pool out of 10 × 150GB disks. The RAID level chosen can withstand two disk failures, and one of the disks is a hot spare that will be used to restore the pool until the faulted disks can be swapped out. Obviously, this could be scaled up with larger disks. E.g. with 20 2TB disks, I would configure them into a 30TB pool.

The RAID level chosen does have performance implications. Since a file is typically spread over multiple disks, reading it can use the combined bandwidth of all drives. However, write performance is reduced when the same file is written to multiple drives for fault tolerance.

With techniques such as RAID or Storage Area Networks, there is a lot you can do to expand the storage capacity of a system. However, at some point data can get too big to be managed on a single system, e.g. if you have too much throughput or need system-level rather than disk-level fault tolerance. In such a scenario, a different, distributed software architecture is needed to scale up further, though hard issues around data consistency arise once you have more than one system that are responsible for the same data.


The largest storage systems use specialized parallel file systems like Lustre, GPFS, BeeGFS, etc. The worlds largest supercomputers all rely on one of these, typical installations reach well into the petabyte range, i. e. they contain literally thousands of hard drives (1 petabyte = 1000 terabytes). These drives are connected to a number of dedicated front end servers (often called I/O-nodes), which provide a unified view of the file system to the rest of the supercomputers.

The use of these filesystems is just as with any other filesystem that's mounted via NFS, the semantics are POSIX-like (POSIX-"like" because many of these parallel file systems deviate subtly from strict POSIX semantics for performance reasons). You do not see, which file is placed on which disk, not even which server manages its data. This is abstracted away by the file system. Files may even be striped across several servers to achieve a higher throughput.

Of course, file systems like these have higher latencies than a local disk, so they perform best when data accesses are large and sequential. Thus, putting a database on them is generally not the best idea, however, it can be done. As I said, the use of a parallel file system is not much different from the use of any other file system mounted via NFS.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.