Does a replicated distributed file system minimise the need for durability?

Question

I've been investigating various distributed file systems, like Gluster, Ceph, Moose and Lizard.

I'm also familiar with various key/value store type systems, some of which do not perform any system calls related to durability (such as the fsync() function) because replication reduces the likelihood of data loss. E.g., if you replicate a key/value record across 3 servers, and one server goes down, your data is not lost.

My question is, do replicated distributed file systems that provide a certain level of redundancy (such as with a 4x replication factor) minimise the need to request durability from the operating system, and if so, by how much? (I.e., is there a time delay in replicating files, so that, for example, there is only a 0.2 second window for time loss given no network partitions?)

Note that my application uses the fsync() system call, which reduces writes to 30 records per second, whereas without this call, the writes are around 200,000 per second (which I'm guessing is because of the heavy use of the operating system cache). I'd like to keep the same level of reliability while improving performance via replication/redundancy. (I.e., 30 records per second means - assuming the hardware and file system are behaving correctly - a window of ~0.03 seconds in which failure can occur, and I want to compare that to the alternative, higher performing approach using redundancy.)

Answer 1

Key value stores that don't persist to disk are generally billed as caches, implying there is another long-term durable storage your application manages somewhere else. If your data is important to keep long-term, a cache won't help you, even if distributed. You want to write it to disk, and you want to protect against corruption and application mistakes by taking regular backups. Distributed file systems usually try to work in large batch sizes for efficiency. Their "time between automatic flushes" is much higher than working with a local disk.

Your modern higher-throughput durable databases generally work by having an append-only commit log. See cassandra for example. Those commit logs are flushed to disk either before acknowledging a write, or on a periodic basis (like every 10 seconds), depending on how paranoid you are.

The commit log protects against failure, but the queries are actually handled by an in-memory table data structure, which occasionally gets persisted to disk in a more random-access form than the commit log, whenever the commit log gets too full. In other words, you're mostly working out of RAM, while different forms of durability are handled concurrently by other parts of the database.

Keeping these all synchronized correctly is a very difficult problem that requires a lot of tuning to get right. I would seriously recommend against rolling your own storage.

Answer 2

Yes and No. Yes, obviously if you have two or more nodes your app is more durable, but No, normally you would still expect the same level of durability from the OS. You are protecting yourself from things like hardware failure and application crashes rather than OS failures.

Given that, I expect that the key/value systems you reference do sync to disk at some point, they just do it more efficiently that your own application. As you note, writing to disk is slow, so you want to be clever about how and when you do it to ensure that it is optimised.

Answer 3

Yes, and we can understand why with basic probability. If I have system where each node has a 10% chance of failure over a given time frame, if I set up 2 nodes, the chance that both of them fail during that time frame is 1% (0.1 * 0.1). With three, the chance that all fail during that period drops to 0.1% (0.1 * 0.1 * 0.1) and so on an so forth.

What this means is that you can create more robust systems using less reliable components by adding more nodes. The other side of this is that the costs of increasing the reliability of a single piece of hardware is exponential. That is, each increase in reliability is smaller and more expensive than the last increase. For example, it might cost 10 times as much to get from 90% to 95% reliability than it took to get from 80% to 90%. So instead of spending millions on one machine, you can get a bunch of commodity hardware that's moderately reliable and if some of it fails, you just replace it.

It seems like you are looking at a similar situation. By adding redundancy, you don't need each write to be as resilient. There are new challenges that come up with this related to how you know which nodes have saved successfully and have the correct values etc. but fundamentally, yes, this is a viable strategy that underpins a lot of contemporary system architecture.

A lot of these concepts (and more) are covered in the doctoral thesis of Joe Armstrong who was a major force behind Erlang.