I've been investigating various distributed file systems, like Gluster, Ceph, Moose and Lizard.
I'm also familiar with various key/value store type systems, some of which do not perform any system calls related to durability (such as the
fsync() function) because replication reduces the likelihood of data loss. E.g., if you replicate a key/value record across 3 servers, and one server goes down, your data is not lost.
My question is, do replicated distributed file systems that provide a certain level of redundancy (such as with a 4x replication factor) minimise the need to request durability from the operating system, and if so, by how much? (I.e., is there a time delay in replicating files, so that, for example, there is only a 0.2 second window for time loss given no network partitions?)
Note that my application uses the
fsync() system call, which reduces writes to 30 records per second, whereas without this call, the writes are around 200,000 per second (which I'm guessing is because of the heavy use of the operating system cache). I'd like to keep the same level of reliability while improving performance via replication/redundancy. (I.e., 30 records per second means - assuming the hardware and file system are behaving correctly - a window of ~0.03 seconds in which failure can occur, and I want to compare that to the alternative, higher performing approach using redundancy.)