I've been investigating various distributed file systems, like Gluster, Ceph, Moose and Lizard.

I'm also familiar with various key/value store type systems, some of which do not perform any system calls related to durability (such as the fsync() function) because replication reduces the likelihood of data loss. E.g., if you replicate a key/value record across 3 servers, and one server goes down, your data is not lost.

My question is, do replicated distributed file systems that provide a certain level of redundancy (such as with a 4x replication factor) minimise the need to request durability from the operating system, and if so, by how much? (I.e., is there a time delay in replicating files, so that, for example, there is only a 0.2 second window for time loss given no network partitions?)

Note that my application uses the fsync() system call, which reduces writes to 30 records per second, whereas without this call, the writes are around 200,000 per second (which I'm guessing is because of the heavy use of the operating system cache). I'd like to keep the same level of reliability while improving performance via replication/redundancy. (I.e., 30 records per second means - assuming the hardware and file system are behaving correctly - a window of ~0.03 seconds in which failure can occur, and I want to compare that to the alternative, higher performing approach using redundancy.)


Yes and No. Yes, obviously if you have two or more nodes your app is more durable, but No, normally you would still expect the same level of durability from the OS. You are protecting yourself from things like hardware failure and application crashes rather than OS failures.

Given that, I expect that the key/value systems you reference do sync to disk at some point, they just do it more efficiently that your own application. As you note, writing to disk is slow, so you want to be clever about how and when you do it to ensure that it is optimised.

  • Thanks @Ewan. I assume OS = Operating System (like device drivers, filesystem modules, etc), in which case traditional databases face them same class of errors as direct file access (e.g., they are both “above” the OS level). – magnus Oct 10 '18 at 23:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.