Context: I'm developing software that reproduce the functionality of Windows Home Server Drive Extender, where users use multiple hard disks to prevent data loss.
The software exists since Dec-2009, but it has issues with renames (detailed below).

I have a directory, dirA, that users can work in: they can create, modify, rename and delete files & sub-directores in dirA.
I want to keep another directory, dirB, in sync with dirA.
The goal is to have a backup of all the files in dirA, in case the drive it's on fails (dirB would be on another drive).

What I'd like, is a discussion on finding a working algorithm that would achieve the above, with the limitations listed below.

1. Something asynchronous - I don't want to stop file operations in dirA while I work in dirB.
2. I can't assume that I can just blindly rsync dirA to dirB on regular interval - dirA could contain millions of files & directories, and terrabytes of data. Completely walking the dirA tree could take hours.

Those two requirements makes this really difficult.
Having it asynchronous means that when I start working on a specific file from dirA, it might have moved a lot since it appeared.
And the second limitation means that I really need to watch dirA, and work on atomic file operations that I notice.

Current (broken) implementation:
1. Log all file & directory operations in dirA.
2. Using a separate process, read that log, and 'repeat' all the logged operations in dirB.

Why is it broken:

echo 1 > dirA/file1
# Allow the 'log reader' process to create dirB/file1:
    log = "write dirA/file1"; action = cp dirA/file1 dirB/file1; result = OK
echo 1 > dirA/file2
mv dirA/file1 dirA/file3
mv dirA/file2 dirA/file1
rm dirA/file3
# End result: file1 contains '1'
# 'log reader' process starts working on the 4 above file operations:
    log = "write file2"; action = cp dirA/file2 dirB/file2; result = failed: there is no dirA/file2
    log = "rename file1 file3"; action = mv dirB/file1 dirB/file3; result = OK
    log = "rename file2 file1"; action = mv dirB/file2 dirB/file1; result = failed: there is no dirB/file2
    log = "delete file3"; action = rm dirB/file3; result = OK
# End result in dirB: no more files!

Another broken example:

echo 1 > dirA/dir1/file1
mv dirA/dir1 dirA/dir2
# 'log reader' process starts working on the 2 above file operations:
    log = "write file1"; action = cp dirA/dir1/file1 dirB/dir1/file1; result = failed: there is no dirA/dir1/file1
    log = "rename dir1 dir2"; action = mv dirB/dir1 dirB/dir2; result = failed: there is no dirA/dir1
# End result if dirB: nothing!
  • 1
    I'm not sure that this is a programmers question. Somewhere between SO and SU really. – Jon Hopkins Jan 15 '11 at 12:45
  • Are Dir A and B on the same computer? If so, try Symbolic linking Dir A to B – Jason Jan 15 '11 at 12:53
  • What I want is copies of all files in dirA. If dirA disappears, because the drive it is on fails, then I'd still have dirB, which is on another drive. – Guillaume Boudreau Jan 15 '11 at 13:12
  • @Guillaume did you consider RAID solutions? There exist hardware RAIDs and also software RAID applications which do what you need. They actually do exactly what I described in my answer above. – Eugene Mayevski 'Callback Jan 15 '11 at 13:15
  • Yes, I have, and RAID is not an option. – Guillaume Boudreau Jan 15 '11 at 13:17

The right approach is to

a) synchronize contents of dirA and dirB initially, b) capture all file events on dirA and replay them on dirB.

This is easily done programmatically by using file system filter driver that will capture all operations on dirA and pass them to dirB system for replaying.

It is important, though, that the complete journal is written on dirA if there's no connection to dirB system, AND that all events for dirA are captured. Losing any information about dirA changes would require complete re-synchronization of dirB with dirA.

  • This is the approach I implemented already, as detailed in the "Current (broken) implementation" paragraph. And the two examples that follow explain why this doesn't work. Those examples are only a very small sample of all the problems that arise when trying to implement this using the above approach. There are countless other cases where replaying the events from dirA is either pretty difficult, or just impossible. – Guillaume Boudreau Jan 15 '11 at 13:20
  • @guillaume no, you have not. You keep logs which are incomplete. If you have initial state synchronized on dirA and dirB, then all changes, if they are saved properly, will be replayed on dirB as well. – Eugene Mayevski 'Callback Jan 15 '11 at 13:28
  • Both directories are empty when I start, so my logs are complete. The problem is that when directories or files are renamed 'too fast' (i.e. before the first rename can be replayed on dirB) then I loose track of where the files are in dirA, and thus can't know what to copy in dirB. – Guillaume Boudreau Jan 15 '11 at 13:32
  • @Guillaume you can't "lose track" -- no matter how fast the operations are performed, if all of them are captured, then you don't lose anything. If you do, this means that your capturing mechanism is broken. – Eugene Mayevski 'Callback Jan 15 '11 at 13:34
  • @Guillaume Now I see your mistake. It is " log = "write dirA/file1"; action = cp dirA/file1 dirB/file1; result = O" . Your log must contain not "cp ... ", but "write block of data to dirB/file1". To capture this kind of changes you need a file system filter driver. – Eugene Mayevski 'Callback Jan 15 '11 at 13:36

rsync does exist on Windows [1], [2]; try it. It knows how to synchronize directories, directly or over network.

  • As I already said, rsync isn't an option, since dirA can be several terabytes, and contain millions of files & directories, and I want both dirB to be up to date 'all the time'. Just running one rsync would take hours, so I'd need to be running rsync 24/7 to achieve the fastest possible sync, and even then, it would take several hours for one small change to make it into dirB. – Guillaume Boudreau Jan 15 '11 at 18:40
  • With these requirements, you need a way to trigger replication once the FS registers a change. Scanning the whole gigantic source is not feasible. NTFS has something for this, but I don't know anything about this beside the fact. On Linux, GlusterFS seems to do the right thing. – 9000 Jan 15 '11 at 18:46

To resolve the issues listed above (and countless others), I created an algorithm that checks for the 'future' name of a file. So if I want to work on file1, and it doesn't exist anymore, I then look in the subsequent operations to find where it is now, and use that as the source to create the file1 copy in dirB. Using this, I resolved all the test cases I had that failed previously.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.