I am planning to write a program that will produce digest logs for file changes in a particular directory. The idea is that whenever a file is deleted in a particular directory, it will show up in the digest log.

In the case that a file is heavily modified and not outright deleted, I also want it show up in the digest log. For example, suppose that 30% of a file has been modified when compared to 24 hours ago. This file should also be listed in the digest log.

Now the question is how does the program detect that >30% of a file has been modified between two particular instants 24 hours apart. I can simply keep all copies of all data that is modified but that is extremely disk-space costly. I would like to optimize the algorithm so that it takes as little disk-space as possible.

One possible solution is that for every file, I mentally split it up into 10 parts. Then, I compute the CRC32 checksum for each part and store it somewhere. Then on the next day, I perform the same CRC32 checksums and compare with the checksums from yesterday. If 3/10 parts of the file do not match checksums, then I know that 30% of the file has been modified. Is there a better solution?

  • This question is probably better fit for stack overflow
    – Jessie
    Aug 1 '19 at 18:00
  • Which operating system and file system? Different operating systems and filesystems have different filesystem observer APIs.
    – Alexander
    Aug 1 '19 at 18:07
  • I am on linux and using the ext4 file system Aug 1 '19 at 18:08
  • Have you considered using git? stackoverflow.com/a/2528129/3608792
    – Dan Wilson
    Aug 1 '19 at 18:12
  • I feel that your algorithm will fail big time if the file is simply extended as your before and after CRC blocks will never match and you will indicate that the file has changed 100%.
    – Peter M
    Aug 1 '19 at 19:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.