I am planning to write a program that will produce digest logs for file changes in a particular directory. The idea is that whenever a file is deleted in a particular directory, it will show up in the digest log.
In the case that a file is heavily modified and not outright deleted, I also want it show up in the digest log. For example, suppose that 30% of a file has been modified when compared to 24 hours ago. This file should also be listed in the digest log.
Now the question is how does the program detect that >30% of a file has been modified between two particular instants 24 hours apart. I can simply keep all copies of all data that is modified but that is extremely disk-space costly. I would like to optimize the algorithm so that it takes as little disk-space as possible.
One possible solution is that for every file, I mentally split it up into 10 parts. Then, I compute the CRC32 checksum for each part and store it somewhere. Then on the next day, I perform the same CRC32 checksums and compare with the checksums from yesterday. If 3/10 parts of the file do not match checksums, then I know that 30% of the file has been modified. Is there a better solution?