I have a larger directory structure (dir + sub directories) with files. It contains files of certain types. For one particular type (let's say with appendix
.foo) I need to figure out if files have been added, changed or deleted.
The first approach was to iterate over all files and check the timestamp of the files. It works pretty ok for local files, but once they reside on a network file system it gets too slow.
And, in order to detect deleted files I have to create and index anyway. One idea is to create hash values for the relevant files and store them per directory. Unfortunately the normal case is that nothing changes most of the time, yet I still have to recalculate all hash values (slow) just to detect nothing changed.
Remark: I am coding in Qt 5.6/C++ and code has to work on Windows, Mac, and Linux, but the question`s focus is more the concept and not working code.
--- edit ---
Some clarifications (as asked below)
- it is about files for flight simulation. When the user adds or changes files I have to rerun some tasks like parsing those files. The
foofiles are special files I care about, while the 10000s other flight simulator files can be ignored. My goal is to find out if I have to start the expensive parsing process, which is only required if any
- there is indeed something like a monitor watching changed files in Qt
QFileSystemWatcher. But I cannot guarantee to monitor all the time, also it would not be needed. When I start my software I need to find out if any
foofile changed, if so I start parsing, otherwise skip this step.
- There are some 100 up to 10000
foofiles within a directory structure of >50000 files
- The files change because the user installs new features or maybe deletes some. This happens while my software is not running, I already said I cannot monitor all the time. So I need something which works after I have started my software
- There is no client / server and it needs to work on Linux, Mac and Windows (but that was already mentioned in the original post)