Efficient way to check for changes to the contents of folders

Question

I am creating an application that maintains a database of files of a certain type in a given folder (and all subfolders) Initially the program will recurse the folders and add any file it finds of that type to the database.

I want the application to have the ability to re-scan the folder and add any files that were not there the last time the folders were scanned.

It can't use the date created property of the file because there is a high chance of a file being added to the folders that isn't a new file.

I am wondering what the most efficient way of doing this is, and if there is a way that doesn't involve checking each file is in the database already (which, if there are 5000 files would mean 5000 queries of a list 5000 items in size, or 25 million 'checks' for the sql engine to perform)

I suppose a more specific question to acheive the same goal would be - is there a property of a file (in Microsoft Windows) that will reliably tell you when that file arrived in that folder.

Edit: The app would not be running all the time, so monitoring the folder for change events is not an option. A typical scenario might be. Run the app. get new files. close the app. A week later (after normal computer usage and files being added to the folder) run the app again, look for changes since the app was last used.

Answer 1

Looking on your profile on StackOverflow, seems you're (or were) a C++ Builder developer, so I suggest to use the Shell API to get notified of modifications on given folder. If you go down this path, creating an service-like (or even a real windows service) program would be my path.

Edit: another way is getting a "hash"/"signature" calculated from the name of the files and some other attribute. So you calculate it each time you scan and compare with one you saved on DB.