What is a good way of setting up a "shared index" of file metadata, when there can be no shared process such as a database server?
I'll explain the scenario: A server contains M (say 10000) large files (say e.g. image files). each file is in a unique subdirectory with guid name. The root directory is shared as windows share and will be consumed by several (N) clients running a desktop application. The desktop application is being implemented so can be designed in any way. The data stores are on either local folders or shared network folders, and can be set up by each client (without central administration) - so there is no central machine that can run a database with the metadata index. Each client may have use different folders. E.g
client1 uses \\server1\share1\[1000 guid directories]
client2 uses \\server1\share1\[1000 guid directories]
\\server1\share2\[1000 guid directories]
In the above scenario, the clients could theoretically share an index filed stored in the root \\server1\share1.
The number of users will be relatively low (typically < 20) so there is every chance that having shared indices is overkill, and instead each client should just scrape to a local index file and work from that.
What I want to emulate is that the server had a scraping/caching process running, and a DB frontend that would let clients query for the files by their metadata. I can't have a process on the server, however.
In the naive approach, all N clients will scrape the M-files and store the metadata in memory. Depending on whether they also persist it on disk, the number of times a file is scraped will be NxM or NxMxK if each user does K sessions. This is obviously wasteful and will incur a ton of network traffic for no good reason. The data is appended to (new files added) quite rarely, but read often. Modifications to existing files are also rare. The data volume could be 0-10G while the total metadata volume is 0-10M.
This is pretty similar to the way windows (up to Win7 I believe) drops a Thumbs.db in the folder where thumbnails have been created, even for shared folders.
What is a good approach for keeping a "shared index" of the metadata? Example: after a client has scraped all the files he dumps an index file in the root. A client that writes to the data could also immediately update the index file. The next client coming along could simply use the pre-scraped metadata. I considered using a proper file DB (e.g. SQLite) but they aren't a good fit for a shared file consumed by multiple processes even if most only read.
Another method would be to have local SQLite files at each client, with clients simply dumping their copy in the server dir for others to download and use as their local copy. Ideas?
