What is a good way of setting up a "shared index" of file metadata, when there can be no shared process such as a database server?

I'll explain the scenario: A server contains M (say 10000) large files (say e.g. image files). each file is in a unique subdirectory with guid name. The root directory is shared as windows share and will be consumed by several (N) clients running a desktop application. The desktop application is being implemented so can be designed in any way. The data stores are on either local folders or shared network folders, and can be set up by each client (without central administration) - so there is no central machine that can run a database with the metadata index. Each client may have use different folders. E.g

client1 uses  \\server1\share1\[1000 guid directories]                  
client2 uses  \\server1\share1\[1000 guid directories]
              \\server1\share2\[1000 guid directories]

In the above scenario, the clients could theoretically share an index filed stored in the root \\server1\share1.

The number of users will be relatively low (typically < 20) so there is every chance that having shared indices is overkill, and instead each client should just scrape to a local index file and work from that.

What I want to emulate is that the server had a scraping/caching process running, and a DB frontend that would let clients query for the files by their metadata. I can't have a process on the server, however.

In the naive approach, all N clients will scrape the M-files and store the metadata in memory. Depending on whether they also persist it on disk, the number of times a file is scraped will be NxM or NxMxK if each user does K sessions. This is obviously wasteful and will incur a ton of network traffic for no good reason. The data is appended to (new files added) quite rarely, but read often. Modifications to existing files are also rare. The data volume could be 0-10G while the total metadata volume is 0-10M.

This is pretty similar to the way windows (up to Win7 I believe) drops a Thumbs.db in the folder where thumbnails have been created, even for shared folders.

What is a good approach for keeping a "shared index" of the metadata? Example: after a client has scraped all the files he dumps an index file in the root. A client that writes to the data could also immediately update the index file. The next client coming along could simply use the pre-scraped metadata. I considered using a proper file DB (e.g. SQLite) but they aren't a good fit for a shared file consumed by multiple processes even if most only read.

Another method would be to have local SQLite files at each client, with clients simply dumping their copy in the server dir for others to download and use as their local copy. Ideas?

  • can you explain the restriction more fully? can you change the client? – Ewan May 17 '17 at 9:33
  • Tried to clarify a bit - the clients are being implemented so can work any way they want. The restriction on the server bit is that the client simple desktop so don't want to require users to set up processes on their storage servers only for indexing of their data. But perhaps having shared indices is too complicated if the number of users is low – Anders Forsgren May 17 '17 at 10:44
  • are you saying you are not allowed to write a program which runs on the server? you have to only use a standard file share. – Ewan May 17 '17 at 11:01
  • I think you are on the right track, but to see what works best in your environment, and for what you really need, you will probably have to try it out. Depends very much on the network speed, the reliabilty of the network share, the freqency of changes to the data, and if it is feasible to let all other clients wait until one has finished the indexing for the current state. If you can prevent multiple writers to the index file, you will avoid lots of ugly problems. – Doc Brown May 17 '17 at 11:12
  • @Ewan exactly: Customers install the client app, and may or may not use one or more network shares. They can't install indexing db processes on the network machines (it wouldn't be worth the effort, and we don't want to maintain and support a separate product) – Anders Forsgren May 17 '17 at 13:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.