1

I've got an extremely oniony(deep) folder structure which contains Appx 1,000,000 text-based files on a network share. Using windows search is extremely slow and unreliable. I've created some text files which lists of all the files and have searched that text file to speed up searching, but it's time to mature this and make it a little more automated and accessible to more users.

The index/cache will contain the file path and some additional meta data which I extract from the files upon indexing. Appx 10 fields. This is all searchable.

I'm leaning towards a sql or sqlite database table but not sure what else I should be looking that.

What type of database or data structure is best suited for this purpose? What other topics/keywords can I research regarding this?

For front end, I'll just wrap up a console application or simple winform to search the index.

Improve this question
3
  • Consider how you are going to search for matches. Are you considering regex? Try reading some Russ Cox.
    – Kain0_0
    Commented Dec 3, 2020 at 2:35
  • 2
    If I would been in your shoes, I would try not to reinvent the wheel, but use something like Lucene.Net.
    – Doc Brown
    Commented Dec 3, 2020 at 6:33
  • ElasticSearch is also very very efficient when it comes to indexing and searching (it's build on top of Lucene).
    – BionicCode
    Commented Dec 3, 2020 at 16:33

1 Answer 1

Reset to default
1

I suppose the choice would be based on the type of searches your users would most likely do.

If the vast majority of searches are simply related to the data in the terminal nodes of your file tree, then you could do this in SQL.

However, if your searches are likely to include looking for children of a specific node somewhere half-way up the tree, then it could be an extremely costly affair for an SQL database and a NoSQL store would be more appropriate.

Also, if your metadata differs from file to file and can't be standardised as SQL fields, then NoSQL would also make more sense.

Here is how I would do it in SQL: have a table that has a parent field and for each file/directory has a record that keeps track of their parent node. Then you could just search it flatly, and recursively build a path to the specified node as needed.

I would also store the path in a separate field for speed of access. Though it will make reindexing slower, it will dramatically speed up regular lookups.

That being said, I think NoSQL is a better approach in this case and will ultimately work faster.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.