-2

I have a directory which has approx. 100k JSON files.

I want to retrieve a list of file names, where the file satisfies a set of conditions, with a filter condition in same format as we can pass the filter object in a MongoDB find query.

Does the native file system itself have some sort of logic to maintain indexing as MongoDB does for its documents, or do there exist some tools to do the same?

The files can be added, updated, removed at any point of time, so indexing should be maintained throughout.

I am working on some NodeJS application so would prefer some npm tool that is cross platform compatible.

  • If you were targeting a single O/S there might be a local search service you could (ab)use for this (Windows Search, MacOS Spotlight, ...). However, those search services are designed to index all files in an environment rather than 100k files in a single directory. They are also doing generic text indexing and unlikely to support intelligent queries based on JSON structures. Furthermore, keeping that many files in a single directory will often lead to performance issues depending on the filesystem. I think what you are really looking for is a cross-platform database solution ;-). – Stennie Jul 26 '17 at 12:36
  • 1
    The filesystem is out of the range of software and into the area known as operating systems. To know if this sort of thing is optimized, you should refer to your operating system, though in my experience, operating systems don't like many small files. I have to agree with @Stennie on this one. – Neil Jul 26 '17 at 13:39
3

You haven't described the "set of conditions" in any detail or with specific examples, but unless they are very simple conditions (e.g. filename.startswith('r')), mainstream filesystems lack the indexing and query features you seem to be describing. That is what databases are good at, and what they are for. Load the data and/or metadata you want to query into a database, do the queries, and process the resulting files.

If you have 100K JSON-serialized documents that you need to do (indexed) queries over, load them into a database and use the database indexing and query features. This is exactly what document-structured databases such as MongoDB were invented for.

Side note: There have been attempts to add database-like capabilities onto filesystems. The BeOS file system is a notable example. But they have not been present in mainstream-successful operating systems. One could argue this would be a good feature to have, but it imposes considerable complexity: basically a full database implementation beneath the filesystem. Bit of a chicken-egg problem there; what storage capabilities or abstractions does the database-like filesystem rely upon? Could be raw disk blocks, but many databases now prefer a file-like infrastructure. That implies a lower-level, non-database-like filesystem. If you're doing multiple filesystem stages, why not rely on more classical abstraction layers: raw disk, logical disk, filesystem, database? To support the notion of filesystem-that's-also-database-queryable, you have to assume that managing data as files is going to be a major use case. And you have to assume that handling general queries in the file system is going to be valuable enough to compensate the extra complexity. Historically and in the evolution of filesystems and large dataset handling, those assumptions have not borne out especially well.

Today's highest-scale filesystems indeed use indexing techniques such as B-trees, but indexing on a simple set of attributes (e.g. file names) in support of reasonable performance at scale rather than generalized querying. Typical practice would be managing queries inside a database that is layered atop a less-capable filesystem. You might store the resulting datasets/payloads in the database (i.e. decomposed into native database tables or documents), in the database as blobs (e.g. blobs in relational DBMS, GridFS in MongoDB), or externally (e.g. as files).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.