What is the field/discipline called that deals with the automated management of very large and heterogeneous collections of files?

Question

I work for an academic research lab whose file storage footprint is growing past the point where it can be properly managed "by hand", i.e. without extensive automation. (The number of files we have now in the low millions, but growing rapidly.)

I figure that this problem must have been around for a few decades by now, and therefore that an entire discipline of software engineering must have grown around it¹.

Unfortunately, I'm sorry to say, I know nothing about this hypothesized discipline, not even what it is called, which makes it hard for me to get started with it.

I imagine that this hypothetical field is a subfield of the general field of "databases" (or "data storage"), but is adapted to the special databases that we call "file systems".

I want to learn about the types of programs that are used to automatically manage collections of millions, maybe billions of files. The "management" I'm referring to here includes:

1. deciding where files should be stored, and for how long;
2. deleting/moving files at the appropriate times;
3. maintaining a "meta-database", containing a rich set of metadata for each file;
4. providing easy-to-use interactive facilities for searching for and retrieving file metadata or the files themselves.

Q: What search keywords should I use to begin getting acquainted with this domain?

---

^{1 We have relational databases, so why reinvent the wheel? In other words, why don't we just distill what we want to keep, stick it into a relational database, and delete the files? The answer to this boils down to the need for flexibility. For one thing, the files we keep vary greatly in how long we want/have to keep them for. Some need to be kept only for roughly a week, others need to be kept for at least 10 years, and there are a few other longevities in between. Also, the files we keep are very heterogeneous in nature, and I suspect converting their contents so that they could be put into an RDB would require a very large number of tables and very complicated schemas. Most importantly, though, the types of files that we want to keep change from month to month, as the pipelines that generate them evolve. This "schema volatility" is a characteristic of cutting-edge research data that, IMO, makes it and RDBs a poor fit.}

Answer 1

Perhaps Digital Asset Management is the term you're looking for.

It involves designing and using a variety of applications for:

• Cataloging
• Indexing
• OCR-ing
• Format conversion
• Digitization
• Version/Revision control
• Previsualization
• Access control
• Keeping and updating metadata
• Digital Rights Management
• Workflow
• Virtualization of extinct operating systems for native use of extinct data formats. ...and so on

Of digital assets, including media files (video, audio, etc.). Content management, digital media management, archival studies (computing applied to).