13

I have noticed where I work people are keen on storing information in file names, and parsing the file names.

To me this doesn't seem to be especially good practice. I already see the occasional issues with scripts globbing for a file, and getting the wrong one because another file matches first.We are also discussing how to get around problems with separators for the fields.

Is it considered bad practice or not?

What are other accepted solutions for retrieving files from a file system based on some type of metadata?

  • It depends a lot on what exactly is being stored on the filename. Can you give us some examples? – T. Sar Jun 26 at 12:39
14

Yes I think it's bad practice. It is subject to all sorts of problems - for example length limits, encoding issues and conflicts due to duplicate data.

Better is to use a "master file" (sometimes called manifest or index) that contains metadata and paths to the files. Or something similar in a database, register or whatnot. Or to put the meta data inside the actual files, at the top level of some datastructure contained in the file in for example JSON or XML.

This is somewhat analogous to the concept of putting information, or namespacing keys in key-value stores. I think this is ok as long as you use it only to namespace and do quick lookups - the key components are not there to provide parsable information. If you need that information, duplicate it into the value (file in the above case).

  • 3
    You are raising gut points. But there are situations when it nevertheless makes sense to put the information in the filename. Think of mail attachments which have to be routed or processed in a rule-based fashion. If many parallel processes have to change the master file, it might become a bottleneck. – Axel Kemper Oct 2 '13 at 13:33
  • As a database developer, I naturally think of using a database instead of a manifest file (one of the reasons I ask here for alternative methods). That would solve the concurrent access problem, but is a more complex solution. – wobbily_col Oct 2 '13 at 14:36
  • 1
    @wobbily_col, depending on the system that you use, there may be support for extended file attributes available. – Hellion Oct 2 '13 at 16:57
  • @AxelKemper There's only so much info you can fit in a name. There's more metadata than the name and author. – Tulains Córdova Jan 22 '15 at 8:57
  • Not to mention filenames can be altered by someone outside of your system, breaking any expected formats. Even when you have appropriate file permissions enforced, it ends up being a brittle solution. – Berin Loritsch Jun 17 at 15:26
4

First, metadata is a blurry concept.

That said, many cases of metadata in files already exist:

  • version numbers of libraries
  • date and time of images, or at least sequence index
  • file type, which triggers what application should open the file
  • name of your home directory, which must be your session username

Nevertheless, that short list is not an argument in favor of the practice.

Alternatives are:

  • handle metadata in the FS level, like Apple old HFS for instance
  • put metadata in the file itself, like Exif for images or ID3 for sounds
  • put metadata in another file or in a database, like most media managers.
  • 5
    Everything is a blurry concept. Even "blurry", "concept" and "everything" are blurry concepts. – Tulains Córdova Jan 23 '15 at 1:01
3

It sounds like you need a database.

There are lots of security issues with putting user data in file names. Let's say that you have a file for each user ("username.txt"). What happens what someone registers the username "../../../../etc/passwd" depends on how you are filtering user input.

Database frameworks will sometimes assist you with sanitizing user input.

  • Actually, many operating systems store user names in directory names, which are called home directory. – mouviciel Oct 3 '13 at 6:57
  • That's because somebodies software has to be at the bottom of the stack. It doesn't mean that everyone has to work on that level. I'm not going to argue the merit of databases, because programmers have been using them for more than 50 years. – Eric Wimberley Oct 19 '13 at 11:04
  • 1
    @mouviciel I'm not aware of any operating system that parses the user name out of the user's home directory name. Windows and Unix-like systems both store the name of the directory in some kind of database and load it into the environment when the user logs in. Under both systems, you can end up with the home directory name being different to the user name (e.g. renaming users, or if you have two windows instals on the same system partition). – Jules Jan 22 '15 at 10:02
2

No... well.. not necessarily.

So long as you have a strict convention and common parsing and validation means (scripts, libraries etc) readily available you are good to go.

Take for example packaging and dependency management systems (Maven, NuGet and the likes). Though many will use specific files for metadata to store the more advanced information, basic information is often part of the file name itself. Relying on strict conventions the file name can contain the most pertinent information about the package : it's vendor, it's name, it's version, it's type. Sometimes that is all you need... 4 or 5 short pieces of information.

If the metadata is simple then a file naming convention makes perfect sense requiring nothing to put in place. It can be strengthened with very simple tools and scripts, no database needed, no specialised infrastructure just a few scripts and a naming convention.

If nothing out there quite does what you need and your needs are simple i'd start with this.

your requirements outgrow this convention ? extend it with a proper metadata file. You later need better search for this ? There are already good solutions out there for searching files that get get you to where you need.

It's not that I dislike databases, quite the contrary they are really powerful and useful but they require some amount of overhead to get going. They need to be installed, backed up, maintained, you will need staff that, if not completely dedicated, will need to dedicate part of their time to this infrastructure. They are also more complex and cryptic to the laymen, loose the dev that set you up and your system will be stuck in time until you find a replacement.

Never underestimate the power of low tech with the proper oversight it can get you a long way.

And by the time you outgrow your low tech solution you will have gathered all the experience and requirements to implement the perfect system for your needs.

  • Never underestimate the power of inertia. Changing a low tech solution into something more robust takes a lot more effort than just not doing it that way to begin with. – Berin Loritsch Jun 17 at 15:40
  • 1
    @BerinLoritsch same argument applies to all solution, low-tech or hitech... one could argue that hitech requiring more systems inter dependency actually makes this situation worst, not easier. That said, there is threshold where a simple low-tech solution becomes more convoluted than it's full blown high-tech counterpart. – Newtopian Jun 17 at 17:06
  • 1
    Yep, and I'm untying a couple such examples on a project now. Bottom line is that there does need to be a tighter interface than the filesystem more times than not. Unfortunately, most low-tech systems I inherit don't have appropriate thought or design applied to them. The number of exceptions I can count on one hand. – Berin Loritsch Jun 17 at 17:44
0

My take on this is that you may have seen some code somewhere that does sloppy or brittle things with file names, but that does not mean that "storing metadata in filenames" is bad in general.

File names are metadata- they are data about the data in the file, independent of the file data itself. In fact, filenames are so old that they are probably the canonical example of metadata.

If you consider that file extensions are just the end part of the filename, then the filename-as-metadata concept becomes even more unavoidable.

-1

First, let us agree what a file is. A file is a packaged data with a name that can be transmitted, received, created and deleted with (very close to) atomic operations.

Many file systems (Mac OS, and more recent Linux file systems) implement "forks", often used to store resources and metadata. This approach to storing metadata was problematic in that traditional network transfer methods, backup and restore methods and file copying methods were inconsistent, especially when the source and destination file systems understood file forks differently.

The file name is used to hold metadata because a) it is always there, b) metadata has always been present in the file name (at least in the use of file extensions), and c) the file name undergoes very little translation when moving between systems (case distinctions, character set limitations, character limitations aside).

So, the file name is visible, portable, and manageable. This is not a bad thing for storing some metadata.

Probably the best solution to address general file metadata is to use a content repository, where the content repository can be configured with the metadata schema to be used for the files. In many cases this is overkill, but, IMHO, is the way to go for serious metadata management.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.