14

I have noticed where I work people are keen on storing information in file names, and parsing the file names.

To me this doesn't seem to be especially good practice. I already see the occasional issues with scripts globbing for a file, and getting the wrong one because another file matches first.We are also discussing how to get around problems with separators for the fields.

Is it considered bad practice or not?

What are other accepted solutions for retrieving files from a file system based on some type of metadata?

2

7 Answers 7

15

Yes I think it's bad practice. It is subject to all sorts of problems - for example length limits, encoding issues and conflicts due to duplicate data.

Better is to use a "master file" (sometimes called manifest or index) that contains metadata and paths to the files. Or something similar in a database, register or whatnot. Or to put the meta data inside the actual files, at the top level of some datastructure contained in the file in for example JSON or XML.

This is somewhat analogous to the concept of putting information, or namespacing keys in key-value stores. I think this is ok as long as you use it only to namespace and do quick lookups - the key components are not there to provide parsable information. If you need that information, duplicate it into the value (file in the above case).

5
  • 3
    You are raising gut points. But there are situations when it nevertheless makes sense to put the information in the filename. Think of mail attachments which have to be routed or processed in a rule-based fashion. If many parallel processes have to change the master file, it might become a bottleneck. Oct 2, 2013 at 13:33
  • As a database developer, I naturally think of using a database instead of a manifest file (one of the reasons I ask here for alternative methods). That would solve the concurrent access problem, but is a more complex solution. Oct 2, 2013 at 14:36
  • 1
    @wobbily_col, depending on the system that you use, there may be support for extended file attributes available.
    – Hellion
    Oct 2, 2013 at 16:57
  • @AxelKemper There's only so much info you can fit in a name. There's more metadata than the name and author. Jan 22, 2015 at 8:57
  • Not to mention filenames can be altered by someone outside of your system, breaking any expected formats. Even when you have appropriate file permissions enforced, it ends up being a brittle solution. Jun 17, 2019 at 15:26
7

First, metadata is a blurry concept.

That said, many cases of metadata in files already exist:

  • version numbers of libraries
  • date and time of images, or at least sequence index
  • file type, which triggers what application should open the file
  • name of your home directory, which must be your session username

Nevertheless, that short list is not an argument in favor of the practice.

Alternatives are:

  • handle metadata in the FS level, like Apple old HFS for instance
  • put metadata in the file itself, like Exif for images or ID3 for sounds
  • put metadata in another file or in a database, like most media managers.
1
  • 7
    Everything is a blurry concept. Even "blurry", "concept" and "everything" are blurry concepts. Jan 23, 2015 at 1:01
5

It sounds like you need a database.

There are lots of security issues with putting user data in file names. Let's say that you have a file for each user ("username.txt"). What happens what someone registers the username "../../../../etc/passwd" depends on how you are filtering user input.

Database frameworks will sometimes assist you with sanitizing user input.

3
  • Actually, many operating systems store user names in directory names, which are called home directory.
    – mouviciel
    Oct 3, 2013 at 6:57
  • That's because somebodies software has to be at the bottom of the stack. It doesn't mean that everyone has to work on that level. I'm not going to argue the merit of databases, because programmers have been using them for more than 50 years. Oct 19, 2013 at 11:04
  • 1
    @mouviciel I'm not aware of any operating system that parses the user name out of the user's home directory name. Windows and Unix-like systems both store the name of the directory in some kind of database and load it into the environment when the user logs in. Under both systems, you can end up with the home directory name being different to the user name (e.g. renaming users, or if you have two windows instals on the same system partition).
    – Jules
    Jan 22, 2015 at 10:02
3

First, let us agree what a file is. A file is a packaged data with a name that can be transmitted, received, created and deleted with (very close to) atomic operations.

Many file systems (Mac OS, and more recent Linux file systems) implement "forks", often used to store resources and metadata. This approach to storing metadata was problematic in that traditional network transfer methods, backup and restore methods and file copying methods were inconsistent, especially when the source and destination file systems understood file forks differently.

The file name is used to hold metadata because a) it is always there, b) metadata has always been present in the file name (at least in the use of file extensions), and c) the file name undergoes very little translation when moving between systems (case distinctions, character set limitations, character limitations aside).

So, the file name is visible, portable, and manageable. This is not a bad thing for storing some metadata.

Probably the best solution to address general file metadata is to use a content repository, where the content repository can be configured with the metadata schema to be used for the files. In many cases this is overkill, but, IMHO, is the way to go for serious metadata management.

2

No... well.. not necessarily.

So long as you have a strict convention and common parsing and validation means (scripts, libraries etc) readily available you are good to go.

Take for example packaging and dependency management systems (Maven, NuGet and the likes). Though many will use specific files for metadata to store the more advanced information, basic information is often part of the file name itself. Relying on strict conventions the file name can contain the most pertinent information about the package : it's vendor, it's name, it's version, it's type. Sometimes that is all you need... 4 or 5 short pieces of information.

If the metadata is simple then a file naming convention makes perfect sense requiring nothing to put in place. It can be strengthened with very simple tools and scripts, no database needed, no specialised infrastructure just a few scripts and a naming convention.

If nothing out there quite does what you need and your needs are simple i'd start with this.

your requirements outgrow this convention ? extend it with a proper metadata file. You later need better search for this ? There are already good solutions out there for searching files that get get you to where you need.

It's not that I dislike databases, quite the contrary they are really powerful and useful but they require some amount of overhead to get going. They need to be installed, backed up, maintained, you will need staff that, if not completely dedicated, will need to dedicate part of their time to this infrastructure. They are also more complex and cryptic to the laymen, loose the dev that set you up and your system will be stuck in time until you find a replacement.

Never underestimate the power of low tech with the proper oversight it can get you a long way.

And by the time you outgrow your low tech solution you will have gathered all the experience and requirements to implement the perfect system for your needs.

3
  • Never underestimate the power of inertia. Changing a low tech solution into something more robust takes a lot more effort than just not doing it that way to begin with. Jun 17, 2019 at 15:40
  • 1
    @BerinLoritsch same argument applies to all solution, low-tech or hitech... one could argue that hitech requiring more systems inter dependency actually makes this situation worst, not easier. That said, there is threshold where a simple low-tech solution becomes more convoluted than it's full blown high-tech counterpart.
    – Newtopian
    Jun 17, 2019 at 17:06
  • 2
    Yep, and I'm untying a couple such examples on a project now. Bottom line is that there does need to be a tighter interface than the filesystem more times than not. Unfortunately, most low-tech systems I inherit don't have appropriate thought or design applied to them. The number of exceptions I can count on one hand. Jun 17, 2019 at 17:44
1

My take on this is that you may have seen some code somewhere that does sloppy or brittle things with file names, but that does not mean that "storing metadata in filenames" is bad in general.

File names are metadata- they are data about the data in the file, independent of the file data itself. In fact, filenames are so old that they are probably the canonical example of metadata.

If you consider that file extensions are just the end part of the filename, then the filename-as-metadata concept becomes even more unavoidable.

1

I want to rank metadata. Yes, a filename should contain metadata, because it actually is metadata, or do you refer to files by their inode number?

  1. The filename: This is the first info you ever get about the file. Think of it like a title of a book. Thus, in addition to uniquely identify the file, it should be short summary of its content. For example lenna.bmp indicates that the file is the face of Lena Forsén, encoded as a Windows bitmap file. A better name would be lena_with_hat.bmp, because that is what the picture is, but since lenna is an established name, we continue with that name.

  2. The file headers: These should contain at least contain all information required to process the data, in this case, a picture. The header would tell you that the picture is 512x512 pixels, and that each pixel occupies 3 bytes. Here, some information may be implicit from the fact that it is a Windows bitmap file. For example, you need to know which one is red, green and blue, or if pixels are stored as separate planes. Since it is a Windows bitmap file, it is known that data is stored BGR, BGR, BGR .... It should be noted that different applications may require different metadata. For example, to replicate the camera that was used to take the picture, apperture, film size, and focal length would be needed.

  3. The index file: Here, you store everything else, that would not fit into the other two categories. It could be additional notes about the picture (location, copyright info) although these could go into (2) as well. It could also contain a list of related pictures.

Notice that sometimes, the filename may contain information found in the file headers. This is not ideal but could be the case when there are multiple versions of the same file. In this particular example, there could be files like

`lenna_512x512.bmp`, `lenna_256x256.bmp`, and `lenna_128x128.bmp`

While these names contains information that are already in the file header, the filenames have to be unique, and they should be descriptive. What makes these files unique? They have different resolution. Thus, it is not a bad choice to include the pixel dimensions in the filename. As an alternative schema, you could use the suffixes _hires _midres, and _lowres, though it would be tricky to come up with names if you have many versions. The ultimate solution here is to store the different versions in the same file, but for some reason, you do have to stick to a file format that does not support that, so you have to do it like this.

Not the answer you're looking for? Browse other questions tagged or ask your own question.