File system for abstracting S3 object storage

Question

We're looking for a scalable way to implement a file browser feature backed by a robust object storage system for the files. We were looking at an S3 compatible service, such as MinIO for this. Since these files will be accessed by other features as well, we thought of giving these objects immutable unique ids, independent of the folders in which they appear in the file browser.

Most S3 storages don't seem to support file paths but tend to promote using the a path structure as the key. Unfortunately, the keys will not be immutable when files would be moved in the file browser. While we could work around this and not have immutable keys, it would be less ideal.

On the other hand, this wouldn't allow us to have empty directories in the file browser. A workaround for this could be to have a hidden file in every directory, but that feels a bit like a hack.

We were thinking of just generating a unique id per upload and keep a list of paths with the objects ids they point to, in another database. From this, we could build up the directory structure and display the file browser. Renaming files and folders wouldn't interfere with the existing keys, so other features can still access the files, independent of their file path in the file browser.

Would this be a valid approach or are there other ways to implement this? Maybe there are even existing solutions out there that we're unaware of, we'd love to hear.

Answer 1

You have a choice between sanity and robustness.

The sanest way to do this is indeed to generate the unique ID. You then don't need to disturb S3 (or whatever) if, for example, you rename a file.

The trouble is that this gives you a huge heap of securely stored meaningless objects – since without a name, you have little idea what the object actually is.

So you have add a path-to-ID mapping layer to give meaning to the meaningless.

But what about robustness? Let's assume that your object store will survive anything up to a medium-size earthquake. That is fine, but only if the map also survives, because if it doesn't, you just have an unstructured pile of nameless data.

So you do need to pay careful attention to the design of the mapper. In your place I would have one privileged server which does the mapping – a nice single point of failure – and back it up to S3 every few seconds.

There are plenty of claimed-to-be-robust databases out there, but if you use one, you will be risking your whole business model on the whims of somebody you do not control. Their bugs will be your problem. And their business decisions will be even more your problem, if one day they decide that the particular database they are offering is unfashionable and they are not going to offer it as a service any more.

Different people will have different opinions on this last point – but, from experience, I prefer it if everything is my fault: both bugs and business decisions.