3

Given the interfaces that major OSes (Windows, macOS/OS X/Mac OS X, Linux) provide to file systems, can file systems be designed and implemented in a way that is largely independent of OS?

I'm not at all an expert on file systems and I've never implemented one but I am an enthusiastic user of file systems. My naive expectation is that almost all file system logic is independent of operating system. The OS queries a file system about paths in a C-compatible way, the file system answers in a C-compatible way, and no deep interdependencies are necessary. In practice, this seems to not be the case. For example, simply reading from an ext2/3/4 file system when using Windows or macOS takes significant effort from a user perspective.

Are there technical reasons that file systems seem deeply entwined with operating systems? Do the major OSes interact with file systems in fundamentally incompatible ways? Alternatively, is this mostly due to legal issues? Or maybe the premise of the question is flawed and there just isn't any demand for portable file systems?

(I realize that most OSes incorporate core file system drivers within the OS kernel itself but most also allow loadable OS modules to describe new file systems so the question remains: are there/why are there not OS modules for file systems that are largely OS-independent?)

As a non-expert, I might be wrong on some specifics. For example, maybe the tools/extensions that allow Windows and macOS to read and write ext2/3/4 file systems do share plenty of code with the Linux kernel. Even in that case, a good answer should still explain the apparent difficulties and why other OS + file system combinations are not as easy.

EDIT: To be clear, I'm not asking about incompatibilities at the ABI level. Many particularities can create issues at the ABI level and most are not specific to this question. This question is about whether the APIs provided to file systems by the major OSes are fundamentally incompatible.

  • 2
    How about FAT32? It's what most USB drives use and every major OS can use it seamlessly. But the answer here is the same as for any other thing you might want standardising: most groups involved aren't incentivised to: 1. give up controlling something; and 2. make it easier for you to go elsewhere. – jonrsharpe Mar 24 '17 at 3:12
  • The thing that you hoped for (some kind of portability across OS) was possible at the block level. In fact, virtual machines sort of works like that. Given a block storage device (which might be physical/hardware or logical/software/virtualized), you can run your own operating system and file system module on top of it. – rwong Mar 24 '17 at 3:19
  • Low-level code, such as C compiled into binary code, are OS-specific. They depend on specific C runtime libraries, as well as OS API or system calls. At a lower level, they also depend on specifics on hardware, such as IRQ, CPU architecture, CPU privileged modes, special registers, etc. Therefore, the fact that file system drivers are OS-specific is a consequence of them often being written using low-level language such as C and then compiled into binary code for that OS. – rwong Mar 24 '17 at 3:21
  • What does "The OS queries a file system about paths in a C-compatible way" mean? What is a C-compatible way? Do you mean creates C-strings to handle file paths? If so, then you're incorrect. Apple's original HFS used Pascal strings and did not store files by path, but by a reference number (sort of similar to an inode number). From Wikipedia: Files were referenced with unique file IDs rather than file names, and file names could be 255 characters long. – user1118321 Mar 24 '17 at 3:33
  • @jonrsharpe FAT32 is a good practical suggestion but my intent really is to ask about the programming challenges of implementing a portable file system. I do suspect that the last part of your comment is correct but I'm hoping for an answer from someone who feels authoritative or can give citations. – Praxeolitic Mar 24 '17 at 3:37
4

I will give you an example that shows that the answer to your question is both "Yes" and "No": FUSE.

FUSE stands for Filesystem in Userspace. FUSE is a Linux kernel filesystem driver that doesn't actually implement a filesystem; instead, it provides an abstraction layer, an API, and an ABI that exports a protocol for interacting with the FUSE kernel driver from userspace. The second component of FUSE is a userspace library that implements an API wrapper for that protocol.

The end result of all that, is that you can write a filesystem driver in userspace, by linking against the libfuse library and implementing the required callbacks. There are also a set of language bindings for almost every language imaginable, so that you can, for example, write a filesystem driver in Python, Ruby, PHP, ECMAScript, Java, C♯, Scala, and whatever language you can think of.

Now, what does that have to do with this question? Well, the FUSE kernel-userspace protocol and the FUSE library have been implemented on other operating systems as well, and code written against some subset of the FUSE library API is mostly portable across operating systems. So, in that sense, the answer to your question is "Yes, filesystem drivers can be implemented in a portable manner".

But! Here's the catch. Or, actually two catches. First off, FUSE itself is rather complex. For example, the Linux kernel filesystem driver implementing the kernel-userspace protocol is 9000 lines. The macOS kernel filesystem driver is over 17000 lines. And the userspace library which implements the API for userspace filesystem drivers to write against is another 10000 lines. So, that's 36000 lines of code, just to smooth over the differences between Linux and macOS. Remember: those 36000 lines don't actually do anything. They only implement the common API so that FUSE userspace filesystem drivers written against the API can run on macOS and Linux.

And, for example, there are no ports for Windows. There are, however, similar projects for Windows, and some of them provide a compatibility layer for FUSE. But those compatibility layers usually aren't perfect, because the architecture of Windows filesystem drivers is just too different from Unix filesystem drivers (from which both Linux and macOS are descended).

The second catch is that FUSE filesystems tend to be slow. Now, part of that is because they are implemented in userspace and crossing the kernel-userspace boundary is expensive. But part of that is also because FUSE abstracts away a lot of the nitty-gritty OS-specific stuff like the block layer, the I/O scheduler, the filesystem cache, the various zerocopy implementations, and so on. And because FUSE abstracts them away, FUSE userspace filesystems can't take advantage of them.

So, to recap: Yes, FUSE (or rather the various filesystems implemented using FUSE) are existence proof that it is possible to design an API that abstracts away the differences between OSs. But no, filesystem drivers are not "almost all logic independent of the operating system". Looking at FUSE, you need between ~10000 to ~20000 lines of kernel code for each and every operating system, plus an additional ~10000 lines of code for the userspace library to create an abstraction layer that lets you write filesystems in an OS-independent manner. The actual filesystems themselves, are usually much smaller. For example, the FUSE-based sshfs is only ~3000 lines of code. In other words: ~3000 lines of OS-independent code vs. ~10000–20000 lines of OS-dependent code. And another "no" to practicality: you cannot write a high-performance filesystem implementation this way.

ZFS is another interesting case study: typically, most Unix-like OSs have a layered implementation of filesystems with a similar layering. Physical block devices on the bottom, logical block devices (RAID) in the middle, filesystems on top. One filesystem lives on exactly one logical block device, one logical block device lives on one or more physical block devices. ZFS has a completely different layering: it has only two layers, storage pools and filesystems. The boundary is somewhere in the middle of what would be the middle layer in a traditional layering scheme. Filesystems dynamically allocate space from storage pools, one filesystem can live on multiple storage pools, one storage pool can host multiple filesystems. This required extensive changes to the Solaris kernel. Now, if ZFS even requires extensive changes to the OS for which it was designed, then you can probably imagine that you cannot just port it to Linux or Windows without any changes.

I think your main problem is that your view of filesystems is much too simplistic:

The OS queries a file system about paths in a C-compatible way, the file system answers in a C-compatible way, and no deep interdependencies are necessary.

Umm. That's a pretty boring filesystem. Why? Well, you didn't write anything about the filesystem accessing the disk. In the real world, filesystems need to read and write from a disk or a RAID. Or the network in case of a network filesystem. They have to interact with the OS's filesystem cache. They have to interact with the OS's block cache. The have to interact with the OS's I/O scheduler. They have to interact with the OS's virtual memory system (for stuff like mmap). In Linux, it needs to interact and fit into the Linux Virtual Filesystem Switch (VFS). There are users and permissions and roles and attributes and security contexts and domains and access restrictions and ACLs and policies and such. These work completely different between Unix and Windows, for example.

For example, maybe the tools/extensions that allow Windows and macOS to read and write ext2/3/4 file systems do share plenty of code with the Linux kernel.

Not with the Linux kernel, no, they don't. However, the Linux Ext developers actually develop two implementations of Ext in lock-step, the Linux kernel filesystem driver, and libext2fs (part of e2fsprogs), which is a userspace library that implements (mostly read-only) support for ext2/3/4 (and is in turn used by tools such as debugfs, e2fsck, tune2fs and mke2fs). And the userlevel tools available for Windows and macOS could use this library if they wanted to.

  • "And the userlevel tools available for Windows and macOS could use this library if they wanted to." ... I've actually used a kernel driver for ext2 on Windows that was based on libext2fs, so I don't think there's any necessity for such tools to be user level. – Jules Mar 25 '17 at 4:36
  • This is a great answer. From the FUSE example, I'm fully convinced that the OS APIs are somehow fundamentally incompatible. The answer could still be improved with just a few details on why that is. It may help to know that I found it surprising that filesystems directly interact with RAID, OS caches, I/O schedulers, and virtual memory systems. My naive expectation was that those OS features would all be entirely above the filesystem during disk interaction (or perhaps below in the case of RAID). – Praxeolitic Mar 28 '17 at 5:43
4

Yes, they can, the reason they usually aren't is simply because the people in control of the various OSs often are not interested in making things compatible with other platforms. Linux has drivers for a wide variety of filesystems, including NTFS, and there are ext2/3 drivers for Windows as well.

Microsoft created NTFS because they wanted a comprehensive security capability through access control lists, a function FAT32 did not have. They implemented, plugged it into windows, and were done. They had no reason to concern themselves with the needs of any competing operating system. Eventually, other people made an NTFS driver for other operating systems and it is in use in many systems today.

Your suspicion:

Or maybe the premise of the question is flawed and there just isn't any demand for portable file systems?

Is mostly what's going on.

Ultimately, your disks can be viewed as a massive string of arbitrary bytes, and reading and writing those bytes is fundamentally just like reading and writing the bytes of an individual file. There isn't anything in NTFS that requires Windows, just knowledge of the structure of the data so you can write a driver to provide the common file access methods. Same with any other filesystem.

Secrecy of a filesystem specification can make writing those drivers more difficult, but that isn't a technical limitation.

Your question is the computer equivalent of "why can't I put a GM water pump in my Toyota vehicle?" There's really nothing fundamentally different between GM and Toyota water pumps, it's just that each company doesn't care about being compatible with their competitor's parts. They sometimes actively avoid it, but indifference is usually sufficient.

  • I do strongly suspect that this answer is all there is to say about the matter but if a newer answer can give an interesting and more technical description of the difficulties of supporting the multiple OS filesystem APIs, I will readily consider changing the accepted answer. – Praxeolitic Mar 24 '17 at 6:21
  • 1
    @Praxeolitic: you're confusing a filesystem, which is a system for organizing bytes in a disk, and the various drivers that provide the abstracted way of accessing those bytes in a "friendly" manner. The trouble is that each OS provides different abstractions, even to the same filesystems. – whatsisname Mar 24 '17 at 6:25
  • If you could clarify to some extent how the major OSes do provide fundamentally different abstractions to filesystems, that would be a great addition to your answer. (Again, I'm no expert, but if it helps, I essentially imagine every filesystem responding to OS queries about paths by returning a type (directory/symlink/file, etc.), a handle, and a size, regardless of OS.) – Praxeolitic Mar 24 '17 at 6:37
  • @Praxeolitic It's simply a matter of the OS having made note of what implementation of the abstraction to talk to for files under each mount point, e.g., /media/usbstick is FAT32 and /media/externaldrive is ext3. Most filesystems follow the same directories-and-files-with-attributes model as most OSes, so the abstraction isn't even particularly sophisticated. – Blrfl Mar 24 '17 at 11:25
  • 1
    @Praxeolitic: "I essentially imagine every filesystem responding to OS queries " again, you're confusing a filesystem with a driver. A filesystem is a method of organization. It's a layout of books on a bookshelf. It doesn't perform actions or respond to queries, drivers do that. – whatsisname Mar 24 '17 at 14:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.