32

Why don't Windows/Linux use relational databases (RDBMS)?

I know they use file systems to store all data but don't you think it is more efficient to use databases like we use in web sites/web apps?

Please elaborate on the use of a file system over a database for storage.

This is not a duplicate of When should use of database be preferred over parsing data from a text file? I am talking in terms of only operating system contexts, and that question is generalized.

  • 32
    A file system is a database. – user7043 Jun 3 '15 at 6:07
  • 20
    Because file systems are necessary to implement data bases. – Kilian Foth Jun 3 '15 at 6:10
  • 16
    Windows uses a database, it is called "Registry". Or do you mean "relational database"? That's a different question. – Doc Brown Jun 3 '15 at 6:10
  • 6
    @gnasher729 The file system is a very particular kind of database, and as such only good for particular kinds of data. Other kinds of data are better served with different kinds of databases (e.g. relational). – user7043 Jun 3 '15 at 14:51
  • 6
    @KilianFoth, not really. You could write to a raw disk partition (which is not comparable to an OS file). – Paul Draper Jun 3 '15 at 19:08
60

Today, most database management systems (e.g. PostGreSQL, MongoDB, etc...) internally keep their data inside OS files (in the past, some DBMSs used raw disk partitions directly).

On recent computers still using spinning hard disks, the disk is so slow - relative to the CPU or the RAM - that adding a few software layers is not relevant. SSD technology might change that a bit, and some file systems are optimized for SSDs.

Files are present in most OSes in general for historical and social reasons (in particular, C compilers and most tools - editors, linkers - want files, so there is a chicken and egg issue), and because there are a lot of very good file system implementations.

BTW, some essential system facilities can use databases. For example on Linux PAM can be configured to use information in databases (but this is rarely done in practice). Also, some mail servers may store some or most of their data in databases (e.g. Exim).

Files are slightly lower abstractions than databases, so they can be easier to implement (as the file systems & VFS layer in the Linux kernel) and faster to use. In particular, the operations on files are much more restricted than those on databases. In fact, you could see files or file systems as some very restricted databases!

You could design an operating system without any files, but with some other orthogonal persistence machinery (e.g. having every process be persistent, then you don't care much explicitly about storage, since the OS is managing persistent resources). This has been done in several academic operating systems(1) (and also in the Smalltalk and Lisp machines of the 1980s, somehow in the IBM System i, a.k.a. AS/400, and in some toy projects linked from osdev), but when you design your OS this way you cannot leverage on many existing tools (e.g. you also need to make your compiler and your user interface from scratch, and that is a lot of work).

Notice that microkernel operating systems might not need files provided by kernel layers since the file systems are just application servers (e.g. Hurd translators running in userland). Look also at the unikernel approach in today's MirageOS

Linux (and probably Windows, which got most of its inspiration from VMS & Unix) need files to work. At the very least, the init program (the first program started by the kernel) must be an executable stored in a file (often /sbin/init, but it could be systemd these days), and (nearly) all other programs are started with execve(2) syscall so must be stored in a file. However, FUSE enables you to give file-like semantics to non-file things.

Notice also that on Linux (and perhaps even Windows, which I don't know and never used) sqlite is a library managing some SQL database in a files and providing an API for that. It is widely known that Android (a Linux variant) uses a lot of sqlite files (but it still does have a POSIX-like file system).

Read also about application checkpointing (which, on many current OSes, is implemented to write the process state in files). Pushed to the extreme, that approach does not need to manually write application files (but only to persist the entire process state using the checkpointing machinery).

Actually, the interesting question is why do current operating systems still use files, and the answer is legacy, and economic and cultural reasons (sadly, most programming languages and libraries today still want files).


Note 1: persistent academic OSes include Lisaac & Grasshopper, but these academic projects seem to be inactive. Look also into http://tunes.org/ ; it is inactive, but has gotten lots of discussions around such topics.

Note 2: the notion of file has widely changed over time (look at this answer about my first programming experiences): the first MSDOS on 1980s IBM PCs (no directories!), the VMS -on 1978 Vaxen- (had both fixed-record files and sequential files, with a primitive versioning system), the 1970s mainframes (IBM/370 with OS/VS2 MVS) had a very different notion of files and file systems (in particular because at their time the ratio of hard disk access time to core memory access time was a few thousand - so at that time disk ran relatively faster than today, even if today's disks are absolutely faster than in the previous century, today the CPU / disk speed ratio is about a million; but we now have SSDs). Also, files are less (or even not) useful when the memory is persistent (as on CAB500 magnetic drum, 1960s; or future computers using MRAM)

  • 9
    It's also worth pointing out that some filesystems actually have a number of RDBMS features. For example, file metadata (particularly extended metadata) in BeFS is indexed with B+trees, and the BeOS file manager had a SQL-like lookup engine that searched indexed metadata to find files. – greyfade Jun 3 '15 at 16:03
  • 2
    I am not daring putting them in my answer, but both tunes.org & J.Pitrat's blog could widen your views about software & operating systems. – Basile Starynkevitch Jun 3 '15 at 17:25
  • 4
    @greyfade: A filesystem is an object database. No filesystems I know of have the ability to answer relational queries (e.g. files with mod times in a certain range.) You have to do that by querying the mod time of all files and filtering yourself. Some filesystems perform decently when used directly as an object database (storing millions of very small files, where the filename is the key), but others do ok with this kind of workload. – Peter Cordes Jun 3 '15 at 19:23
  • 3
    @PeterCordes: BeFS did that. Because all the metadata was B+tree-indexed, it supported range queries, wildcards, joins, and other fun stuff. I remember hearing that Microsoft was doing the same thing in WinFS. – greyfade Jun 3 '15 at 20:49
  • 4
    The PalmOS was one fairly mainstream OS that did not have a filesystem. Instead it had a relational database that was implemented directly onto RAM/flash (the original hardware didn't use flash memory like iPhones today but used battery-backed static RAM for both RAM and disk). – slebetman Jun 4 '15 at 5:37
23

Although this is opinion-based, I think it's just another historical artifact. Early OSes used a simple file system design for performance that was reasonably strongly tied to the characteristics of the hardware available at the time, and it's been the same way ever since. It's difficult to change the old file read/write APIs for more transactional query/insert APIs once they were established.

All current file systems have a requirement to be backward compatible with these old APIs.

Microsoft did think about replacing the file system with an RDBMS-based one, in the Longhorn development. That was too much of a change for them to pull off, but you do see their efforts continue in the form of Windows Search (where an RDBMS is used to store a copy of metadata) and features such as SQL Server's Filestream system (where a database table of file data is exposed to the OS as an ordinary directory allowing both Windows Explorer access to the data, and SQL queries of the same data).

Other OSs do have RDBMS filesystems. AS/400s used to have these, though I never learned enough about them; I remember how weird it appeared at the time). I think other mainframe systems have the same kind of approach.

  • 1
    If memory serves you may be thinking of the DB2 UDB on OS/400 aka i5/OS (now called just "IBM i"): publib.boulder.ibm.com/iseries/v5r2/ic2924/info/rzamb/… – Brian Cline Jun 4 '15 at 6:11
  • 1
    Yes, it would be great to BEGIN TRANSACTION / COMMIT on file permissions instead of doing a "find with -exec". The elevation of low-level primitive filesystem diddling into adminland is accidental and should go the way of the programming plugboard. The "filesystem" as a proper bytestream storage and metadata management system (although interpretation of bytestream content should still be left to the application layers, otherwise headaches will occur)? Yes, we want! – David Tonhofer Jun 1 '16 at 10:37
12

The real reason is a lack of need for it. Layering databases on top of files, rather than merging them, handles the vast majority of situations at least as well as a merged solution with substantially reduced complexity. In some situations others have mentioned, we've also layered parts of files on top of databases (such as permissions structures). In that case, the database managing those permissions is remarkably simpler than a commercial RDBMS.

There are advantages to merging them, but so far those have been few and far enough between that the movement is growing slowly. Consider how rare it is for people to say "Give me the 3rd column of every invoice I've received since 2010, and sum them together," or "don't let me delete this file until I've removed it from the Excel spreadsheet also."

File systems have a few advantages over relational databases that keep them going:

  • They are far simpler. This is a big deal when bootstrapping a computer. Even on Android, where they have an RDBMS for storage, they have plain old images for managing the initial bootloading process.
    • It is easier to define their limitations. In an unlimited machine, RDBMs provide quite a lot of power. However, in the file system world, there are a lot of limitations which stem from trying to be fast when directly layered on top of a spinning disk. It is harder to prove that an RDBMS query does not exceed those limitations than it is to provide the same guarantees for a file system.
  • They handle hierarchical structures better. In many cases, it is still natural for people to store files in a hierarchical form. In RDBMSes, that is a special case. File systems optimize for that special case, RDBMSes do not.
  • Reliability. It is much easier to prove that two layers work independently than to prove that one giant system works perfectly. RAID arrays, fail-safe journals in times of power failures, and other advanced features are easier to implement in a layer below the layer dealing with things like ACID or foreign key constraints.
  • 1
    reliability: you can run the DB on top of RAID just like you can run a filesystem on a RAID device, vs. using a disk directly. Journaling needs to be done inside the filesystem / DB, though (unless you instead want to provide correctness guarantees by disabling write caching, and never reordering I/Os. i.e. sync mode.) +1 for all your other points, esp. fast heirarchical performance where a ton of stuff in one subdir doesn't slow down performance in another subdir. Unless each dir or file is a different table... – Peter Cordes Jun 3 '15 at 19:16
  • reliability: the IBM i series OSes are designed to be more reliable than you can imagine, being designed for mainframe style usage. Hierarchies are only there because of filesystem limitations, hence MS wanting to later search and DB operations ontop of the existing filesystem. Look to gmail as an example how you can have a hierarchy without using hierarchies! – gbjbaanb Jun 7 '15 at 14:01
3

I think the other answers provide a wide spectrum of reasons as to why operating systems do not rely on relational databases internally/exclusively so I'll just share an interesting piece of information that I once stumbled upon.

Apparently, there are technologies that allow you to mount relational databases as file systems when their use is justified. Oracle DBFS (Database File System) is an example. This snippet from the documentation explains the rationale behind it quite well:

Database File System (DBFS) leverages the features of the database to store files, and the strengths of the database in efficiently managing relational data, to implement a standard file system interface for files stored in the database. With this interface, storing files in the database is no longer limited to programs specifically written to use BLOB and CLOB programmatic interfaces. Files in the database can now be transparently accessed using any operating system (OS) program that acts on files.

The solution provides a set of interfaces (command line clients, code libraries) for LOB data that is stored in database tables. This can be used on Windows and Linux operating systems (although as far as I can tell, the level of integration varies between them)

Oracle DBFS components

Source: docs.oracle.com

According to the documentation, the file system should be possible to use in a transparent manner on Linux

On Linux, the dbfs_client also has a mount interface that utilizes the Filesystem in User Space (FUSE) kernel module to implement a file-system mount point that provides transparent access to the files stored in the database and requires no changes to the Linux kernel. It receives standard file system calls from the FUSE kernel module, and translates them into OCI calls to the PL/SQL procedures in the DBFS Content Store.

Therefore, the answer to your question is that, in general, there's no reason for an operating system to use a relational database as a file system (and in case of the core components of an OS, this would actually be troublesome). At the same time it is possible for one to do it when some problem calls for it.

2

The main function of any OS is to facilitate interactions between applications, hardware and the users.

So.. why don't Windows/Linux OS use relational Databases (RDBMS)? This is a question of biblical proportions, but the short answer is: There is not any real benefit to be gained from using a complex structure such as an rdbms as a file system.

"Relational" is the operative word in "Relational Database" and most data stored in a file system is unrelated to other data. File systems generally are implemented as limited databases, just not relational ones.

  • Maybe a better question would be - why do applications need databases instead of simply persist data to files? I've never found a satisfactory answer to this question. All the supposed benefits of a relational database can be obtained with a file sustem – Sridhar Sarnobat Mar 23 '17 at 4:53

protected by gnat Jul 2 '15 at 6:49

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.