121

Files generally indicate their encoding with a file header. There are many examples here. However, even reading the header you can never be sure what encoding a file is really using. For example, a file with the first three bytes 0xEF,0xBB,0xBF is probably a UTF-8 encoded file. However, it might be an ISO-8859-1 file which happens to start with the ...


30

Size is not so much of an issue, the ability to query and maintain the data however is. If, for example, Greenhaven Press decides they want to change their name to Greenhaven Press International, you'll have to find the record, deserialize it, change it, serialize it, pump it back into the database. Consider this: does storing these objects as serialized ...


25

You cannot. If you could do that, there would not be so many web sites or text files with “random gibberish” out there. That's why the encoding is usually sent along with the payload as meta data. In case it's not, all you can do is a “smart guess” but the result is often ambiguous since the same byte sequence might be valid in several encodings.


23

That would be super obnoxious. You: Wouldn't be able to see them all at once Wouldn't be able to place comments explaining why things are set the way they are Wouldn't be able to have documentation on what commands do, or what possible options are right in the configuration It would make backing up/version controlling your configs more obnoxious. Using a ...


15

On Linux (and 1980s era Unixes), a storage device (quite often a disk partition on some hard disk, or on some SSD) is a block device (see this) so is a [sub-]sequence of blocks (which is the basic unit of physical I/O). The physical block size depends on the hardware (old IDE disks had a block size of 512 bytes, new large SATA disks have a block size of ...


12

In the vast majority of languages and frameworks, the stream is an abstract concept, and methods can be designed to accept generic streams (which can be substituted for in-memory data) instead of concrete file system objects. I personally have not found much benefit to unit-testing file operations beyond that level, because there are so many things that can ...


11

Given that most file systems store the contents of files in individual blocks that are not necessarily contiguous on the physical disk, but linked via pointer structures, it seems that such a mode - "inserting" rather than "appending" or "overwriting" - ought to be possible, and could certainly be made more efficient than what we have to do now: read the ...


11

Theoretically, you could implement a file that would allow this sort of thing. For maximum flexibility, though, you'd need to store a pointer to the next byte along with every byte in the file. Assuming a 64-bit pointer, that would mean that 8 of every 9 bytes of your file would be composed of internal pointers. So it would take 9000 bytes of space to ...


11

Yes, if you only do the replacement on Windows, and turn it off when running on other systems. Doing the replacement on Unix-like systems is wrong because \ is a valid character in a file or directory name on Unix-like platforms. On these platforms, only NUL and / are forbidden in file and directory names. Also, some Windows API functions (mostly the lower ...


11

It's called deduplication. Some filesystems do it (like ZFS), some block-level storage systems do it (like NetApp), some backup systems do it (rsnapshot), source code managment systems do it (Git, bzr, fossil) It's not so rare, just that until recently it was an expensive option for generic filesystems. Note that it's not a good idea to do it as you ...


10

"Consumer-oriented" (whatever that may mean) usages tend to use Title Case, even in the Unix world. This is more often the case when the organization in question has people whose whole job is to think about user experience. For example, my Ubuntu desktop has folders in the home directory called Downloads, Pictures, Documents, etc. Same goes for my OSX ...


10

There are two reasons why this is desirable. Directories cannot be arbitrarily large. E.g. some (reasonably modern!) filesystems are limited to 32000 entries in a single directory. The number of commits in the Linux kernel is in that order of magnitude. Subdividing the commits by their first two hex digits limits the top-level size to 256 entries. The sub-...


9

This really depends on the DB system, but one major thing you have to consider with BLOBs is transaction processing. By externalization to the filesystem, one takes changes to the binary data out of the transactions. That will typically result in faster write operations, opposed to the situation where the DB assures you ACID compliance with full rollback ...


9

Read speed and caching is an important factor, but it's not the only factor, and perhaps not even the primary factor in selecting a filesystem's block size. Every block on your filesystem has overhead associated with it. The filesystem must track which blocks are free, which blocks belong to which files, etc. This overhead must be stored on the disk ...


8

This is a perfectly fine way of representing tree-shaped data. A file system is a tree database, why re-implement one on top of it? The most well-known implementation of this idea is the Windows Registry. Its main flaw is that it implements a filesystem alongside the filesystem and doesn't support the filesystem API, which means you can't use filesystem ...


7

The fastest way is just to compare hash code of files having same size. This is the idea of of this answer on SO (see the second command line and its explanations). There is no security issue while detecting duplicated files, therefore I would recommend a fast hashing code. For instance the project ccache uses MD4: ccache uses MD4, a very fast ...


7

It's usually an issue with bandwidth. If you're serving up hundreds of videos an hour, then you're tying up the bandwidth in and out of the database, mostly copying buffers. It's also an issue if you have naive queries (possibly auto-generated by an ORM tool) that simply select all columns from a table. You're also subject to file fragmentation like a ...


7

On Unix they were an accident at least initially https://plus.google.com/+RobPikeTheHuman/posts/R58WgWwN9jp Second, and much worse, the idea of a "hidden" or "dot" file was created. As a consequence, more lazy programmers started dropping files into everyone's home directory. I don't have all that much stuff installed on the machine I'm ...


7

I believe that's just called the "filename" as well, which makes thing fairly confusing: Discussions of filenames are complicated by a lack of standardisation of the term. Sometimes "filename" is used to mean the entire name, such as the Windows name c:\directory\myfile.txt. Sometimes, it will be used to refer to the components, so the filename in this ...


6

The SEFS - Self Expiring File System prototype implementation has been designed based on the freely available FUSE file system. I have done this for my Masters Degree in Computer Science.


6

You may want to look into SQL Server FileTables. The idea is to provide the best of both worlds: file system level access and performance, along with database access and integrated security and services. The database does have a performance over-head in some cases. Just compare a hard-coded HTML file on a webserver to one that has to fetch the contents from ...


6

Because reading multiple blocks allows the OS to cache the next few blocks and avoid the latency when the application eventually needs them. The biggest slowdown in accessing spinning disk drives (HDD and optical disk) is the seek time, literally the reading head moving to where the track with the block is. This is alleviated with solid state but reading ...


6

This depends heavily on the filesystem and the operating system. For example, on Unix, there are no folders at all: Windows has completely different terminology. Classic MacOS had a different terminology, which, in OSX, is now mixing with Unix terminology.


6

That's due to the mechanics. A disk is a surface which rotates around its axis at a high speed (in reality several surfaces). The surface is divided into concentric tracks, and a motor controls the electromagnetic head to move to the right track. The head then just waits until the information it wants to read/write passes under it. So the problem is then ...


5

To know why this is ugly, you have to know how a database is saved on the hard drive (specifically rows). The physical contents of a row saved on the disk is divided into its static and dynamic counterparts. Fields such as int, byte, char(n) which have a fixed length are listed first. What follows is a number of fixed length which refers to the number of ...


5

Assuming you don't have an immediate need to query over those JSON fields, then there's nothing wrong with this approach, so long as the performance hit of deserializing/serializing isn't a problem for your program. Later on, let's say the requirement comes in to order/filter based on that "sales_rank" JSON field. You'd write a program to run over the data ...


Only top voted, non community-wiki answers of a minimum length are eligible