15
votes
Does a file system "see" the storage device as a (very large) byte array?
On Linux (and 1980s era Unixes), a storage device (quite often a disk partition on some hard disk, or on some SSD) is a block device (see this) so is a [sub-]sequence of blocks (which is the basic ...
- 32.7k
7
votes
What is the name for the non-extension part of a filename?
I believe that's just called the "filename" as well, which makes thing fairly confusing:
Discussions of filenames are complicated by a lack of standardisation of the term. Sometimes "filename" is ...
- 179
7
votes
Accepted
Building a program that truly deletes everything
You can very difficultly obtain a true irrecoverable deletion of data. This is not related to algorithms but to physical properties of storage media.
You can only hope to reduce the risk (or ...
- 80.6k
6
votes
Accepted
Why do disks write data in chunks of page size?
That's due to the mechanics.
A disk is a surface which rotates around its axis at a high speed (in reality several surfaces). The surface is divided into concentric tracks, and a motor controls the ...
- 80.6k
5
votes
Chat application - write to file and then save in database
The efficiency part of the question can't be answered with the given context since all we know is a client will make a request every 2.5 seconds. At a high level we don't know how many clients there ...
- 161
5
votes
Storing Local Filesystem Paths in Database
Youre correct in suggesting you should probably avoid storing the full filepath in your database.
A solution to this is to store a relative path to the files in the database, and store the root file ...
- 1,123
5
votes
Accepted
Can file systems be designed and implemented in an OS-portable way?
I will give you an example that shows that the answer to your question is both "Yes" and "No": FUSE.
FUSE stands for Filesystem in Userspace. FUSE is a Linux kernel filesystem driver that doesn't ...
- 104k
5
votes
fake filesystem for unit tests
The only thing which comes to my mind is to create needed file structure in a temporary directory and work on it, but that's not a perfect solution
Did you try it? This solution works well for many ...
- 214k
5
votes
Accepted
Why use strong checksums to detect random errors in a filesystem like btrfs?
A question is which width of xxhash is used?
Due to the birthday paradox the chance of an accidental collision is higher than you might think. For example there's a 50% chance of an accidental ...
- 762
4
votes
Can file systems be designed and implemented in an OS-portable way?
Yes, they can, the reason they usually aren't is simply because the people in control of the various OSs often are not interested in making things compatible with other platforms. Linux has drivers ...
- 27.7k
4
votes
How do you create a Composite file in C++
There are many composite file formats: tar, zip, 7z to name a few. There are basically two ways to create any of them in c++. Either download and import some library that will do the work for you or ...
- 115k
4
votes
Accepted
Store file in filesystem, and its metadata to the database atomicly
Here's a partial, but more practical and likely applicable, answer. If follow-on processes only consider files via their metadata entries, then the file upload and the update to the metadata need not ...
- 6,731
4
votes
Solving file system dependency with database storage
There are a lot of trade-offs in dealing with files in a database. Due to some practical constraints database manufacturers provide a means of storing the files outside the database file structure to ...
- 46.3k
3
votes
Why do disks write data in chunks of page size?
Your mechanical hard drive reads and writes data at 100 MB per second, using a strong magnet. To change a single byte, you’d have to turn that magnet on for ten nanoseconds. In order to not destroy ...
- 47.5k
3
votes
Does a file system "see" the storage device as a (very large) byte array?
Everything Basile Starynkevitch says is correct. I will add a bit more. Indeed disk drives were "block" drives, but block devices (and many other devices) were presented in two forms: "raw" and "...
- 41
3
votes
Accepted
Referencing custom Python modules and data files
What's crucially missing from your files hierarchy is __init__.py. This is the distinctive differences between a collection of Python scripts/data and a Python package.
I'm also wondering why you are ...
- 146
3
votes
What are the benefits of storing data contiguously?
Depends on so many factors, but here are a couple points to consider:
Recognize that if you use a very large file format that is "contiguous," its contiguousness is logical, not physical. The ...
- 26.9k
3
votes
How to detect the encoding of a file?
The character encoding can generally not be determined completely. However, there are many hints:
ASCII contains only bytes with values below 0x7F, originally it is a 7 bit encoding, but the byte ...
- 357
3
votes
Delete filesystem data associated with database record by delete hook, or internally in service logic?
Neither. Better would be not to use file system directly, but to allow database to handle the files as binary fields, or so called BLOBs.
Transactions
File systems are not transactional. If you use ...
- 864
3
votes
Building a program that truly deletes everything
This is a matter of opinion and/or marketing. In Linux such a program is called a shredder.
Overwriting with random data prior to overwriting with 0s is recommended
Such programs don't usually claim ...
- 39.5k
3
votes
What are some architectures and designs I could use to optimize the performance of file text data lookups?
You want it fast.
And you want it portable.
The principal thing working against you is random I/O.
The more you can read "small" records sequentially, the better.
There's a fixed cost to ...
- 7,605
3
votes
Accepted
How to Design a Secure Script for Conditional File Access Based on Time and API Conditions?
To be honest, if you give the program to the user, they can find some way to bypass security so long as the have file permissions. Security through obscurity is never enough. It can be one layer of ...
- 42k
2
votes
What are the benefits of storing data contiguously?
strictly append-only - no updates to already-written data are allowed.
This sounds suspiciously like a log. Did you consider a log-oriented data storage? Like a blockchain or logstash or git?
The ...
- 12.5k
2
votes
What are the benefits of storing data contiguously?
This appears to be case of premature optimisation - it is an attempt to performance-tune a design that has not exposed a performance problem and in doing so, tightly couples the file format to ...
- 3,188
2
votes
Accepted
One row database table or JSON file
With a database approach you can relate that line with the others in the database. With the file you'll need to read it and initialize a CLOB to process it.
In other words, the data will exist for the ...
- 1,461
2
votes
Detect when a file is created (on a webserver) and ready for use in one of many directories
If you create a system of different components which you don't have under your control, at a certain level of complexity some of the components will fail sometimes. That is especially true when your ...
- 214k
2
votes
Why do disks write data in chunks of page size?
All memory devices at every level of the memory hierarchy (from L1 cache, main memory, disk...) offer sequential access as a faster mode compared to random access. Random access requires ...
- 34.4k
2
votes
How to append a chunk of fixed size data to a file and make sure this chunk doesn't get fragmented on disk?
You don't. At least in the physical disk access sense.
There may be a way for particular platforms to allocate multiple contiguous chunks (e.g. ask for them all in one go), but it doesn't matter if ...
- 11.6k
2
votes
Does a replicated distributed file system minimise the need for durability?
Key value stores that don't persist to disk are generally billed as caches, implying there is another long-term durable storage your application manages somewhere else. If your data is important to ...
- 148k
Only top scored, non community-wiki answers of a minimum length are eligible