Questions tagged [file-systems]
techniques to organize and store files with their data on a computer.
100
questions
165
votes
5
answers
802k
views
How to detect the encoding of a file?
On my filesystem (Windows 7) I have some text files (These are SQL script files, if that matters).
When opened with Notepad++, in the "Encoding" menu some of them are reported to have an encoding of "...
- 3,112
53
votes
3
answers
16k
views
What is the benefit of writing to a temp location, And then copying it to the intended destination?
I am writing an application that works with satellite images, and my boss asked me to look at some of the commercial applications, and see how they behave. I found a strange behavior and then as I was ...
- 2,534
50
votes
9
answers
15k
views
Why is filesystem preferred for logs instead of RDBMS?
Question should be clear from its title. For example Apache saves its access and error logs in files instead of RDBMS no matter on how large or small scale it is being utilized.
For RDMS we just have ...
- 1,239
28
votes
11
answers
3k
views
Why are there so many competing filesystem designs? [closed]
Just a quick question, but why are there so many file systems still competing and in use today? (ntfs, fat32, ext3(ffs), etc)
It seems that file system designers could agree upon the best aspects of ...
- 6,303
26
votes
4
answers
6k
views
Why is the Git .git/objects/ folder subdivided in many SHA-prefix folders?
Git internally stores objects (Blobs, trees) in the .git/objects/ folder. Each object can be referenced by a SHA1 hash that is computed from the contents of the object.
However, Objects are not ...
- 4,799
25
votes
1
answer
7k
views
What is the name for the non-extension part of a filename? [closed]
Given the file path:
/some/path/abc.txt
The filename is "abc.txt", and extension is "txt".
What is the "industry standard", unambiguous name for the "abc" part?
For reference, in both java's older ...
- 1,996
18
votes
5
answers
35k
views
Is it safe to convert Windows file paths to Unix file paths with a simple replace?
So for example say I had it so that all of my files will be transferred from a windows machine to a unix machine as such: C:\test\myFile.txt to {somewhere}/test/myFile.txt (drive letter is irrelevant ...
- 789
17
votes
6
answers
51k
views
Is it wise to store a big lump of json on a database row
I have this project which stores product details from amazon into the database.
Just to give you an idea on how big it is:
[
{
"title": "Genetic Engineering (Opposing Viewpoints)&...
- 857
13
votes
7
answers
5k
views
Why can we not insert into files without the additional writes? (I neither mean append, nor over-write)
This occurs as a programming language independent problem to me.
I have a file with the content
aaabddd
When I want to insert C behind b then my code needs to rewrite ddd to get
aaabCddd
Why can ...
- 795
12
votes
3
answers
11k
views
Lowercase in Linux file names
As I find that UpperCase are really readable for first letter word separation in long complex names, I tend to give some of my Linux files names with some UpperCase. Mostly executables, some ...
- 596
12
votes
2
answers
716
views
Does a file system "see" the storage device as a (very large) byte array?
I want to know how does a file system write to and read from a storage device.
I think this is how it works:
A file system doesn't access the storage device directly, but rather the storage device ...
- 335
8
votes
4
answers
4k
views
Why is Tortoise SVN case sensitive?
I've recently encountered this using TortoiseSVN, but I assume it will be the same for CVS based programs (correct me?).
Out of pure curiosity, is there any reason why the CVS filesystem is case-...
- 183
8
votes
4
answers
2k
views
What exactly does it mean that storing “large blobs in the database reduces performance”?
To someone who knows database internals this may be an easy question, but can someone explain in a clear way why storing large blobs (say 400 MB movies) in the database is supposed to decrease ...
- 187
8
votes
7
answers
11k
views
Automatically delete files after they expire
I've got this idea for some time and I was wondering if anyone has seen such a feature/app in any operating system and if you haven't, what do you think about it. Where do you think I should begin?
...
- 229
8
votes
4
answers
864
views
File shredder algorithm
As a part of learning system programming, I am looking to implement a file shredder. The simplest way (and probably seen as naive) would be to replace the data bytes with zeroes (I know OS splits the ...
- 81
8
votes
1
answer
7k
views
Efficient data structure to implement fake file system
I want to implement a data structure that will hold the paths of directories, sort of fake file system.
Input:- I have a text configuration file containing the paths as follows
...
C:/temp1
C:/...
- 89
7
votes
3
answers
367
views
Examples of limitations in IT due to different bit length by design [closed]
I am teaching the course "Introduction in Programming" for the first-year students and would like to find interesting examples where the datatype size in bits, chosen by design, led to certain known ...
- 1,880
7
votes
3
answers
7k
views
How many threads should access the file system at the same time?
We have a module in an application which stores data in multiple files and multilevel directories and access them from multiple threads (both reads and writes). The directory structure is based on a ...
- 71
6
votes
3
answers
2k
views
What's the point of hidden files?
What is the point of hidden files? In Microsoft Windows they exist, in Mac OS X they exist and in Linux they exist. It seems to me that it just makes detecting malware more difficult. The only upside ...
- 216
6
votes
3
answers
1k
views
Why would anyone want to build a file system for windows? [closed]
I saw an ad on StackOverflow today for a project called WinFsp.
The site mentions the following features:
Allows for easy development of file systems in user mode. There are no restrictions on what ...
- 429
6
votes
2
answers
420
views
Is there a fundamental reason for keeping file systems and versioning systems separate?
I've done some work looking at Next3 and ext3cow, and a few other file systems that provide snapshots and/or file versioning. It seems a bit strange to me that file systems only go part-way with ...
- 13.3k
6
votes
1
answer
564
views
Solving file system dependency with database storage
I have a software project A which makes API calls to a third-party software B that is heavily based on data stored on the file system. Also, those software and file systems are distributed on servers ...
- 245
6
votes
5
answers
20k
views
How to Model a simple file-system by UML class diagram
I want to model a file system which contains both files and directories, and directories can contain either files or other directories.
This is what I have reached so far:
In OOSE book, however, a ...
- 265
5
votes
2
answers
2k
views
Why do filesystems read and write in blocks?
I read that file systems usually access data in blocks, whose size is integral multiple of disk block size. Why don't they read individual disk blocks?
- 63
5
votes
2
answers
12k
views
Finding duplicate files? [duplicate]
I am going to be developing a program that detects duplicate files and I was wondering what the best/fastest method would be to do this? I am more interested in what the best hash algorithm would be ...
- 304
5
votes
7
answers
3k
views
How do i speedily traverse a file system while extracting/extrapolating various data and provide user feedback?
I'm working on a system file scanner that reveals info about various files (e.g. size, last used, duplicates, etc). Currently I'm traversing the file system once just to get a good measure of the ...
- 193
5
votes
1
answer
1k
views
Why use strong checksums to detect random errors in a filesystem like btrfs?
Btrfs supports crc32c, xxhash, sha256 and blake2b as checksums when storing and reading files. crc32c and xxhash are designed to detect random errors while sha256 and blake2 are considered ...
- 61
5
votes
2
answers
3k
views
Storing Local Filesystem Paths in Database
I'm developing a webapp where I have sets of data stored locally on my computer and I run a tool which transforms the data and uploads it to my webapp. However, I need to be able to rerun the tool on ...
- 159
4
votes
5
answers
580
views
Is it plausible to use a filesystem-based configuration format rather than an INI file?
After having had endless issues with INI-style configuration files and parsing them correctly (let along getting binary data right), a crazy idea crossed my mind recently:
What if an INI file like
# ...
- 329
4
votes
4
answers
1k
views
Why do disks write data in chunks of page size?
In my understanding, even if i want to overwrite a byte in middle of a file, OS and/or disk will read the content of the size of page, modify one byte and then write the contents back.
What is the ...
- 251
4
votes
2
answers
3k
views
Is it a bad idea to sync file system with remote server using HTTP?
I have started a project which will duplicate Dropbox or Google Drive behavior but using Amazon S3 az a backend.
Idea is very simple, a Node.js server that watchs a directory for file changes and PUT ...
- 1,990
4
votes
1
answer
5k
views
File system implementation in MongoDB with GridFS
I am working on two projects that will both implement a Webdav server backed by a MongoDB GridFS. In each case, there is the potential for the system to store tens of millions of files spread across ...
- 143
4
votes
1
answer
5k
views
Directory structure (file system design)
I was looking at how file systems are designed and noticed that most places say that the directory hierarchy can be implemented using a hash table.
Could someone please explain me how using a hash ...
- 43
4
votes
2
answers
420
views
Dealing with potential failures when appending data to a file on disk
I'm designing an application that will be appending blobs to a file on disk (local filesystem) and I'm currently thinking of how to deal with consistency issues that could occur if:
The application ...
- 547
3
votes
2
answers
768
views
Why is it not standard to implement abstraction layers for the file system?
I have been taught to access databases through abstraction layers. I was wondering why it is not also standard practice to access the file system through an abstraction layer? It seems to me unit ...
- 1,079
3
votes
3
answers
337
views
Filesystems that use logging: If you're writing the data in the log (on disk), and in the actual locations themselves (also on disk) then...?
Aren't you essentially writing the same data twice, to the disk? Doesn't this cause a slowdown of a factor of ~2x?
What optimizations can be made to minimize the cost of having to write things twice?
- 253
3
votes
4
answers
1k
views
Can file systems be designed and implemented in an OS-portable way?
Given the interfaces that major OSes (Windows, macOS/OS X/Mac OS X, Linux) provide to file systems, can file systems be designed and implemented in a way that is largely independent of OS?
I'm not at ...
- 1,634
3
votes
4
answers
786
views
Can memristors change programming paradigms? In what way?
Think about persistent storage that can preserve pointers: among other things there is no need to serialize/deserialize data. For example, you typically build a tree structure and then serialize it ...
- 5,643
3
votes
2
answers
2k
views
Store file in filesystem, and its metadata to the database atomicly
I have to store many pdf/jpg/png file of max 10mb in a filesystem, and need to save their metadata on a database.
The SFTP and the DB may be on different nodes. On WS, I've a local db where I can ...
- 133
3
votes
1
answer
640
views
Filesystem superblocks and their backup copies
I'd like to understand how (modern) filesystems are implemented and having trouble to fully understand superblocks and their backups. I reference ext4 and btrfs, but the questions may also apply to ...
- 281
3
votes
5
answers
3k
views
Best practice to sync long paths to filesystems where path length is limited?
I wrote a sync tool to synchronize folders/files from Alfresco to Windows.
PROBLEM:
In Alfresco, /files/can/have/very/very/very/long/filepaths/like/this.txt
The Windows API prevents me from creating ...
- 1,062
3
votes
1
answer
2k
views
NoSQL as file meta database
I am trying to implement a virtual file system structure in front of an object storage (Openstack). For availability reasons we initially chose Cassandra, however while designing file system data ...
- 193
3
votes
3
answers
429
views
File I/O - How HDD or SSD works with OS file system? [closed]
I've studied how data transfer with secondary storage(HDD or SSD) works.
Would you mind to check that my understanding is correct?
File system
block is basic read/write logical unit which is used in ...
- 133
3
votes
1
answer
1k
views
VBA Outlook: quickly find subfolder
I have the following structure in my Outlook Public Folders.
-Public Folders
--1001_RandomProject
--1002_AnotherProject
--1003_Yetanotherproject
...
and so on, basically there's a couple of thousand(...
- 143
3
votes
1
answer
2k
views
How to organize my site's file system properly?
Doing some reading on Stack Overflow, I've found a lot of information suggesting that proper organization of a file system is crucial to a well-written web app. One of the key pieces of evidence is ...
- 133
3
votes
1
answer
669
views
How to store the file names, start offset and length while avoiding the issue of self imposed limits (lookup table) or having to scan the entire file?
I am attempting to learn more about C and it's descendants(C++ mainly). I have decided that I would like to create a "file system" of sorts. Not a particularly advanced one mind you but something to ...
- 33
2
votes
3
answers
333
views
Is there an established convention for separating Windows file names in a string?
I have a function which needs to output a string containing a list of file paths. I can choose the separation character but I cannot change the data type (e.g. I cannot return a List<string> or ...
- 9,758
2
votes
2
answers
576
views
How are non-folder files called? [closed]
Maybe it looks like a weird question, but what term should be attributed inside the code for files that are not folders to differentiate them from folders?
If I need write 2 functions isFolder() and ...
- 49
2
votes
4
answers
338
views
Where can I learn about how programs handle file systems like NTFS?
I'm writing a program to handle the xbox 360's STFS files. I'm encountering all sorts of problems about how to keep track of used vs unused blocks, reading non-sequential files, etc. These aren't so ...
- 1,159
2
votes
3
answers
777
views
Storing lots of large strings with frequent "appends" and few reads
In my current project, I need to store a very long ASCII string to each instance of a given object. This string will receive an 2 appends per minute and will not be retrieved so frequently.
The ...
- 161