Questions tagged [file-systems]
techniques to organize and store files with their data on a computer.
-2
votes
0answers
46 views
How to use file system as cache for products which are expensive to produce? [on hold]
Context
The consumer client components consume files. They access files in two step:
Clients call a REST API with a parameters what they want, and the API responds with a file path.
Clients access ...
1
vote
1answer
71 views
How to access a file stored locally on server?
I am working on a web application which allows users to review pdf documents. These documents are submitted from another public facing website. A typical workflow involves:
A document is uploaded on ...
-1
votes
2answers
99 views
How to append a chunk of fixed size data to a file and make sure this chunk doesn't get fragmented on disk?
So i want to understand how DBMS implementation works
To give an example :
MySQL implements each tables with its own pages, which are 16KB
so each table is a file, and is a multiple of 16KB, ...
0
votes
1answer
57 views
Does a replicated distributed file system minimise the need for durability?
I've been investigating various distributed file systems, like Gluster, Ceph, Moose and Lizard.
I'm also familiar with various key/value store type systems, some of which do not perform any system ...
1
vote
3answers
229 views
Why do disks write data in chunks of page size?
In my understanding, even if i want to overwrite a byte in middle of a file, OS and/or disk will read the content of the size of page, modify one byte and then write the contents back.
What is the ...
0
votes
2answers
95 views
Referencing custom Python modules and data files
I'd want to deploy my Python code and relevant static files such that only a copy of a folder is needed. That is, all the paths inside are relative. The release is to a web server, which calls scripts ...
1
vote
1answer
488 views
Chat application - write to file and then save in database
I have followed this approach that is described here to implement a simple chat application: https://code.tutsplus.com/tutorials/how-to-create-a-simple-web-based-chat-application--net-5931
I'm ...
0
votes
2answers
86 views
Detect when a file is created (on a webserver) and ready for use in one of many directories
I have an odd, intermittent bug that is happening on a web server.
One of the methods triggers the creation of a small file (3kb), in a folder. The folder is based on the current year, month and ...
1
vote
2answers
393 views
Store file in filesystem, and its metadata to the database atomicly
I have to store many pdf/jpg/png file of max 10mb in a filesystem, and need to save their metadata on a database.
The SFTP and the DB may be on different nodes. On WS, I've a local db where I can ...
0
votes
1answer
247 views
Using A XML as a Directory File To support A file Managing Application
Second Year Software Engineering student here.
I want to make a file managing system for a C# notetaking app, every note will be represented by a file and will display a small preview of it, ...
12
votes
2answers
597 views
Does a file system “see” the storage device as a (very large) byte array?
I want to know how does a file system write to and read from a storage device.
I think this is how it works:
A file system doesn't access the storage device directly, but rather the storage device ...
1
vote
0answers
110 views
Classiest file system locations for my Linux app to write its files?
I have an application I am writing on Linux. It is a Java webapp intended to be run on Tomcat. When it initializes, my application will copy some standalone java utility programs to the host ...
3
votes
2answers
106 views
Dealing with potential failures when appending data to a file on disk
I'm designing an application that will be appending blobs to a file on disk (local filesystem) and I'm currently thinking of how to deal with consistency issues that could occur if:
The application ...
3
votes
2answers
405 views
Can file systems be designed and implemented in an OS-portable way?
Given the interfaces that major OSes (Windows, macOS/OS X/Mac OS X, Linux) provide to file systems, can file systems be designed and implemented in a way that is largely independent of OS?
I'm not at ...
1
vote
1answer
129 views
Controlling permissions for content on web server (pattern/architecture)
I’m working on a proof of concept for a personal project and am unsure how to go about handling ‘permissions’ on content that is uploaded into the application.
Problem:
In this application users ...
2
votes
6answers
436 views
What are the benefits of storing data contiguously?
I am designing an application file format which will store chunks of user data, ranging from a few bytes to a few gigabytes - median size probably in the 10MB - 30MB range.
I have the option of ...
0
votes
1answer
181 views
One row database table or JSON file
If I have data that I will only need to update very rarely (once a month), would it be a good idea to use a JSON file instead of a database table with only 1 row?
7
votes
1answer
2k views
What is the name for the non-extension part of a filename? [closed]
Given the file path:
/some/path/abc.txt
The filename is "abc.txt", and extension is "txt".
What is the "industry standard", unambiguous name for the "abc" part?
For reference, in both java's older ...
-1
votes
1answer
269 views
How do you create a Composite file in C++ [closed]
I am looking to create a "Composite file" in C++, basically a composite file is a file containing files, (examples: .docx, .jar, etc) these files can usually be renamed as .zip and opened with a .zip ...
4
votes
2answers
919 views
Storing Local Filesystem Paths in Database
I'm developing a webapp where I have sets of data stored locally on my computer and I run a tool which transforms the data and uploads it to my webapp. However, I need to be able to rerun the tool on ...
1
vote
1answer
1k views
Storing uploaded images for website
I'm developing a website (using PHP, JS, and MYSQL) which allow user to upload images. My requirements are as below:
User is able to upload 1 or multiple images at a time.
Website is able to save ...
3
votes
3answers
694 views
Why would anyone want to build a file system for windows? [closed]
I saw an ad on StackOverflow today for a project called WinFsp.
The site mentions the following features:
Allows for easy development of file systems in user mode. There are no restrictions on what ...
3
votes
1answer
514 views
VBA Outlook: quickly find subfolder
I have the following structure in my Outlook Public Folders.
-Public Folders
--1001_RandomProject
--1002_AnotherProject
--1003_Yetanotherproject
...
and so on, basically there's a couple of thousand(...
3
votes
1answer
648 views
What is the most sensible design for making files available for download from a URL?
This is what I need to do, in a nutshell:
Generate Excel spreadsheet files (programmatically).
Store these .xlsx files in a location where they can be accessed by users later. These files need to be ...
4
votes
3answers
722 views
What's the point of hidden files?
What is the point of hidden files? In Microsoft Windows they exist, in Mac OS X they exist and in Linux they exist. It seems to me that it just makes detecting malware more difficult. The only upside ...
1
vote
2answers
189 views
How are non-folder files called? [closed]
Maybe it looks like a weird question, but what term should be attributed inside the code for files that are not folders to differentiate them from folders?
If I need write 2 functions isFolder() and ...
1
vote
2answers
184 views
Can version control systems use the filesystem log to capture changes?
I was trying to find a "perfect" syncing program between a network share and a local folder, when I realised that it's probably impossible to do it right unless all the filesystem operations were ...
22
votes
4answers
4k views
Why is the Git .git/objects/ folder subdivided in many SHA-prefix folders?
Git internally stores objects (Blobs, trees) in the .git/objects/ folder. Each object can be referenced by a SHA1 hash that is computed from the contents of the object.
However, Objects are not ...
1
vote
2answers
1k views
Avoid data manipulation by user
I have a C# program (could be any programming language) that saves data to a file on a memory (hard disk, USB drive, etc.). The program uses this data for monitoring its operation time, but it could ...
0
votes
1answer
75 views
What is a simple way to let a user select a folder from a tree?
I have a Python-Flask app in which users can place files into a folder.
As of now the directory structure is something like:
/app
/storage
/templates
.
.
.
server.py
The user ...
6
votes
1answer
4k views
Efficient data structure to implement fake file system
I want to implement a data structure that will hold the paths of directories, sort of fake file system.
Input:- I have a text configuration file containing the paths as follows
...
C:/temp1
C:/...
0
votes
2answers
199 views
Is renaming an 'alias' for moving?
Is it true to say (on Windows and Unix\Linux\OS X) that renaming a file or directory is just an alias for moving?
e.g.
Are there any side effects to either which are not present on the other?
Does '...
1
vote
2answers
234 views
Why do filesystems read and write in blocks?
I read that file systems usually access data in blocks, whose size is integral multiple of disk block size. Why don't they read individual disk blocks?
0
votes
1answer
234 views
Best way to centralise a php project (except bitbucket)
The structure is as follows:
A: this is the live/production website
B: this is the staging, which is a copy of live
C: this is the testing environment for designers
2 developer and 2 designers. Only ...
0
votes
1answer
303 views
Efficient way to map changes in a filesystem hierarchy
I'm currently working on a project that will enable file searching based on metadata found in the file.
It'll be comprised of 2 parts:
a filesystem crawler that passively scans for changes and ...
0
votes
1answer
47 views
Static analysis for finding capitalisation / case inconsistencies in file names
TLDR; I'm looking for ideas on how to flag code containing file names/paths that have inconsistent capitalisation with the actual file/directory.
Situation
I am migrating a significant code base ...
0
votes
4answers
190 views
Why is there little use of filesharing as compression (outside of libraries)?
Recently I was looking for a program that will run as a daemon and find files that have the same size/type, check if they're the same, then make both a hard link to a single copy if they are. And I ...
2
votes
1answer
601 views
Data structure well suited for duplicate entries
I'm in the process of getting to know (modern) filesystems. As part of it, I came across log structured filesystems that also handle allocations in a log structured way. I wonder how they handle ...
3
votes
1answer
345 views
Filesystem superblocks and their backup copies
I'd like to understand how (modern) filesystems are implemented and having trouble to fully understand superblocks and their backups. I reference ext4 and btrfs, but the questions may also apply to ...
-2
votes
2answers
137 views
file quantity limit in a directory on a linux file server and why?
What is a good limit to use on the quantity of files in a directory, and why?
EDIT:
Why shouldn't someone create a system that puts hundreds of thousands of files in the same directory?
Why I ask:
...
11
votes
5answers
22k views
Is it safe to convert Windows file paths to Unix file paths with a simple replace?
So for example say I had it so that all of my files will be transferred from a windows machine to a unix machine as such: C:\test\myFile.txt to {somewhere}/test/myFile.txt (drive letter is irrelevant ...
3
votes
1answer
481 views
How to store the file names, start offset and length while avoiding the issue of self imposed limits (lookup table) or having to scan the entire file?
I am attempting to learn more about C and it's descendants(C++ mainly). I have decided that I would like to create a "file system" of sorts. Not a particularly advanced one mind you but something to ...
8
votes
7answers
3k views
Why can we not insert into files without the additional writes? (I neither mean append, nor over-write)
This occurs as a programming language independent problem to me.
I have a file with the content
aaabddd
When I want to insert C behind b then my code needs to rewrite ddd to get
aaabCddd
Why can ...
2
votes
2answers
2k views
Why must directories be empty before being deleted?
As far as I know, deleting a non empty directory could work the same way as deleting an empty directory: by removing the pointer to the directory's metadata there would be no pointers to the items it ...
1
vote
0answers
618 views
What database structure is suitable for tracking File audits?
I need to track permissions and access requests to a file server in a database. I'm given the full path of the folder and am considering parsing the path (splitting on the "/" character) and creating ...
3
votes
2answers
582 views
Why is it not standard to implement abstraction layers for the file system?
I have been taught to access databases through abstraction layers. I was wondering why it is not also standard practice to access the file system through an abstraction layer? It seems to me unit ...
7
votes
5answers
27k views
Is it wise to store a big lump of json on a database row
I have this project which stores product details from amazon into the database.
Just to give you an idea on how big it is:
[{"title":"Genetic Engineering (Opposing Viewpoints)","short_title":"...
8
votes
4answers
808 views
What exactly does it mean that storing “large blobs in the database reduces performance”?
To someone who knows database internals this may be an easy question, but can someone explain in a clear way why storing large blobs (say 400 MB movies) in the database is supposed to decrease ...
7
votes
3answers
5k views
How many threads should access the file system at the same time?
We have a module in an application which stores data in multiple files and multilevel directories and access them from multiple threads (both reads and writes). The directory structure is based on a ...
4
votes
2answers
9k views
Finding duplicate files? [duplicate]
I am going to be developing a program that detects duplicate files and I was wondering what the best/fastest method would be to do this? I am more interested in what the best hash algorithm would be ...
