Questions tagged [file-systems]

techniques to organize and store files with their data on a computer.

Filter by
Sorted by
Tagged with
1 vote
0 answers
101 views

How to store an encrypted filesystem in a single file on disk on multiple platforms (maybe not portably), or alternatives

I'm in the very conceptual phase of designing an open source password manager that provides distributed Vaults that can be simultaneously accessed and managed from multiple devices with the promise of ...
Nick Williams's user avatar
2 votes
1 answer
89 views

What are some architectures and designs I could use to optimize the performance of file text data lookups?

I would like to write a program that essentially handles text data and metadata of files locally on a machine's filesystem. There is no need for any network activity. I am working with large ...
the_endian's user avatar
  • 1,152
0 votes
0 answers
29 views

Enabling file editing on a file stored on a server through custom routes and POST/GET Requests

I am creating a platform which allows multiple users to edit a file simultaneously and I was wondering if using a web server with routes could be the correct architecture for this goal. I am trying to ...
Fabrizio's user avatar
  • 115
2 votes
3 answers
371 views

File I/O - How HDD or SSD works with OS file system? [closed]

I've studied how data transfer with secondary storage(HDD or SSD) works. Would you mind to check that my understanding is correct? File system block is basic read/write logical unit which is used in ...
obanadingyo's user avatar
0 votes
1 answer
151 views

How handle many files effectively?

I'm looking for a way to work with a huge amount (~10^9) of files (all sizes) in .NET using unspecified file system (NTFS, BTRFS...). What I have done up to now is to store them in evenly in a folder-...
Andreas Zita's user avatar
-1 votes
1 answer
54 views

File system for abstracting S3 object storage

We're looking for a scalable way to implement a file browser feature backed by a robust object storage system for the files. We were looking at an S3 compatible service, such as MinIO for this. Since ...
dumazy's user avatar
  • 1,219
0 votes
2 answers
292 views

Building a program that truly deletes everything

We all know that if we delete a file, the operating system is recycling it but doesn't actually delete it. It just removes it from the directory indexes, and until the data is needed and overwritten, ...
VJZ's user avatar
  • 127
0 votes
2 answers
86 views

Delete filesystem data associated with database record by delete hook, or internally in service logic?

Let's have many rows in the SQL database where every record has an image property that holds the path to file in the filesystem. Many database libraries have the ability to set-up hooks that are ...
Baterka's user avatar
  • 165
0 votes
1 answer
387 views

Large file uploading in chunks

How to upload large(mb/gb) of video files from client to app server? The solution as of now I know is pretty simple and widely used is to break file in chunks at at client side and send http post ...
stkUser's user avatar
  • 81
1 vote
1 answer
361 views

Custom File System Index/Cache - How to save index

I've got an extremely oniony(deep) folder structure which contains Appx 1,000,000 text-based files on a network share. Using windows search is extremely slow and unreliable. I've created some text ...
GisMofx's user avatar
  • 379
5 votes
1 answer
1k views

Why use strong checksums to detect random errors in a filesystem like btrfs?

Btrfs supports crc32c, xxhash, sha256 and blake2b as checksums when storing and reading files. crc32c and xxhash are designed to detect random errors while sha256 and blake2 are considered ...
kjdf's user avatar
  • 61
1 vote
0 answers
31 views

(Installable) File System Driver, Shell Extension Handler or another approach?

I have an application that does some document handling as part of its functionality. I would like to implement a "Drive" emulator that would allow users to navigate and manage documents as ...
Digital Camel's user avatar
-3 votes
1 answer
166 views

Small scale document management system architecture / patterns

Im usually working with line of business desktop software. Mostly based on a single database. Pretty often one of the requirements is to keep track of some files. Or the only way to implement a ...
Zuldaan's user avatar
  • 99
-4 votes
1 answer
235 views

When to use dot files in Git repositories?

My Git repository contains dot files required by my tooling: .dockerignore .gitignore .pre-commit-config.yaml I'm using Azure Pipelines for CI/CD, which is typically defined in azure-pipelines.yml, ...
Krzysztof Czelusniak's user avatar
0 votes
1 answer
72 views

PHP: What code should be removed to its own helper class and where should such classes be located in the filesystem?

The following is an example requesting an explanation for one specific file in one specific filesystem, not helper classes generally. I have configured a LEPP stack on a CentOS server. The server ...
Muckee's user avatar
  • 111
0 votes
1 answer
3k views

fake filesystem for unit tests

I am writing an application in c++ and Qt5. It would be very convenient for me to create a virtual file system so I can unit test code working on files. I have found that in Qt4 there was a ...
Michał Walenciak's user avatar
0 votes
1 answer
70 views

Test logging library

I've been designing and developing a very scalable logging library for a while. The main goal of this library is pretty simple. Like many others projects, a simple goal does not mean a simple way the ...
VRoxa's user avatar
  • 111
6 votes
1 answer
549 views

Solving file system dependency with database storage

I have a software project A which makes API calls to a third-party software B that is heavily based on data stored on the file system. Also, those software and file systems are distributed on servers ...
Gabriel Robaina's user avatar
2 votes
0 answers
118 views

Is it possible to implement insert file operation in modern extent-based filesystems?

The problem was already discussed here. But there was not consensus on this topic. I have some thoughts on how insert operation can be implemented for some popular file systems. If FS has extent-...
Anton Astafiev's user avatar
0 votes
1 answer
108 views

I'm building a file management app and need to know the best approach for dealing with folders

So you open up file explorer on your computer and can create folders, add files, rename, etc. I'm doing the same thing in a web application that interacts with the file system on the server. I'm ...
user1447679's user avatar
-3 votes
2 answers
300 views

Hosting a file system in production?

Sorry if this is broad. But I'm confused on how to scale a file system. I'm currently working on the server side of a webapp with Node(Express)js, its file system module, and mongoDB. I'm storing ...
boogie man's user avatar
2 votes
0 answers
121 views

What is the most efficient way to find filesystem changes in percentage?

I am planning to write a program that will produce digest logs for file changes in a particular directory. The idea is that whenever a file is deleted in a particular directory, it will show up in the ...
Michael John's user avatar
1 vote
1 answer
376 views

How to access a user's local file from a web application?

I am working on a web application which allows users to review pdf documents. These documents are submitted from another public facing website. A typical workflow involves: A document is uploaded on ...
software_writer's user avatar
0 votes
3 answers
319 views

How to append a chunk of fixed size data to a file and make sure this chunk doesn't get fragmented on disk?

So i want to understand how DBMS implementation works To give an example : MySQL implements each tables with its own pages, which are 16KB so each table is a file, and is a multiple of 16KB, ...
OneAndOnly's user avatar
2 votes
3 answers
115 views

Does a replicated distributed file system minimise the need for durability?

I've been investigating various distributed file systems, like Gluster, Ceph, Moose and Lizard. I'm also familiar with various key/value store type systems, some of which do not perform any system ...
magnus's user avatar
  • 674
4 votes
4 answers
1k views

Why do disks write data in chunks of page size?

In my understanding, even if i want to overwrite a byte in middle of a file, OS and/or disk will read the content of the size of page, modify one byte and then write the contents back. What is the ...
Ashish Negi's user avatar
0 votes
2 answers
720 views

Referencing custom Python modules and data files

I'd want to deploy my Python code and relevant static files such that only a copy of a folder is needed. That is, all the paths inside are relative. The release is to a web server, which calls scripts ...
Felix's user avatar
  • 357
1 vote
1 answer
1k views

Chat application - write to file and then save in database

I have followed this approach that is described here to implement a simple chat application: https://code.tutsplus.com/tutorials/how-to-create-a-simple-web-based-chat-application--net-5931 I'm ...
Bryan's user avatar
  • 119
0 votes
2 answers
237 views

Detect when a file is created (on a webserver) and ready for use in one of many directories

I have an odd, intermittent bug that is happening on a web server. One of the methods triggers the creation of a small file (3kb), in a folder. The folder is based on the current year, month and ...
Matt's user avatar
  • 253
3 votes
2 answers
2k views

Store file in filesystem, and its metadata to the database atomicly

I have to store many pdf/jpg/png file of max 10mb in a filesystem, and need to save their metadata on a database. The SFTP and the DB may be on different nodes. On WS, I've a local db where I can ...
Federico Ponzi's user avatar
0 votes
1 answer
300 views

Using A XML as a Directory File To support A file Managing Application

Second Year Software Engineering student here. I want to make a file managing system for a C# notetaking app, every note will be represented by a file and will display a small preview of it, ...
Xeorge Xeorge's user avatar
12 votes
2 answers
713 views

Does a file system "see" the storage device as a (very large) byte array?

I want to know how does a file system write to and read from a storage device. I think this is how it works: A file system doesn't access the storage device directly, but rather the storage device ...
joseph_m's user avatar
  • 335
1 vote
0 answers
115 views

Classiest file system locations for my Linux app to write its files?

I have an application I am writing on Linux. It is a Java webapp intended to be run on Tomcat. When it initializes, my application will copy some standalone java utility programs to the host ...
DWoldrich's user avatar
  • 119
4 votes
2 answers
410 views

Dealing with potential failures when appending data to a file on disk

I'm designing an application that will be appending blobs to a file on disk (local filesystem) and I'm currently thinking of how to deal with consistency issues that could occur if: The application ...
D. Jurcau's user avatar
  • 547
3 votes
4 answers
1k views

Can file systems be designed and implemented in an OS-portable way?

Given the interfaces that major OSes (Windows, macOS/OS X/Mac OS X, Linux) provide to file systems, can file systems be designed and implemented in a way that is largely independent of OS? I'm not at ...
Praxeolitic's user avatar
  • 1,634
1 vote
1 answer
231 views

Controlling permissions for content on web server (pattern/architecture)

I’m working on a proof of concept for a personal project and am unsure how to go about handling ‘permissions’ on content that is uploaded into the application. Problem: In this application users ...
zaza's user avatar
  • 119
1 vote
6 answers
1k views

What are the benefits of storing data contiguously?

I am designing an application file format which will store chunks of user data, ranging from a few bytes to a few gigabytes - median size probably in the 10MB - 30MB range. I have the option of ...
jl6's user avatar
  • 423
0 votes
1 answer
262 views

One row database table or JSON file

If I have data that I will only need to update very rarely (once a month), would it be a good idea to use a JSON file instead of a database table with only 1 row?
Goose's user avatar
  • 1,878
25 votes
1 answer
7k views

What is the name for the non-extension part of a filename? [closed]

Given the file path: /some/path/abc.txt The filename is "abc.txt", and extension is "txt". What is the "industry standard", unambiguous name for the "abc" part? For reference, in both java's older ...
Bohemian's user avatar
  • 1,986
-1 votes
1 answer
345 views

How do you create a Composite file in C++ [closed]

I am looking to create a "Composite file" in C++, basically a composite file is a file containing files, (examples: .docx, .jar, etc) these files can usually be renamed as .zip and opened with a .zip ...
Jesse McDonald's user avatar
5 votes
2 answers
3k views

Storing Local Filesystem Paths in Database

I'm developing a webapp where I have sets of data stored locally on my computer and I run a tool which transforms the data and uploads it to my webapp. However, I need to be able to rerun the tool on ...
CC Inc's user avatar
  • 159
1 vote
1 answer
3k views

Storing uploaded images for website

I'm developing a website (using PHP, JS, and MYSQL) which allow user to upload images. My requirements are as below: User is able to upload 1 or multiple images at a time. Website is able to save ...
4 Leave Cover's user avatar
6 votes
3 answers
1k views

Why would anyone want to build a file system for windows? [closed]

I saw an ad on StackOverflow today for a project called WinFsp. The site mentions the following features: Allows for easy development of file systems in user mode. There are no restrictions on what ...
MarioDS's user avatar
  • 429
3 votes
1 answer
1k views

VBA Outlook: quickly find subfolder

I have the following structure in my Outlook Public Folders. -Public Folders --1001_RandomProject --1002_AnotherProject --1003_Yetanotherproject ... and so on, basically there's a couple of thousand(...
Sam's user avatar
  • 143
2 votes
1 answer
994 views

What is the most sensible design for making files available for download from a URL?

This is what I need to do, in a nutshell: Generate Excel spreadsheet files (programmatically). Store these .xlsx files in a location where they can be accessed by users later. These files need to be ...
B. Clay Shannon-B. Crow Raven's user avatar
6 votes
3 answers
2k views

What's the point of hidden files?

What is the point of hidden files? In Microsoft Windows they exist, in Mac OS X they exist and in Linux they exist. It seems to me that it just makes detecting malware more difficult. The only upside ...
Kellen Stuart's user avatar
2 votes
2 answers
566 views

How are non-folder files called? [closed]

Maybe it looks like a weird question, but what term should be attributed inside the code for files that are not folders to differentiate them from folders? If I need write 2 functions isFolder() and ...
kitty's user avatar
  • 49
1 vote
2 answers
327 views

Can version control systems use the filesystem log to capture changes?

I was trying to find a "perfect" syncing program between a network share and a local folder, when I realised that it's probably impossible to do it right unless all the filesystem operations were ...
Milind R's user avatar
  • 155
26 votes
4 answers
6k views

Why is the Git .git/objects/ folder subdivided in many SHA-prefix folders?

Git internally stores objects (Blobs, trees) in the .git/objects/ folder. Each object can be referenced by a SHA1 hash that is computed from the contents of the object. However, Objects are not ...
Qqwy's user avatar
  • 4,789
1 vote
2 answers
2k views

Avoid data manipulation by user

I have a C# program (could be any programming language) that saves data to a file on a memory (hard disk, USB drive, etc.). The program uses this data for monitoring its operation time, but it could ...
Tobias Knauss's user avatar