Questions tagged [file-storage]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
2 answers
156 views

Creating associated resources before creating the main one in a REST setting

I'm not sure how common of a scenario this is, but what are some approaches to creating associated entities before you have the actual main entity they're supposed to be linked to? (basically having ...
string_loginUsername's user avatar
2 votes
1 answer
833 views

Saving and reading files in user specific application data in C or C++

Most useful or reasonably complex applications need to save data such as user settings or saved games or browser history. I have been working on applications and games in C or C++ but I am not sure ...
user16217248's user avatar
  • 1,167
-1 votes
1 answer
54 views

File system for abstracting S3 object storage

We're looking for a scalable way to implement a file browser feature backed by a robust object storage system for the files. We were looking at an S3 compatible service, such as MinIO for this. Since ...
dumazy's user avatar
  • 1,219
0 votes
1 answer
2k views

How could creating symlinks/aliases to files on S3 work, while still taking advantage of CDN optimizations?

I am trying to imagine a website/system where only 1 copy of a file is ever stored, but multiple users get their own custom link to the file, and it uses the CDN features. For this question we can ...
Lance's user avatar
  • 2,587
-2 votes
2 answers
278 views

Why does most software break or refuse to continue immediately if the folder where their data is stored is not available? [closed]

Most software creates a directory (usually in ~/Library or ~/Library/Application Support in MacOS) to store user preferences, browser history, etc. Most software attempts to create their data ...
user avatar
7 votes
7 answers
7k views

What is the size of the number 65535 in bytes? [closed]

As I got to know there are 256 possible combinations to get for 1 byte. If I understand it correctly, it should mean that you can display any number out of numbers 0-255 and this very number would use ...
treesarecool's user avatar
-1 votes
1 answer
383 views

Best way to store and fetch image bytes?

I have a case in my startup where I need to fetch images in bytes as fast as possible. At the moment I'm storing the images in Azure Storage then I fetch them on the run and cache them into my ...
Faisal's user avatar
  • 107
0 votes
1 answer
386 views

Large file uploading in chunks

How to upload large(mb/gb) of video files from client to app server? The solution as of now I know is pretty simple and widely used is to break file in chunks at at client side and send http post ...
stkUser's user avatar
  • 81
5 votes
2 answers
5k views

Share files between microservices

We have a service (lets say FileService) that provides API to store/download files into various stores (S3/local file etc.,) We have a scenario where one microservice (Service A) writes a file to S3 ...
Neelima's user avatar
  • 59
1 vote
0 answers
84 views

Streaming File Storage in Distributed (and Containerized) Systems

I'm trying to implement a system that allows a user to upload files over HTTP, saves the file to object storage as well as any metadata surrounding the file that already exists to a NoSQL database (i....
foxtrotuniform6969's user avatar
3 votes
2 answers
1k views

In DDD, is it worth it to define a Bounded Context for file access?

I'm designing an editor as a desktop application that opens, saves and saves as documents from files, something very common in fact. I already have Bounded Contexts for my business rules. Naively, I ...
Loïc Morvan's user avatar
0 votes
1 answer
337 views

designing a secure/encrypted file storage for a multi-tenant saas app with local and cloud storage in mind

as the title suggest, I need to handle the tenant files in a multi-tenant saas app. fairly common scenario I believe and should have been solved by the gurus already. right now the app is in infancy ...
Waku-2's user avatar
  • 143
1 vote
0 answers
29 views

Supplying Kubernetes pods with data from a persistent volume and writing back

To keep it simple, I have this service that stores data in a persistent volume. There are pods that are created by an API which use a subset of the stored data to perform an operation and generate new ...
Jurgy's user avatar
  • 111
0 votes
2 answers
69 views

How to store legacy source texts snapshots with FTS (full text search) all through their history?

we have a legacy project, on Windows. It goes on from 1990-s. Until very recently it was not backed by any version control, today it is moved into Git. The question is all the prior snapshots. As of ...
Arioch's user avatar
  • 117
2 votes
0 answers
118 views

Is it possible to implement insert file operation in modern extent-based filesystems?

The problem was already discussed here. But there was not consensus on this topic. I have some thoughts on how insert operation can be implemented for some popular file systems. If FS has extent-...
Anton Astafiev's user avatar
1 vote
2 answers
1k views

Is it safe to store application files to user's temp directory?

I'm developing a Node.js library that collects data whenever user runs tests. I would like to persist the data on local disk so that after the test run, user can run a command accessing that data. The ...
saaskis's user avatar
  • 29
0 votes
1 answer
668 views

Recommended Long Term Data Storage Format

I have an application (written in Go) with a lot of GPS location information. Each record consists of a latitude and longitude plus somewhere around 50 fields of various pieces of meta data, some of ...
Brad Peabody's user avatar
0 votes
1 answer
247 views

Image upload workflow azure

I'm working on a new project. It is a smartphone application made with react-native. In the app, you can create "cards" which are having a title, a description and a photo, taken by the user's phone. ...
RomanTenger's user avatar
0 votes
1 answer
120 views

How do database storage engines handle new columns?

I really don't know which StackExchange site this belongs on. It didn't seem to match the "on-topic" list of DBA.SE or SO.SE ... so please let me know if I should move it. I also couldn't find a ...
Simon Whitehead's user avatar
0 votes
1 answer
389 views

When is the point where a JSON text file storage is not enough?

I guess it was Robert C. Martin who campaigned against using SQL in one of his apps, where he was just fine without DB and SQL initially. Later, thanks to the abstraction a DB implementation could be ...
andras's user avatar
  • 212
1 vote
0 answers
37 views

Best way to spread/shard file location on a network UNC

We are developing a system whereby documents/files will be stored on a specialized Content Server and uploaded via a client. However we want to be able to develop this so if we need to, we can swap ...
user183872's user avatar
2 votes
3 answers
115 views

Does a replicated distributed file system minimise the need for durability?

I've been investigating various distributed file systems, like Gluster, Ceph, Moose and Lizard. I'm also familiar with various key/value store type systems, some of which do not perform any system ...
magnus's user avatar
  • 674
2 votes
0 answers
80 views

How to write disk benchmarking software?

I am coding a disk (HDD/SSD) benchmarking software. I have a few questions on how a disk should be benchmarked (regardless of programming language used). This is my pseudo code for testing sequential ...
Canol Gökel's user avatar
25 votes
2 answers
8k views

Why can FAT16 not store more than 2 GB?

All the sites I go to look for informations on FAT16 just declaratively state that it can not allocate more than 2 GB. OK. Great. I believe you. But how do you come to that conclusion (other than just ...
SangoProductions's user avatar
4 votes
4 answers
3k views

Is storing files of up to 50MB in size in a database for use by multiple servers a reasonable idea? Example inside

I'm in the process of designing a server responsible for serving files that are between 10MB and 50MB in size. Initially we will run two instances of the server (lets call them fs1 and fs2), with ...
mal's user avatar
  • 273
-1 votes
1 answer
274 views

How do different file types generally store data?

I am working on a project, for which I want to create my own file format to store large amounts of data. I am trying to decide how that file format should be built to be as efficient as possible. The ...
zomnombom's user avatar
  • 101
0 votes
1 answer
300 views

Using A XML as a Directory File To support A file Managing Application

Second Year Software Engineering student here. I want to make a file managing system for a C# notetaking app, every note will be represented by a file and will display a small preview of it, ...
Xeorge Xeorge's user avatar
0 votes
2 answers
1k views

What is the use cases for Azure file storage?

I'm using Azure file storage and there was a huge number of files uploaded. A single file will be 250kb in size and there will be 7000 files in a directory. So when the user request I need to get the ...
rebornx's user avatar
  • 113
1 vote
1 answer
2k views

Big amount of text: database vs file

I have a web application with medium-high traffic where users can save designs made by fabricjs, so each saved design is a json with many parameters. So basically two days after launch users have ...
Msam85's user avatar
  • 119
2 votes
1 answer
994 views

What is the most sensible design for making files available for download from a URL?

This is what I need to do, in a nutshell: Generate Excel spreadsheet files (programmatically). Store these .xlsx files in a location where they can be accessed by users later. These files need to be ...
B. Clay Shannon-B. Crow Raven's user avatar
4 votes
3 answers
3k views

Design pattern for large amounts of overflowing data?

Our current queues publish messages that consumed by 3rd party services with rate limits. Currently the messages are retried with exponential back-off. However there could be cases where data is ...
kreek's user avatar
  • 317
1 vote
1 answer
62 views

How to deal with dual servers and temporary files

Our application is hosted on a dual-server environment - when one server is overloaded with work, a secondary server will take its place in processing certain actions. For most purposes, this isn't a ...
Zibbobz's user avatar
  • 1,552
0 votes
1 answer
169 views

What is a simple way to let a user select a folder from a tree?

I have a Python-Flask app in which users can place files into a folder. As of now the directory structure is something like: /app /storage /templates . . . server.py The user ...
rohithpr's user avatar
  • 101
5 votes
2 answers
2k views

Why do filesystems read and write in blocks?

I read that file systems usually access data in blocks, whose size is integral multiple of disk block size. Why don't they read individual disk blocks?
Ishan's user avatar
  • 63
1 vote
3 answers
676 views

Why do we need a format for binary executable files

When binary files (i.e. executables) are saved they usually have a format (e.g. ELF or .out) where we have a header containing pointers to where data or code is stored inside the file. But why don't ...
sarthak's user avatar
  • 199
4 votes
3 answers
5k views

storing and retrieving millions of documents using c#

I am working on an integration project, where my “app/web service” will sit in the middle serving documents. Basically, a request is sent with the document id as part of the query string, I check if ...
MicrosoftDevX's user avatar
2 votes
1 answer
538 views

Binary data storage scheme (saving user-uploaded files)

In our app we are currently saving binary data to the database (terrible, I know; but this is legacy stuff and it wasn't my decision). We're trying to migrate this data out to an external device and I'...
Vivin Paliath's user avatar
5 votes
3 answers
2k views

Storing object-graphs with class-evolution in Java with transformation (long time archiving)

Abstract A common problem is to store objects (with graph), and to load them back. This is easy as long the stored object representation matches the executing code. But as time goes by, requirements ...
Dag's user avatar
  • 161
1 vote
2 answers
5k views

All images in one folder for a website?

When creating websites, is it better to have a lot of images in one folder, or have them separated? It's something that has always been in my mind but I never researched enough about it, and googling ...
aleation's user avatar
  • 129
0 votes
2 answers
171 views

Interaction between programs

I am writing an interactive program in which it takes speech input from the user for a specific list of commands. The list of commands will be stored locally in a graph and based on the usage their ...
quick-'s user avatar
  • 101
1 vote
3 answers
2k views

File storage for a blog component: database or filesystem?

I'm going to develop a fairly basic re-usable blog component, with simple CRUD operations using ASP.Net. Which method of storing blog posts would be best suited to the situation in terms of ...
DGibbs's user avatar
  • 113
0 votes
1 answer
573 views

Are Python sockets suitable for file synchronization?

I'm working in an organisation with limited funds. They can't afford a business account on Dropbox. However, they would find it useful to have all their files synchronized on local machines. I've ...
Sheldon's user avatar
  • 213
3 votes
5 answers
3k views

Best practice to sync long paths to filesystems where path length is limited?

I wrote a sync tool to synchronize folders/files from Alfresco to Windows. PROBLEM: In Alfresco, /files/can/have/very/very/very/long/filepaths/like/this.txt The Windows API prevents me from creating ...
Nicolas Raoul's user avatar
3 votes
1 answer
7k views

How are Java ByteBuffer's limit and position variable's updated?

There are two scenarios: writing and reading Writing: Whenever I write something to the ByteBuffer by calling its put(byte[]) method the position variable is incremented as: current position + size of ...
Dummy Derp's user avatar
5 votes
1 answer
255 views

Why use binary files to stack up different versions on DMSs?

I've used both Liferay and Alfresco trying to use them as the Document Management System for an intranet. I noticed the following: They use the file system and the database to store files They use a ...
edgarator's user avatar
  • 153
2 votes
2 answers
138 views

What is the best strategy for transforming unicode strings into filenames?

I have a bunch (thousands) of resources in an RDF/XML file. I am writing a certain subset of the resources to files -- one file for each, and I'm using the resource's title property as the file name. ...
David Cowden's user avatar
  • 2,903
53 votes
3 answers
16k views

What is the benefit of writing to a temp location, And then copying it to the intended destination?

I am writing an application that works with satellite images, and my boss asked me to look at some of the commercial applications, and see how they behave. I found a strange behavior and then as I was ...
Devdatta Tengshe's user avatar
1 vote
1 answer
280 views

Flexible cloud file storage for a web.py app?

I'm creating a web app using web.py (although I may later rewrite it for Tornado) which involves a lot of file manipulation. One example, the app will have a git-style 'commit' operation, in which ...
benwad's user avatar
  • 117
4 votes
3 answers
1k views

For performance critical situations is storing file metadata in a database better?

As per the title of this question, for extremely performance critical situations, is storing a file's metadata (e.g. location, size, download on, etc) in a database going to allow for better ...
rjzii's user avatar
  • 11.3k
2 votes
3 answers
243 views

Do you do any validation on directories when recursively deleting them?

How do you recursively delete paths of the data that your app created? The deleting process itself is trivial. The question is, do you do extra validation? And if so, what kind? From one point of ...
Coder's user avatar
  • 6,968