Last call to make your voice heard! Our 2022 Developer Survey closes in less than a week. Take survey.

Questions tagged [file-storage]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
-3 votes
2 answers
220 views

Why does most software break or refuse to continue immediately if the folder where their data is stored is not available? [closed]

Most software creates a directory (usually in ~/Library or ~/Library/Application Support in MacOS) to store user preferences, browser history, etc. Most software attempts to create their data ...
user avatar
8 votes
7 answers
5k views

What is the size of the number 65535 in bytes? [closed]

As I got to know there are 256 possible combinations to get for 1 byte. If I understand it correctly, it should mean that you can display any number out of numbers 0-255 and this very number would use ...
user avatar
-1 votes
1 answer
95 views

Best way to store and fetch image bytes?

I have a case in my startup where I need to fetch images in bytes as fast as possible. At the moment I'm storing the images in Azure Storage then I fetch them on the run and cache them into my ...
user avatar
  • 107
0 votes
1 answer
124 views

Large file uploading in chunks

How to upload large(mb/gb) of video files from client to app server? The solution as of now I know is pretty simple and widely used is to break file in chunks at at client side and send http post ...
user avatar
  • 81
2 votes
2 answers
1k views

Share files between microservices

We have a service (lets say FileService) that provides API to store/download files into various stores (S3/local file etc.,) We have a scenario where one microservice (Service A) writes a file to S3 ...
user avatar
  • 29
1 vote
0 answers
49 views

Streaming File Storage in Distributed (and Containerized) Systems

I'm trying to implement a system that allows a user to upload files over HTTP, saves the file to object storage as well as any metadata surrounding the file that already exists to a NoSQL database (i....
user avatar
3 votes
2 answers
429 views

In DDD, is it worth it to define a Bounded Context for file access?

I'm designing an editor as a desktop application that opens, saves and saves as documents from files, something very common in fact. I already have Bounded Contexts for my business rules. Naively, I ...
user avatar
0 votes
1 answer
258 views

designing a secure/encrypted file storage for a multi-tenant saas app with local and cloud storage in mind

as the title suggest, I need to handle the tenant files in a multi-tenant saas app. fairly common scenario I believe and should have been solved by the gurus already. right now the app is in infancy ...
user avatar
  • 143
1 vote
0 answers
17 views

Supplying Kubernetes pods with data from a persistent volume and writing back

To keep it simple, I have this service that stores data in a persistent volume. There are pods that are created by an API which use a subset of the stored data to perform an operation and generate new ...
user avatar
  • 111
0 votes
2 answers
65 views

How to store legacy source texts snapshots with FTS (full text search) all through their history?

we have a legacy project, on Windows. It goes on from 1990-s. Until very recently it was not backed by any version control, today it is moved into Git. The question is all the prior snapshots. As of ...
user avatar
  • 117
2 votes
0 answers
115 views

Is it possible to implement insert file operation in modern extent-based filesystems?

The problem was already discussed here. But there was not consensus on this topic. I have some thoughts on how insert operation can be implemented for some popular file systems. If FS has extent-...
user avatar
0 votes
2 answers
484 views

Is it safe to store application files to user's temp directory?

I'm developing a Node.js library that collects data whenever user runs tests. I would like to persist the data on local disk so that after the test run, user can run a command accessing that data. The ...
user avatar
  • 19
0 votes
1 answer
383 views

Recommended Long Term Data Storage Format

I have an application (written in Go) with a lot of GPS location information. Each record consists of a latitude and longitude plus somewhere around 50 fields of various pieces of meta data, some of ...
user avatar
0 votes
1 answer
234 views

Image upload workflow azure

I'm working on a new project. It is a smartphone application made with react-native. In the app, you can create "cards" which are having a title, a description and a photo, taken by the user's phone. ...
user avatar
0 votes
1 answer
75 views

How do database storage engines handle new columns?

I really don't know which StackExchange site this belongs on. It didn't seem to match the "on-topic" list of DBA.SE or SO.SE ... so please let me know if I should move it. I also couldn't find a ...
user avatar
0 votes
1 answer
191 views

When is the point where a JSON text file storage is not enough?

I guess it was Robert C. Martin who campaigned against using SQL in one of his apps, where he was just fine without DB and SQL initially. Later, thanks to the abstraction a DB implementation could be ...
user avatar
  • 212
1 vote
0 answers
32 views

Best way to spread/shard file location on a network UNC

We are developing a system whereby documents/files will be stored on a specialized Content Server and uploaded via a client. However we want to be able to develop this so if we need to, we can swap ...
user avatar
2 votes
3 answers
102 views

Does a replicated distributed file system minimise the need for durability?

I've been investigating various distributed file systems, like Gluster, Ceph, Moose and Lizard. I'm also familiar with various key/value store type systems, some of which do not perform any system ...
user avatar
  • 614
2 votes
0 answers
67 views

How to write disk benchmarking software?

I am coding a disk (HDD/SSD) benchmarking software. I have a few questions on how a disk should be benchmarked (regardless of programming language used). This is my pseudo code for testing sequential ...
user avatar
25 votes
2 answers
7k views

Why can FAT16 not store more than 2 GB?

All the sites I go to look for informations on FAT16 just declaratively state that it can not allocate more than 2 GB. OK. Great. I believe you. But how do you come to that conclusion (other than just ...
user avatar
4 votes
4 answers
2k views

Is storing files of up to 50MB in size in a database for use by multiple servers a reasonable idea? Example inside

I'm in the process of designing a server responsible for serving files that are between 10MB and 50MB in size. Initially we will run two instances of the server (lets call them fs1 and fs2), with ...
user avatar
  • 273
-1 votes
1 answer
212 views

How do different file types generally store data?

I am working on a project, for which I want to create my own file format to store large amounts of data. I am trying to decide how that file format should be built to be as efficient as possible. The ...
user avatar
  • 101
0 votes
1 answer
277 views

Using A XML as a Directory File To support A file Managing Application

Second Year Software Engineering student here. I want to make a file managing system for a C# notetaking app, every note will be represented by a file and will display a small preview of it, ...
user avatar
0 votes
2 answers
1k views

What is the use cases for Azure file storage?

I'm using Azure file storage and there was a huge number of files uploaded. A single file will be 250kb in size and there will be 7000 files in a directory. So when the user request I need to get the ...
user avatar
  • 113
1 vote
1 answer
2k views

Big amount of text: database vs file

I have a web application with medium-high traffic where users can save designs made by fabricjs, so each saved design is a json with many parameters. So basically two days after launch users have ...
user avatar
  • 119
2 votes
1 answer
938 views

What is the most sensible design for making files available for download from a URL?

This is what I need to do, in a nutshell: Generate Excel spreadsheet files (programmatically). Store these .xlsx files in a location where they can be accessed by users later. These files need to be ...
user avatar
4 votes
3 answers
2k views

Design pattern for large amounts of overflowing data?

Our current queues publish messages that consumed by 3rd party services with rate limits. Currently the messages are retried with exponential back-off. However there could be cases where data is ...
user avatar
  • 317
1 vote
1 answer
59 views

How to deal with dual servers and temporary files

Our application is hosted on a dual-server environment - when one server is overloaded with work, a secondary server will take its place in processing certain actions. For most purposes, this isn't a ...
user avatar
  • 1,492
0 votes
1 answer
155 views

What is a simple way to let a user select a folder from a tree?

I have a Python-Flask app in which users can place files into a folder. As of now the directory structure is something like: /app /storage /templates . . . server.py The user ...
user avatar
  • 101
3 votes
2 answers
1k views

Why do filesystems read and write in blocks?

I read that file systems usually access data in blocks, whose size is integral multiple of disk block size. Why don't they read individual disk blocks?
user avatar
  • 43
1 vote
3 answers
588 views

Why do we need a format for binary executable files

When binary files (i.e. executables) are saved they usually have a format (e.g. ELF or .out) where we have a header containing pointers to where data or code is stored inside the file. But why don't ...
user avatar
  • 199
4 votes
3 answers
5k views

storing and retrieving millions of documents using c#

I am working on an integration project, where my “app/web service” will sit in the middle serving documents. Basically, a request is sent with the document id as part of the query string, I check if ...
user avatar
2 votes
1 answer
532 views

Binary data storage scheme (saving user-uploaded files)

In our app we are currently saving binary data to the database (terrible, I know; but this is legacy stuff and it wasn't my decision). We're trying to migrate this data out to an external device and I'...
user avatar
4 votes
3 answers
2k views

Storing object-graphs with class-evolution in Java with transformation (long time archiving)

Abstract A common problem is to store objects (with graph), and to load them back. This is easy as long the stored object representation matches the executing code. But as time goes by, requirements ...
user avatar
  • 151
1 vote
2 answers
5k views

All images in one folder for a website?

When creating websites, is it better to have a lot of images in one folder, or have them separated? It's something that has always been in my mind but I never researched enough about it, and googling ...
user avatar
  • 129
0 votes
2 answers
168 views

Interaction between programs

I am writing an interactive program in which it takes speech input from the user for a specific list of commands. The list of commands will be stored locally in a graph and based on the usage their ...
user avatar
  • 101
2 votes
3 answers
1k views

File storage for a blog component: database or filesystem?

I'm going to develop a fairly basic re-usable blog component, with simple CRUD operations using ASP.Net. Which method of storing blog posts would be best suited to the situation in terms of ...
user avatar
  • 123
0 votes
1 answer
536 views

Are Python sockets suitable for file synchronization?

I'm working in an organisation with limited funds. They can't afford a business account on Dropbox. However, they would find it useful to have all their files synchronized on local machines. I've ...
user avatar
  • 213
3 votes
4 answers
3k views

Best practice to sync long paths to filesystems where path length is limited?

I wrote a sync tool to synchronize folders/files from Alfresco to Windows. PROBLEM: In Alfresco, /files/can/have/very/very/very/long/filepaths/like/this.txt The Windows API prevents me from creating ...
user avatar
3 votes
1 answer
7k views

How are Java ByteBuffer's limit and position variable's updated?

There are two scenarios: writing and reading Writing: Whenever I write something to the ByteBuffer by calling its put(byte[]) method the position variable is incremented as: current position + size of ...
user avatar
5 votes
1 answer
247 views

Why use binary files to stack up different versions on DMSs?

I've used both Liferay and Alfresco trying to use them as the Document Management System for an intranet. I noticed the following: They use the file system and the database to store files They use a ...
user avatar
  • 153
2 votes
2 answers
137 views

What is the best strategy for transforming unicode strings into filenames?

I have a bunch (thousands) of resources in an RDF/XML file. I am writing a certain subset of the resources to files -- one file for each, and I'm using the resource's title property as the file name. ...
user avatar
  • 2,885
52 votes
3 answers
15k views

What is the benefit of writing to a temp location, And then copying it to the intended destination?

I am writing an application that works with satellite images, and my boss asked me to look at some of the commercial applications, and see how they behave. I found a strange behavior and then as I was ...
user avatar
1 vote
1 answer
276 views

Flexible cloud file storage for a web.py app?

I'm creating a web app using web.py (although I may later rewrite it for Tornado) which involves a lot of file manipulation. One example, the app will have a git-style 'commit' operation, in which ...
user avatar
  • 117
4 votes
3 answers
1k views

For performance critical situations is storing file metadata in a database better?

As per the title of this question, for extremely performance critical situations, is storing a file's metadata (e.g. location, size, download on, etc) in a database going to allow for better ...
user avatar
  • 11.2k
2 votes
3 answers
212 views

Do you do any validation on directories when recursively deleting them?

How do you recursively delete paths of the data that your app created? The deleting process itself is trivial. The question is, do you do extra validation? And if so, what kind? From one point of ...
user avatar
  • 6,970
4 votes
2 answers
1k views

How do other people keep code current accross multiple machines? (something other than DropBox) [duplicate]

I am a hardware/infrastructure guy going back to school to get my degree (FINALLY...) and am taking some low level intro programming classes. On top of the JAVA I am working on at school, I am also ...
user avatar
  • 181
2 votes
2 answers
963 views

What is the best way to store files when using winforms and web?

I have to develop an application (in C#) that has to work with files. The application consists of two versions: a web version and a windows version. Therefore, the files must be stored on a place ...
user avatar
  • 189
1 vote
2 answers
819 views

Storing documents in DMS

I need your opinions and suggestions about storing documents in a DMS system. I think the DMS should save its own copies of the documents, not their original path on the disk. So, the DMS should have ...
user avatar
  • 113
2 votes
3 answers
171 views

best practice: setting up several IDE / frameworks (at file/dir level) [closed]

I don't know how to exactly search for this topic, so if there are a lot of answers to that, please just provide a link :) I'm getting a new laptop in a few weeks and am thinking about setting up a ...
user avatar
  • 31