Questions tagged [file-storage]
The file-storage tag has no usage guidance.
54
questions
-1
votes
1
answer
39
views
File system for abstracting S3 object storage
We're looking for a scalable way to implement a file browser feature backed by a robust object storage system for the files. We were looking at an S3 compatible service, such as MinIO for this. Since ...
- 1,129
0
votes
1
answer
93
views
How could creating symlinks/aliases to files on S3 work, while still taking advantage of CDN optimizations?
I am trying to imagine a website/system where only 1 copy of a file is ever stored, but multiple users get their own custom link to the file, and it uses the CDN features. For this question we can ...
- 2,447
-3
votes
2
answers
233
views
Why does most software break or refuse to continue immediately if the folder where their data is stored is not available? [closed]
Most software creates a directory (usually in ~/Library or ~/Library/Application Support in MacOS) to store user preferences, browser history, etc. Most software attempts to create their data ...
user403833
8
votes
7
answers
6k
views
What is the size of the number 65535 in bytes? [closed]
As I got to know there are 256 possible combinations to get for 1 byte. If I understand it correctly, it should mean that you can display any number out of numbers 0-255 and this very number would use ...
- 151
-1
votes
1
answer
135
views
Best way to store and fetch image bytes?
I have a case in my startup where I need to fetch images in bytes as fast as possible.
At the moment I'm storing the images in Azure Storage then I fetch them on the run and cache them into my ...
- 107
0
votes
1
answer
253
views
Large file uploading in chunks
How to upload large(mb/gb) of video files from client to app server?
The solution as of now I know is pretty simple and widely used is to break file in chunks at at client side and send http post ...
- 81
3
votes
2
answers
3k
views
Share files between microservices
We have a service (lets say FileService) that provides API to store/download files into various stores (S3/local file etc.,)
We have a scenario where one microservice (Service A) writes a file to S3 ...
- 39
1
vote
0
answers
57
views
Streaming File Storage in Distributed (and Containerized) Systems
I'm trying to implement a system that allows a user to upload files over HTTP, saves the file to object storage as well as any metadata surrounding the file that already exists to a NoSQL database (i....
3
votes
2
answers
750
views
In DDD, is it worth it to define a Bounded Context for file access?
I'm designing an editor as a desktop application that opens, saves and saves as documents from files, something very common in fact.
I already have Bounded Contexts for my business rules.
Naively, I ...
- 31
0
votes
1
answer
294
views
designing a secure/encrypted file storage for a multi-tenant saas app with local and cloud storage in mind
as the title suggest, I need to handle the tenant files in a multi-tenant saas app. fairly common scenario I believe and should have been solved by the gurus already.
right now the app is in infancy ...
- 143
1
vote
0
answers
20
views
Supplying Kubernetes pods with data from a persistent volume and writing back
To keep it simple, I have this service that stores data in a persistent volume. There are pods that are created by an API which use a subset of the stored data to perform an operation and generate new ...
- 111
0
votes
2
answers
68
views
How to store legacy source texts snapshots with FTS (full text search) all through their history?
we have a legacy project, on Windows. It goes on from 1990-s.
Until very recently it was not backed by any version control, today it is moved into Git.
The question is all the prior snapshots. As of ...
- 117
2
votes
0
answers
117
views
Is it possible to implement insert file operation in modern extent-based filesystems?
The problem was already discussed here. But there was not consensus on this topic.
I have some thoughts on how insert operation can be implemented for some popular file systems. If FS has extent-...
1
vote
2
answers
679
views
Is it safe to store application files to user's temp directory?
I'm developing a Node.js library that collects data whenever user runs tests. I would like to persist the data on local disk so that after the test run, user can run a command accessing that data. The ...
- 29
0
votes
1
answer
494
views
Recommended Long Term Data Storage Format
I have an application (written in Go) with a lot of GPS location information. Each record consists of a latitude and longitude plus somewhere around 50 fields of various pieces of meta data, some of ...
- 239
0
votes
1
answer
237
views
Image upload workflow azure
I'm working on a new project. It is a smartphone application made with react-native. In the app, you can create "cards" which are having a title, a description and a photo, taken by the user's phone.
...
0
votes
1
answer
85
views
How do database storage engines handle new columns?
I really don't know which StackExchange site this belongs on. It didn't seem to match the "on-topic" list of DBA.SE or SO.SE ... so please let me know if I should move it. I also couldn't find a ...
- 474
0
votes
1
answer
276
views
When is the point where a JSON text file storage is not enough?
I guess it was Robert C. Martin who campaigned against using SQL in one of his apps, where he was just fine without DB and SQL initially. Later, thanks to the abstraction a DB implementation could be ...
- 212
1
vote
0
answers
36
views
Best way to spread/shard file location on a network UNC
We are developing a system whereby documents/files will be stored on a specialized Content Server and uploaded via a client.
However we want to be able to develop this so if we need to, we can swap ...
- 111
2
votes
3
answers
106
views
Does a replicated distributed file system minimise the need for durability?
I've been investigating various distributed file systems, like Gluster, Ceph, Moose and Lizard.
I'm also familiar with various key/value store type systems, some of which do not perform any system ...
- 644
2
votes
0
answers
73
views
How to write disk benchmarking software?
I am coding a disk (HDD/SSD) benchmarking software. I have a few questions on how a disk should be benchmarked (regardless of programming language used).
This is my pseudo code for testing sequential ...
- 175
25
votes
2
answers
8k
views
Why can FAT16 not store more than 2 GB?
All the sites I go to look for informations on FAT16 just declaratively state that it can not allocate more than 2 GB. OK. Great. I believe you. But how do you come to that conclusion (other than just ...
- 361
4
votes
4
answers
3k
views
Is storing files of up to 50MB in size in a database for use by multiple servers a reasonable idea? Example inside
I'm in the process of designing a server responsible for serving files that are between 10MB and 50MB in size.
Initially we will run two instances of the server (lets call them fs1 and fs2), with ...
- 273
-1
votes
1
answer
237
views
How do different file types generally store data?
I am working on a project, for which I want to create my own file format to store large amounts of data. I am trying to decide how that file format should be built to be as efficient as possible.
The ...
- 101
0
votes
1
answer
284
views
Using A XML as a Directory File To support A file Managing Application
Second Year Software Engineering student here.
I want to make a file managing system for a C# notetaking app, every note will be represented by a file and will display a small preview of it, ...
- 139
0
votes
2
answers
1k
views
What is the use cases for Azure file storage?
I'm using Azure file storage and there was a huge number of files uploaded. A single file will be 250kb in size and there will be 7000 files in a directory.
So when the user request I need to get the ...
- 113
1
vote
1
answer
2k
views
Big amount of text: database vs file
I have a web application with medium-high traffic where users can save designs made by fabricjs, so each saved design is a json with many parameters.
So basically two days after launch users have ...
- 119
2
votes
1
answer
962
views
What is the most sensible design for making files available for download from a URL?
This is what I need to do, in a nutshell:
Generate Excel spreadsheet files (programmatically).
Store these .xlsx files in a location where they can be accessed by users later. These files need to be ...
4
votes
3
answers
2k
views
Design pattern for large amounts of overflowing data?
Our current queues publish messages that consumed by 3rd party services with rate limits. Currently the messages are retried with exponential back-off. However there could be cases where data is ...
- 317
1
vote
1
answer
60
views
How to deal with dual servers and temporary files
Our application is hosted on a dual-server environment - when one server is overloaded with work, a secondary server will take its place in processing certain actions. For most purposes, this isn't a ...
- 1,512
0
votes
1
answer
166
views
What is a simple way to let a user select a folder from a tree?
I have a Python-Flask app in which users can place files into a folder.
As of now the directory structure is something like:
/app
/storage
/templates
.
.
.
server.py
The user ...
- 101
4
votes
2
answers
1k
views
Why do filesystems read and write in blocks?
I read that file systems usually access data in blocks, whose size is integral multiple of disk block size. Why don't they read individual disk blocks?
- 53
1
vote
3
answers
630
views
Why do we need a format for binary executable files
When binary files (i.e. executables) are saved they usually have a format (e.g. ELF or .out) where we have a header containing pointers to where data or code is stored inside the file. But why don't ...
- 199
4
votes
3
answers
5k
views
storing and retrieving millions of documents using c#
I am working on an integration project, where my “app/web service” will sit in the middle serving documents.
Basically, a request is sent with the document id as part of the query string, I check if ...
- 141
2
votes
1
answer
532
views
Binary data storage scheme (saving user-uploaded files)
In our app we are currently saving binary data to the database (terrible, I know; but this is legacy stuff and it wasn't my decision). We're trying to migrate this data out to an external device and I'...
- 195
5
votes
3
answers
2k
views
Storing object-graphs with class-evolution in Java with transformation (long time archiving)
Abstract
A common problem is to store objects (with graph), and to load them back. This is easy as long the stored object representation matches the executing code. But as time goes by, requirements ...
- 161
1
vote
2
answers
5k
views
All images in one folder for a website?
When creating websites, is it better to have a lot of images in one folder, or have them separated?
It's something that has always been in my mind but I never researched enough about it, and googling ...
- 129
0
votes
2
answers
170
views
Interaction between programs
I am writing an interactive program in which it takes speech input from the user for a specific list of commands.
The list of commands will be stored locally in a graph and based on the usage their ...
- 101
1
vote
3
answers
2k
views
File storage for a blog component: database or filesystem?
I'm going to develop a fairly basic re-usable blog component, with simple CRUD operations using ASP.Net.
Which method of storing blog posts would be best suited to the situation in terms of ...
- 113
0
votes
1
answer
560
views
Are Python sockets suitable for file synchronization?
I'm working in an organisation with limited funds. They can't afford a business account on Dropbox. However, they would find it useful to have all their files synchronized on local machines.
I've ...
- 213
3
votes
5
answers
3k
views
Best practice to sync long paths to filesystems where path length is limited?
I wrote a sync tool to synchronize folders/files from Alfresco to Windows.
PROBLEM:
In Alfresco, /files/can/have/very/very/very/long/filepaths/like/this.txt
The Windows API prevents me from creating ...
- 1,062
3
votes
1
answer
7k
views
How are Java ByteBuffer's limit and position variable's updated?
There are two scenarios: writing and reading
Writing:
Whenever I write something to the ByteBuffer by calling its put(byte[]) method the position variable is incremented as: current position + size of ...
- 411
5
votes
1
answer
249
views
Why use binary files to stack up different versions on DMSs?
I've used both Liferay and Alfresco trying to use them as the Document Management System for an intranet.
I noticed the following:
They use the file system and the database to store files
They use a ...
- 153
2
votes
2
answers
138
views
What is the best strategy for transforming unicode strings into filenames?
I have a bunch (thousands) of resources in an RDF/XML file. I am writing a certain subset of the resources to files -- one file for each, and I'm using the resource's title property as the file name. ...
- 2,903
53
votes
3
answers
16k
views
What is the benefit of writing to a temp location, And then copying it to the intended destination?
I am writing an application that works with satellite images, and my boss asked me to look at some of the commercial applications, and see how they behave. I found a strange behavior and then as I was ...
- 2,484
1
vote
1
answer
276
views
Flexible cloud file storage for a web.py app?
I'm creating a web app using web.py (although I may later rewrite it for Tornado) which involves a lot of file manipulation. One example, the app will have a git-style 'commit' operation, in which ...
- 117
4
votes
3
answers
1k
views
For performance critical situations is storing file metadata in a database better?
As per the title of this question, for extremely performance critical situations, is storing a file's metadata (e.g. location, size, download on, etc) in a database going to allow for better ...
- 11.3k
2
votes
3
answers
229
views
Do you do any validation on directories when recursively deleting them?
How do you recursively delete paths of the data that your app created?
The deleting process itself is trivial. The question is, do you do extra validation? And if so, what kind?
From one point of ...
- 6,968
4
votes
2
answers
1k
views
How do other people keep code current accross multiple machines? (something other than DropBox) [duplicate]
I am a hardware/infrastructure guy going back to school to get my degree (FINALLY...) and am taking some low level intro programming classes. On top of the JAVA I am working on at school, I am also ...
- 181
2
votes
2
answers
994
views
What is the best way to store files when using winforms and web?
I have to develop an application (in C#) that has to work with files. The application consists of two versions: a web version and a windows version. Therefore, the files must be stored on a place ...
- 189