Questions tagged [file-handling]
File handling refers to the set of tools, functions, and libraries to work with files and file handles. Creating, writing, appending, moving, and deleting files fall in this domain.
163
questions
-3
votes
0
answers
36
views
Re-architecturing our software component [closed]
We have a single Java component which reads messages from multiple sources, REST, kafka, files and ingest them and saves them into cassandra. The component also hands over the messages to processing ...
- 95
4
votes
3
answers
427
views
Checking if a file exists before writing. Always avoid, or sensible with the right use case?
There are few reliable absolutes in this world. One I have relied on is the idea that checking if a file exists before doing something with it just creates an unwanted race condition. Meaning between ...
- 97.7k
0
votes
1
answer
57
views
Selecting the endianness of data in files generated by an embedded system
I have an embedded system running on a little-endian Cortex-M3. This system is able to accept packets from the network and reply to them. Now, I would like the system to start generating files. These ...
- 2,101
0
votes
1
answer
69
views
Regrouping several application files into one
I am working on a C++ application which lets users work on projects.
Each project consists of several files that should not be known / edited manually by the user.
For that we currently use a folder ...
- 1,020
1
vote
1
answer
76
views
How best to keep Excel Workbook field in sync across class instances
I know almost nothing about threads, synchronization, ...
That being said, I am working on a record handler, that hits Excel sheet to pull/push data. There exist, in my code base, a BaseRecordHandler, ...
- 216
-1
votes
2
answers
390
views
Dealing with multiple application instances
I'm developing an application (Java & JavaFX) that writes/reads data (a file). The problem is I don't want to restrict user to run only one instance (of my app) at a time, as I really can't think ...
- 33
0
votes
2
answers
87
views
Transferring data from 1 web app into another offline
I have a web app that will be used locally on 2 different site. The program is a Vue web app made with node.js and express, using MySQL database. Due to limitation (no internet in site 1), there is no ...
- 19
0
votes
1
answer
303
views
How does resuming downloads work?
I have seen download managers manage to resume HTTP downloads from where they stopped. How are they able to resume downloads? Is it related to the Accept-Range Header? Or are there other mechanisms ...
- 109
0
votes
1
answer
302
views
Efficient way to separate text file to header, tail lines and the leftover in between
I'm looking for a more efficient algorithm for the following problem:
The input of the algorithm is a text file and two non-negative integer numbers - the number of headers and the number of tails.
...
- 111
0
votes
1
answer
459
views
Constantly writing a JSON file [closed]
I need to write continuously into a JSON file in C++. The format and structure are defined as requirement. The data is produced in the process itself and needs to be stored multiple times per second. ...
- 27
0
votes
0
answers
31
views
Sync image files from network with mariadb-Database
i have as a mechanical engineering student in my practice semester to create a database that includes about 100k+ paths to images in our network. Every, lets say week, i have to synchronize all ...
- 1
1
vote
0
answers
63
views
Inter-Process Communication: Stream data from single writer process to multiple reader processes
tl;dr - I want to stream text data from one writer process to multiple reader processes. I'm thinking of using a file to achieve this. Is it a good idea?
Using a file would avoid having to maintain ...
- 19
-3
votes
1
answer
116
views
Serving large file (dozens of GB) to client from a server-side query to the Azure SQL database?
I have an SQL database in Azure that is ~4.5 TB / 150 million rows worth of data. Users will need to query this database for large sets of data. Right now, I am having users submit their requests ...
- 1
1
vote
0
answers
186
views
How to (simply) architecture a way to ingest multiple types of large files, process them, and send data in chunks to web services?
Note: All of this would be in AWS
Hi everyone,
What would you guys suggest for building something that:
Takes in several different input file types (ex: csv, json, jsonl,
xml, .gz, ...)
That can be ...
user384015
4
votes
1
answer
1k
views
Is there a standardised way of adding custom metadata to image and other filetypes?
What's a good way of storing and referencing files of different types (csv, pdf, jpg, html), each of which needs my own custom metadata? The metadata stores details of the content and how it was ...
- 637
1
vote
0
answers
57
views
Streaming File Storage in Distributed (and Containerized) Systems
I'm trying to implement a system that allows a user to upload files over HTTP, saves the file to object storage as well as any metadata surrounding the file that already exists to a NoSQL database (i....
6
votes
4
answers
471
views
First time having users - how to deal with backwards compatibility?
I'm reaching out with the following situation:
I'm the author of an application that saves a file format to disk.
There are real users using the application in their workflows.
Frequently, I want to ...
- 171
0
votes
1
answer
91
views
Best technique for sending multiple files in a single JSON post with Vue/javascript?
Using Vue/Electron to post to rails/Postgres backend.
Is there a common design pattern/approach to posting multiple audio file objects BUT in a single post request?
My current process (which works, ...
0
votes
1
answer
2k
views
How to determine ideal chunk size for file writing?
I have a class that is responsible for writing formatted binary objects to a file on a network drive with a 100Mbps bandwidth.
Each time an object is created (via lets call it w.addObj()) it formats ...
- 1,017
0
votes
1
answer
294
views
designing a secure/encrypted file storage for a multi-tenant saas app with local and cloud storage in mind
as the title suggest, I need to handle the tenant files in a multi-tenant saas app. fairly common scenario I believe and should have been solved by the gurus already.
right now the app is in infancy ...
- 143
-4
votes
1
answer
565
views
Better way to process huge files
I have a huge csv file and I have to process that file and do some data manipulation. Right now I'm reading/loading the file in buffer and then doing the data processing work. I find this approach ...
- 101
3
votes
5
answers
384
views
Applying function to file line by line or read entirely into structure first?
I've often found myself with the need to develop tools that process large files over a network and perform an operation to every element in that file. An element may be an individual line or an object ...
- 1,017
0
votes
1
answer
63
views
Test logging library
I've been designing and developing a very scalable logging library for a while.
The main goal of this library is pretty simple. Like many others projects, a simple goal does not mean a simple way the ...
- 111
-2
votes
1
answer
1k
views
Generate large Excel files and response from API
I need to generate a large Excel file (something around 50 megs) and send response to another API which will provide it to the front end for a download option.
My question is if it will be better to ...
- 329
1
vote
1
answer
263
views
How is a software able to read a network file faster than it appears to be possible?
Note this is a general, conceptual question about performance optimization. motivated by the following real-world case.
I have a file on a Windows network drive that has a 100Mbps limt; it is a ...
- 1,017
0
votes
1
answer
77
views
database implementation, how to organize files
I try to build a very small object database, not for production only to learn a bit in my spare time.
I've got a data structure that saves my data in blocks of a specific size und combine them when ...
- 129
2
votes
0
answers
61
views
What is the most efficient way to find filesystem changes in percentage?
I am planning to write a program that will produce digest logs for file changes in a particular directory. The idea is that whenever a file is deleted in a particular directory, it will show up in the ...
- 21
0
votes
1
answer
25
views
Set per-process open file limit in code or in system configuration?
I am working on an internal application that is maintained by the same people that operate the servers the application runs on. We have noticed, while transitioning from Windows to Linux, that some ...
- 369
3
votes
1
answer
1k
views
Design of file I/O -> processing -> file I/O system
I'm in the process of designing a new desktop application which is very different from other stuff I did before, and so I'll be happy if I could be pointed towards the right direction regarding the ...
- 141
0
votes
1
answer
1k
views
Getting file format by checking file header [closed]
I'm writing a program, part of which consists of determining if a given file is a PNG. Knowing that a file doesn't have to be named with its respective filename extension to be of a certain type, I ...
1
vote
1
answer
272
views
How to access a user's local file from a web application?
I am working on a web application which allows users to review pdf documents. These documents are submitted from another public facing website. A typical workflow involves:
A document is uploaded on ...
- 127
0
votes
1
answer
1k
views
How should I separate File Parse Classes?
I want to understand different ways to object design classes. I have three different classes. Generally I am creating a file parser.
The classes:
CustomerData - which shows text Data Model.
A File ...
- 351
2
votes
0
answers
73
views
How to write disk benchmarking software?
I am coding a disk (HDD/SSD) benchmarking software. I have a few questions on how a disk should be benchmarked (regardless of programming language used).
This is my pseudo code for testing sequential ...
- 175
0
votes
3
answers
161
views
How do I deal with file downloads?
I'm currently working on a script (in PHP, but I'm not sure if that's relevant here) which allows users to set up a simple file-sharing system on their server. I'm stuck when trying to figure out how ...
- 220
0
votes
1
answer
122
views
Creating data files inside a package
I'd like to deploy my code to a web server as a package. There Python is subprocessed and scripts are executed. These scripts use data files. But in case a file is needed and it doesn't exist, it ...
- 337
0
votes
2
answers
98
views
Class design : Decide between file and database for storage during runtime [closed]
Based on the demo or license version of my application, I am supposed to either save the data to a file (xml) at a specific location or a relational database. What should be my approach for this case ...
- 21
6
votes
2
answers
3k
views
How to deal with file events with microservice that exposes a REST API
A bit of context.
The company where I work is starting to transition to a modular architecture (something similar to microservices) for our main product, currently monolithic. The first step is to ...
1
vote
1
answer
200
views
How to scale a file based process
I'm looking for a method to solve the following problem:
Assume there is a spool directory with a lot of files that need to be processed. There is a process A that is constantly writing new files to ...
- 127
7
votes
2
answers
1k
views
What is constraining cross-platform asynchronous file I/O?
Looking at a range of cross-platform languages, libraries and GUI toolkits, I often notice a conspicuous absence of support for asynchronous file I/O. This seems like too much of a common factor to be ...
- 1,575
0
votes
1
answer
1k
views
An elegant way to detect the end of an asynchronous recursive file search?
Given a recursive subroutine in single threaded environment which starts numerous asynchronous I/O operations and registers callback functions for each of them. This callbacks will be called on the ...
- 111
-4
votes
2
answers
373
views
Why aren't directories for files created automatically? [closed]
One of the most annoying boneheaded bugs I run into is trying to create a new file in a directory that does not exist. (At least on Windows/NTFS, not sure about other OS/FS combinations.)
Here's how ...
-5
votes
1
answer
192
views
Right Click SaveAs and Open C# Application [closed]
I am wanting to develop a C# application that would allow me to right click on a file and choose an option within the context menu to perform a save as with a date time stamp and then open the newly ...
2
votes
1
answer
8k
views
Storing images in base64 vs binary, in cloud
I'm developing a photo sharing website. Currently, the way I wrote it, a user uploads a photo (jpeg), which is read in as a Base64URL (using FileReader). The backend (Django) uploads this file to ...
- 299
9
votes
3
answers
528
views
Writing at the start of a file something you only know at the end
Background: I'm writing micro controller C code to write an EBML file. EBML is like a binary XML with nested elements, but instead of start and end tags, there is a start ID, length, and then the data....
- 201
3
votes
2
answers
785
views
REST API with files as resources?
I'm storing files as REST API resources, sending and receiving them as base64 strings, along with some other metadata. What's the best way to handle GET for the collection? (e.g GET .../api/files)
...
- 33
23
votes
1
answer
31k
views
Why would anyone use multipart/form-data for mixed data and file transfers?
I'm working in C# and doing some communication between 2 apps I'm writing. I have come to like the Web API and JSON. Now I am at the point where I am writing a routine to send a record between the two ...
- 375
11
votes
1
answer
31k
views
Base64 in Json: Is it a Good Idea for Rest API?
I am developing a Rest API, and I am asking myself:
Is it a good Idea to put base64 encoded data in Json, e.g. for file uploads? What if the base64 contains some of {,},: characters and breaks the ...
2
votes
1
answer
337
views
How is persistence-ignorance relevant to a file format–centered application?
I have a small side-project of about 3,000 sloc. I'm experimenting with building a DDD-like domain model into it, so I have more experience with this style for work. The project is centered around ...
- 123
0
votes
1
answer
1k
views
What's the best way to do horizontally scalable file uploads?
In general, how do websites handle file uploads when they scale horizontally? Obviously if you have a load balancer going to 10 application servers, and each application server just saves an upload ...
- 13
3
votes
2
answers
324
views
Dealing with potential failures when appending data to a file on disk
I'm designing an application that will be appending blobs to a file on disk (local filesystem) and I'm currently thinking of how to deal with consistency issues that could occur if:
The application ...
- 537