Questions tagged [file-handling]

File handling refers to the set of tools, functions, and libraries to work with files and file handles. Creating, writing, appending, moving, and deleting files fall in this domain.

Filter by
Sorted by
Tagged with
0
votes
1answer
54 views

Is there a standardised way of adding custom metadata to image and other filetypes?

What's a good way of storing and referencing files of different types (csv, pdf, jpg, html), each of which needs my own custom metadata? The metadata stores details of the content and how it was ...
2
votes
0answers
39 views

Streaming File Storage in Distributed (and Containerized) Systems

I'm trying to implement a system that allows a user to upload files over HTTP, saves the file to object storage as well as any metadata surrounding the file that already exists to a NoSQL database (i....
6
votes
4answers
405 views

First time having users - how to deal with backwards compatibility?

I'm reaching out with the following situation: I'm the author of an application that saves a file format to disk. There are real users using the application in their workflows. Frequently, I want to ...
0
votes
1answer
56 views

Best technique for sending multiple files in a single JSON post with Vue/javascript?

Using Vue/Electron to post to rails/Postgres backend. Is there a common design pattern/approach to posting multiple audio file objects BUT in a single post request? My current process (which works, ...
0
votes
1answer
81 views

How to determine ideal chunk size for file writing?

I have a class that is responsible for writing formatted binary objects to a file on a network drive with a 100Mbps bandwidth. Each time an object is created (via lets call it w.addObj()) it formats ...
0
votes
1answer
55 views

designing a secure/encrypted file storage for a multi-tenant saas app with local and cloud storage in mind

as the title suggest, I need to handle the tenant files in a multi-tenant saas app. fairly common scenario I believe and should have been solved by the gurus already. right now the app is in infancy ...
-4
votes
1answer
66 views

Better way to process huge files

I have a huge csv file and I have to process that file and do some data manipulation. Right now I'm reading/loading the file in buffer and then doing the data processing work. I find this approach ...
3
votes
5answers
244 views

Applying function to file line by line or read entirely into structure first?

I've often found myself with the need to develop tools that process large files over a network and perform an operation to every element in that file. An element may be an individual line or an object ...
0
votes
1answer
41 views

Test logging library

I've been designing and developing a very scalable logging library for a while. The main goal of this library is pretty simple. Like many others projects, a simple goal does not mean a simple way the ...
-2
votes
1answer
159 views

Generate large Excel files and response from API

I need to generate a large Excel file (something around 50 megs) and send response to another API which will provide it to the front end for a download option. My question is if it will be better to ...
2
votes
1answer
209 views

How is a software able to read a network file faster than it appears to be possible?

Note this is a general, conceptual question about performance optimization. motivated by the following real-world case. I have a file on a Windows network drive that has a 100Mbps limt; it is a ...
0
votes
1answer
54 views

database implementation, how to organize files

I try to build a very small object database, not for production only to learn a bit in my spare time. I've got a data structure that saves my data in blocks of a specific size und combine them when ...
2
votes
0answers
40 views

What is the most efficient way to find filesystem changes in percentage?

I am planning to write a program that will produce digest logs for file changes in a particular directory. The idea is that whenever a file is deleted in a particular directory, it will show up in the ...
0
votes
1answer
21 views

Set per-process open file limit in code or in system configuration?

I am working on an internal application that is maintained by the same people that operate the servers the application runs on. We have noticed, while transitioning from Windows to Linux, that some ...
3
votes
1answer
645 views

Design of file I/O -> processing -> file I/O system

I'm in the process of designing a new desktop application which is very different from other stuff I did before, and so I'll be happy if I could be pointed towards the right direction regarding the ...
0
votes
1answer
752 views

Getting file format by checking file header [closed]

I'm writing a program, part of which consists of determining if a given file is a PNG. Knowing that a file doesn't have to be named with its respective filename extension to be of a certain type, I ...
1
vote
1answer
82 views

How to access a file stored locally on server?

I am working on a web application which allows users to review pdf documents. These documents are submitted from another public facing website. A typical workflow involves: A document is uploaded on ...
0
votes
1answer
607 views

How should I separate File Parse Classes?

I want to understand different ways to object design classes. I have three different classes. Generally I am creating a file parser. The classes: CustomerData - which shows text Data Model. A File ...
2
votes
0answers
63 views

How to write disk benchmarking software?

I am coding a disk (HDD/SSD) benchmarking software. I have a few questions on how a disk should be benchmarked (regardless of programming language used). This is my pseudo code for testing sequential ...
0
votes
3answers
151 views

How do I deal with file downloads?

I'm currently working on a script (in PHP, but I'm not sure if that's relevant here) which allows users to set up a simple file-sharing system on their server. I'm stuck when trying to figure out how ...
0
votes
1answer
85 views

Creating data files inside a package

I'd like to deploy my code to a web server as a package. There Python is subprocessed and scripts are executed. These scripts use data files. But in case a file is needed and it doesn't exist, it ...
0
votes
2answers
93 views

Class design : Decide between file and database for storage during runtime [closed]

Based on the demo or license version of my application, I am supposed to either save the data to a file (xml) at a specific location or a relational database. What should be my approach for this case ...
6
votes
2answers
2k views

How to deal with file events with microservice that exposes a REST API

A bit of context. The company where I work is starting to transition to a modular architecture (something similar to microservices) for our main product, currently monolithic. The first step is to ...
3
votes
0answers
566 views

File upload sync for offline desktop application. (Electron)

We are making an offline desktop application in Electron that needs to sync when the user goes online. Sqlite is used as the database when offline and it gets synced to postgres on the server when ...
1
vote
1answer
132 views

How to scale a file based process

I'm looking for a method to solve the following problem: Assume there is a spool directory with a lot of files that need to be processed. There is a process A that is constantly writing new files to ...
5
votes
2answers
667 views

What is constraining cross-platform asynchronous file I/O?

Looking at a range of cross-platform languages, libraries and GUI toolkits, I often notice a conspicuous absence of support for asynchronous file I/O. This seems like too much of a common factor to be ...
0
votes
1answer
1k views

An elegant way to detect the end of an asynchronous recursive file search?

Given a recursive subroutine in single threaded environment which starts numerous asynchronous I/O operations and registers callback functions for each of them. This callbacks will be called on the ...
-4
votes
2answers
313 views

Why aren't directories for files created automatically? [closed]

One of the most annoying boneheaded bugs I run into is trying to create a new file in a directory that does not exist. (At least on Windows/NTFS, not sure about other OS/FS combinations.) Here's how ...
-5
votes
1answer
159 views

Right Click SaveAs and Open C# Application [closed]

I am wanting to develop a C# application that would allow me to right click on a file and choose an option within the context menu to perform a save as with a date time stamp and then open the newly ...
2
votes
1answer
6k views

Storing images in base64 vs binary, in cloud

I'm developing a photo sharing website. Currently, the way I wrote it, a user uploads a photo (jpeg), which is read in as a Base64URL (using FileReader). The backend (Django) uploads this file to ...
9
votes
3answers
500 views

Writing at the start of a file something you only know at the end

Background: I'm writing micro controller C code to write an EBML file. EBML is like a binary XML with nested elements, but instead of start and end tags, there is a start ID, length, and then the data....
3
votes
2answers
547 views

REST API with files as resources?

I'm storing files as REST API resources, sending and receiving them as base64 strings, along with some other metadata. What's the best way to handle GET for the collection? (e.g GET .../api/files) ...
17
votes
1answer
22k views

Why would anyone use multipart/form-data for mixed data and file transfers?

I'm working in C# and doing some communication between 2 apps I'm writing. I have come to like the Web API and JSON. Now I am at the point where I am writing a routine to send a record between the two ...
9
votes
1answer
22k views

Base64 in Json: Is it a Good Idea for Rest API?

I am developing a Rest API, and I am asking myself: Is it a good Idea to put base64 encoded data in Json, e.g. for file uploads? What if the base64 contains some of {,},: characters and breaks the ...
2
votes
1answer
331 views

How is persistence-ignorance relevant to a file format–centered application?

I have a small side-project of about 3,000 sloc. I'm experimenting with building a DDD-like domain model into it, so I have more experience with this style for work. The project is centered around ...
0
votes
1answer
659 views

What's the best way to do horizontally scalable file uploads?

In general, how do websites handle file uploads when they scale horizontally? Obviously if you have a load balancer going to 10 application servers, and each application server just saves an upload ...
3
votes
2answers
160 views

Dealing with potential failures when appending data to a file on disk

I'm designing an application that will be appending blobs to a file on disk (local filesystem) and I'm currently thinking of how to deal with consistency issues that could occur if: The application ...
9
votes
4answers
6k views

Is it bad practice to read large file in constructor?

So, I am trying to create an English language trie data structure implementation in C++. I have created a Trie and TrieNode class. The TrieNode class takes in its constructor a vector<string> ...
1
vote
0answers
445 views

I want to write an application to compare files, similar to WinMerge. I'm not sure how to design it, though

My current project is to implement a kind of WinMerge clone to compare two files. I am using C++ with the Win32-API (WTL, ATL, that stuff), and I already designed an UI for this. However, I am now ...
3
votes
4answers
671 views

Would you use Byte Arrays or XML if size was not an issue?

In my basic understanding of ByteArrays the benefit is that it is smaller in file size. The down size of a ByteArray is that for any given format you have to know the about the file format to get ...
10
votes
2answers
28k views

Java - Processing a large file concurrently

So at a high level my use case is as follows - I periodically (every 24 hours) get a very large file (size can vary from MBs to 10s of GBs) which I need to process within 24 hours. The ...
-1
votes
1answer
325 views

How do you create a Composite file in C++ [closed]

I am looking to create a "Composite file" in C++, basically a composite file is a file containing files, (examples: .docx, .jar, etc) these files can usually be renamed as .zip and opened with a .zip ...
1
vote
1answer
588 views

Pattern for dealing with different File Types

In my application I fetch files from the user and process it. Im to limit to a certain scope of file types and each one of these has a different processing approach. How can I design the application ...
0
votes
1answer
77 views

Modeling database to store files and search its content

I have a functionality which I must deal. The requirement of a project is to transform papers to pdf and store this files. The main functionality is that the user is being able to search words in the ...
0
votes
1answer
239 views

Optimizing a function that seems to execute really slowly when there are a lot of usernames in the file

I wrote a small program, about 10,500 lines and everything works perfectly. It is a Math game, which has is login and sign-up process and all data is stored on the user's computer in multiple text ...
1
vote
2answers
192 views

Visual Basic Writing to File with Numbered Lines

This is more of a design/user experience kind of question, so if it belongs somewhere else please let me know. I have a program that has multiple PictureBoxes: PictureBox1 PictureBox2 . . ...
0
votes
0answers
67 views

Options for Dumping Java objects to a file for import

Background: I am in the process of writing a suite of applications and have been tasked with tracking "metadata" across the app. This task requires me to process, export, then import, a large amount ...
2
votes
1answer
776 views

How can the PHP send a file to Java api?

I am currently working in a project that is to build a online judging system like codeforces. That is where someone can upload a file for a programming solution and our system will compile the code ...
0
votes
1answer
1k views

Designing a RESTful API for a file manager

I am building a file manager web UI (front end ReactJS, back end Flask). RESTful API seems to be very suitable in this situation at first sight, but I get into trouble. I need API contain: List ...
1
vote
2answers
81 views

How can I efficiently add/delete/get sorted data in Java?

I am writing a telephone directory program and need to store the entries in a CSV formatted file. However, I am having difficulty keeping the entries in order when adding. I was considering reading in ...