Questions tagged [file-formats]
The file-formats tag has no usage guidance.
24 questions
48
votes
4
answers
42k
views
Should UTF-8 CSV files contain a BOM (byte order mark)?
Our line-of-business software allows the user to save certain data as CSV. Since there are a lot of different formats (all called "CSV") in use in the wild, we are tying to decide what the &...
- 9,838
21
votes
2
answers
11k
views
What is the current status of software support for JPEG-2000?
The general recommendation to record original scanned images used to be "use TIFF". But programmers need evolution of format for "evolution of software", and I need to evolve my system to change from ...
- 757
13
votes
8
answers
4k
views
In what configuration file format do regular expressions not need escaping?
I want a configuration file for a .NET program.
This file is to configure pairs of regular expressions.
The regular expressions belong within a hierarchy of sections.
Section1
SubsectionA
...
- 3,417
9
votes
3
answers
2k
views
SQLite database as a data interchange format?
Is it sensible/reasonable to consider using an SQLite database file as a data interchange format? Are there any obvious gothchas or reasons not to do this?
Example. Say I have a master SQL Server ...
- 365
6
votes
2
answers
259
views
Why GUID Partition Table data lack the sector size?
The GPT (GUID Partition Table) is the most accepted modern standard for partitioning of a data storage device. Its unit for the offset and size of a partition is so-called sector*. Is it a well-...
- 169
3
votes
4
answers
1k
views
Would you use Byte Arrays or XML if size was not an issue?
In my basic understanding of ByteArrays the benefit is that it is smaller in file size.
The down size of a ByteArray is that for any given format you have to know the about the file format to get ...
- 1,229
2
votes
3
answers
1k
views
What metadada to save and whether to include it inside the binary file?
I'm wondering if there are any general guidelines or best practices regarding when to split data into a metadata format, as oppose to directly embedding it within the data. (Specific example below).
...
- 135
1
vote
2
answers
465
views
How a standard video file is structured under the hood
I am new to looking into video file formats, and am wondering what it would take to parse one. To do that, I first would need to understand what the format looks like, so that's what this question is ...
- 2,661
1
vote
3
answers
215
views
Using source code instead of XML/JSON or other custom serialization schemes and binary file formats
For a while now I have been toying with the idea of using source code as a file storage format.
My question: How to support format version changes? (loading of older files with structural differences)
...
- 127
1
vote
6
answers
1k
views
What are the benefits of storing data contiguously?
I am designing an application file format which will store chunks of user data, ranging from a few bytes to a few gigabytes - median size probably in the 10MB - 30MB range.
I have the option of ...
- 423
1
vote
3
answers
530
views
What is the simplest translation file format for a user to edit and for a program to parse? [closed]
I have just started developing a small web app for Arabic speakers, and I will need to ask a co-worker (not a programmer) to help me translate all of the labels and documentation into Arabic at some ...
- 369
1
vote
2
answers
541
views
Design Pattern to extract arbitrary field from arbitrary file format
Lets say I have multiple file types:
.json, .csv ... etc
These file types come in different formats:
Second json structure
Extra column added to csv
etc.
I need to extract fields from these files;...
- 132
1
vote
1
answer
2k
views
Fast storage format for huge point clouds (fast read/write)
I am working with huge point clouds (1 billion points). I need to process and display them but because of the size of the data, I can't have it all in memory at the same time.
First of all, I am not ...
- 111
1
vote
0
answers
250
views
Best practice to handle input and output with several csv-like files
I am currently writing an application with the following structure:
input: the equivalent of an excel workbook, i.e. a few tables with different headers and a few scalar values. They represent ...
- 19
0
votes
1
answer
159
views
Selecting the endianness of data in files generated by an embedded system
I have an embedded system running on a little-endian Cortex-M3. This system is able to accept packets from the network and reply to them. Now, I would like the system to start generating files. These ...
- 2,109
0
votes
1
answer
846
views
best way to export binary data with additional text attributes to a stream
I would like to create a command line API in a Java application to export a binary blob to stdout. I would additionally like to export certain attributes about the blob, ideally in a non-binary format....
- 1,861
0
votes
1
answer
699
views
Best file formats for ML training
Hi I'm bulding an ML pipeline with PyTorch to support various tasks and am looking for some advice on efficient ways to store processed data.
The main frame work was 3 layers [data prep] -> [data ...
- 111
0
votes
1
answer
1k
views
Getting file format by checking file header [closed]
I'm writing a program, part of which consists of determining if a given file is a PNG. Knowing that a file doesn't have to be named with its respective filename extension to be of a certain type, I ...
- 111
0
votes
2
answers
96
views
What are formats to store geographic maps for a robot to travel point A to point B
Very beginner. Need to compute route from A to B for a robot. i would like to know how to store area map and compute the route. I can compute shortest path etc using algorithms e.g. dijkstra. What ...
- 47
-1
votes
1
answer
292
views
How do different file types generally store data?
I am working on a project, for which I want to create my own file format to store large amounts of data. I am trying to decide how that file format should be built to be as efficient as possible.
The ...
- 101
-1
votes
2
answers
854
views
How to handle multiple versions of binary file format
I'm writing a reader for the custom binary file format used in one game. The format has multiple versions for example:
struct FileHeader
{
ushort version;
uint fieldX; // version 1
uint ...
- 1,559
-1
votes
1
answer
147
views
Why do HTML and JS feel slow on smartphones? What could be changed to make them faster? [closed]
HTML rendering and javascript engines are very optimized, so why do they feel slow on smartphones?
It seems that HTML's low performance justified the creation of mobiles "apps" which are ...
- 2,280
-2
votes
1
answer
95
views
Choosing a MagicNumber or Signature for a Binary File Format
I'm defining some binary formats to encode some information and dump it to a file.
I want to place a MagicNumber/Signature at the beginning of files encoded in my format to be able to tell them apart ...
- 141
-3
votes
1
answer
97
views
Is it possible to extend gifs to support audio? [closed]
Can the file format gif be extended to support audio as well?
I was looking into gifs that support audio and the main gif sites with "audio gifs" are just looping videos. The problem is how ...
- 101