62
votes
Should UTF-8 CSV files contain a BOM (byte order mark)?
Not for UTF-8, but see the various caveats in the comments.
It's unnecessary (UTF-8 has no byte order) unlike UTF-16/32 and not recommended in the Unicode standard. It's also quite rare to see UTF-8 ...
- 1,950
16
votes
Accepted
In what configuration file format do regular expressions not need escaping?
CDATA sections in XML should do.
Here's a stackoverflow post about it:
https://stackoverflow.com/questions/2784183/what-does-cdata-in-xml-mean
I remember it took me a while to understand how to use ...
- 18.5k
13
votes
In what configuration file format do regular expressions not need escaping?
This is indeed an interesting question, as commonly the requirements for config file formats are somewhat different, so it's understandable that available formats don't really support this requirement....
- 17.9k
11
votes
Should UTF-8 CSV files contain a BOM (byte order mark)?
There still is no widespread convention AFAIK, though certainly UTF-8 is now generally accepted.
The BOM is an awful artifact:
It is invisible (zero-width space).
Some software might break on the ...
- 2,221
10
votes
In what configuration file format do regular expressions not need escaping?
Consider TOML
It handles two different forms of raw strings:
regex = '<\i\c*\s*>'
OR
regex2 = '''I [dw]on't need \d{2} apples'''
- 28.9k
7
votes
Accepted
How do different file types generally store data?
The reason there are many different file formats is that there are many different goals for the way data is formatted. Some of these are in opposition to each other and some are orthogonal to each ...
- 28.9k
7
votes
Accepted
SQLite database as a data interchange format?
It happens to be possible to use SQLite databases as a data interchange format, but it's not a particularly good solution. Why?
You are looking for a compact data exchange format. In contrast, SQLite'...
- 135k
7
votes
Accepted
How a standard video file is structured under the hood
Parsing a complete digital film is an immensely complex task. Because you mostly ask about WebM – a container format – I’ll concentrate on that.
You always start with individual streams containing ...
- 1,163
6
votes
In what configuration file format do regular expressions not need escaping?
NestedText is a configuration file format that makes a point of not requiring any escaping or quoting, which makes it very good for applications like this:
# regex examples from:
# https://support....
- 169
6
votes
How a standard video file is structured under the hood
Video container formats are typically made up of a series of content blocks. A block typically consists of a few marker bytes (important for finding the next block if you get incomplete data while ...
- 15.5k
5
votes
Would you use Byte Arrays or XML if size was not an issue?
It is certainly possible that the JPEG and PNG people might have used XML-based headers if file size were not an issue. Using text would make it more obvious what all the fields mean and so forth.
...
- 12k
5
votes
Using source code instead of XML/JSON or other custom serialization schemes and binary file formats
With the question clarified as to handling differences in the structure of configuration from one version to the next, you lose a lot of potential solutions by using a code as configuration process. ...
- 46.3k
4
votes
In what configuration file format do regular expressions not need escaping?
Roll your own
Seems a simple enough format; just write your own custom parser to deserialize from a plain text file (perhaps just like your first example) into your object model. This would require ...
- 151
4
votes
Why GUID Partition Table data lack the sector size?
A software layer which can read and write the GPT has to use low-level ATA/ATAPI commands for it. Hence, it can simply ask the device for its logical sector size using the ATA command "IDENTIFY ...
- 214k
3
votes
Is it possible to extend gifs to support audio?
In theory, you can add any feature to any format.
In practice, GIF images are so widespread and the new feature, while vaguely related, is so far from the core functionality of the standard format ...
- 110k
3
votes
In what configuration file format do regular expressions not need escaping?
Tab-separated value format (.tsv) is a simple, easy to edit text format that allows any text within a field except TAB and newline characters.
There are no escaping rules in its IANA format spec.
TSV ...
- 5,437
3
votes
How to handle multiple versions of binary file format
Honestly, you have to open the stream and read enough bytes to determine which file format you are actually dealing with first. This solution is not unlike what many graphics tools do to discover the ...
- 46.3k
3
votes
Would you use Byte Arrays or XML if size was not an issue?
Byte arrays are human readable. I've been reading them since I was 10 and I've been human for longer than that. This was before I even had a fancy hex editor. Peek and Poke commands are old friends....
- 115k
3
votes
What are the benefits of storing data contiguously?
Depends on so many factors, but here are a couple points to consider:
Recognize that if you use a very large file format that is "contiguous," its contiguousness is logical, not physical. The ...
- 26.9k
3
votes
Accepted
Selecting the endianness of data in files generated by an embedded system
If your goal isn't interoperability with independently written software, endianness conversion is probably overkill. You're already doing it for network data, which would not strictly be necessary - ...
- 17.9k
2
votes
What are the benefits of storing data contiguously?
strictly append-only - no updates to already-written data are allowed.
This sounds suspiciously like a log. Did you consider a log-oriented data storage? Like a blockchain or logstash or git?
The ...
- 12.5k
2
votes
What are the benefits of storing data contiguously?
This appears to be case of premature optimisation - it is an attempt to performance-tune a design that has not exposed a performance problem and in doing so, tightly couples the file format to ...
- 3,188
2
votes
Would you use Byte Arrays or XML if size was not an issue?
You example XML isn't even a good example, as you're still using a byte array. A 'true' XML for an image, to contrast with byte arrays, would be something like:
<s:Image xmlns="www.w3c.org" width="...
- 27.7k
2
votes
Would you use Byte Arrays or XML if size was not an issue?
Two things:
The benefits of your scheme are small and questionable: yes I can parse and extract metadata a bit more easily. But you only really need one C library to parse an image format and then it'...
- 3,241
2
votes
Should UTF-8 CSV files contain a BOM (byte order mark)?
The only public definition of CSV format is RFC 4180 and such CSV file should use UTF-8 encoding and no byte-order-mark (BOM) because UTF-8 files do not depend on CPU byte order and therefore do not ...
2
votes
Accepted
Design Pattern to extract arbitrary field from arbitrary file format
Regardless of the pattern you use the fundamental thing here is deciding what you're going to model: whatever the file has in it or what your application needs.
There are use cases for either. A text ...
- 115k
2
votes
Using source code instead of XML/JSON or other custom serialization schemes and binary file formats
The other answer has already covered the main point why storing configuration in source is a bad idea. But I wanted to add a few points.
Just about every developer should know how json works, and ...
- 5,998
1
vote
Accepted
Getting file format by checking file header
Use memcmp to compare the PNG header against the first bytes of the file. Other than that you can't avoid opening and reading the file.
Maybe you should fseek back to the start, to reset the file ...
- 116
1
vote
What are formats to store geographic maps for a robot to travel point A to point B
For moving robots or calculate routes to move from point A to point B, you need to use a graph data structure. A graph is a set of nodes and a set of edges that relate nodes. A path is a sequence of ...
- 80.6k
Only top scored, non community-wiki answers of a minimum length are eligible