3

I see a lot of reasons to use binary over text-based formats.

  • With binary, I find it a lot easier. I can use fread(data, sizeof(struct DataStruct), 1, fileptr) to read, or fwrite to write data. I find that a lot easier than finding and using someone else's bloated library to parse and encode data in JSON or XML.
  • Since you're just reading the data directly, there is no parsing involved, making write and especially read operations a lot faster.
  • Since it's binary, instead of text based, the files are a lot smaller.

However, despite all of that, text-based formats are widely used. Why are JSON, XML, OBJ models, and other text-based formats so popular? What other factors did I not consider that makes these inefficient text-based formats desirable?

29
  • 2
  • 4
    The other thing to consider is what happens when you add even one member to your struct/object. Now the files you write can't be read by the previous version and the files you already have can't be read by the new version. With JSON/XML the previous version just ignores the items it doesn't know about and the new version can use a default for the missing fields. You can do that with binary as well but then you have to keep both the new and old reead/write functions and embed a version number in the file. – Jerry Jeremiah Dec 11 '20 at 3:32
  • 7
    @BlockofDiamond, once you add support for versioning to your binary file, you need a parser to consistently read the data and place it in the right elements of the data structure. Then the only advantage left for binary files is their smaller size. – Bart van Ingen Schenau Dec 11 '20 at 7:03
  • 5
    @BlockofDiamond I think you're lacking some imagination here. Think of all the different dimensions two systems can differ in: floating point format, integer size, endianness, bit representations of booleans, struct member layout, preferred text encoding, etc. Various macros could add/remove entire chunks of members from a struct's declaration. Sure, you can add version flags, checks and conversions to make sure all doubles are 64 bit IEEE, to try to work it around it all, but what you're essentially doing is inventing your own shitty ad-hoc standard – Alexander Dec 11 '20 at 15:32
  • 2
    @BlockofDiamond: I responded to your argument about readability. You working in C is not a valid argument against the use of text-based data formats in general - and you asked a general question here. "Just in that case, hidden behind the library function instead of in the developer's code." At its very essence, this argues against the idea of incremental improvements made to code, whether it be frameworks, libraries, or additional abstraction. Just because you can ride a horse bareback and you personally don't mind it, doesn't mean saddles are a waste of time for everyone else. – Flater Feb 17 at 9:52

12 Answers 12

11

JSON and XML are less efficient than pure binary, but what you aren't considering is that they (JSON/XML, and some other formats like YAML) are standardised, while binary isn't - different systems, running in different environments or on different hardware, will encode what would be identical JSON or XML as non-identical binary.

That makes transmitting data between different systems less reliable and is a large factor in why JSON is a common format for data exchange in, say, HTTP requests. If you serialise data to a "plaintext"-like format its much easier to work with as a receiver.

Another consideration is also human readability, which may or may not be important depending on the field you work in, but being able to read and inspect stored data easily can help in debugging.

8
  • I see, but other than the endian of the systems in question, are there any examples of times when the binary wouldn't be identical? – Block of Diamond Dec 11 '20 at 3:05
  • 2
    supported/selected character sets and whether it uses null-terminating or specified length strings when dealing with text, what integer sizes are supported/used and how it might differentiate between them (might it try to parse 2 32bit numbers as a single 64bit one?), and ofcourse, big vs small endian. There may be others i'm not thinking of right now, but you only really need a few differences for it to be useful to standardise. – AIWalker Dec 11 '20 at 3:17
  • As for null-terminating or specified length strings, isn't that up to the coder, not the system? Whether I write the code to interpret data as null terminated vs specified length strings? Also couldn't they just use bitfields in structs or fixed size integers such as uint32_t? – Block of Diamond Dec 11 '20 at 3:20
  • 1
    Systems where int is a different size from what you think it is. Systems with different padding conventions. Systems trying to read your C-written binary from Rust, Go, Java, (insert your least favourite language here). – Philip Kendall Dec 11 '20 at 3:21
  • 5
    it being "up to the coder" is part of why these things are transferred in a format like JSON in a standardised encoding (in my experience, usually utf8). you might choose to use null terminating strings and 32 bit ints, but if you need to send data to someone who uses specified lengths and/or 64 bit ints, you'd potentially have a problem. if you know for sure on both ends what the format would be, and speed/size is important, then yes, binary would be a good solution. Thats not always the case. – AIWalker Dec 11 '20 at 3:27
7
  • They're human readable: you can read and write them in plain text editor
  • They're standardized: which means there are a wealth of off the shelf tooling available to work with them
  • They're easy: you don't need to know whether your system is big or small endian, you don't need to know how strings are encoded in-memory in your language+platform combination (in-memory strings usually uses weird encoding like UTF-16/UTF-32 or some non standard extension of ASCII codepage because they are fixed width encoding, but these are wasteful for data storage/exchange which is almost always in UTF-8 as they produce smaller files), you don't need to know the memory alignment requirement of struct in your programming language+platform
  • They're in standard libraries: most modern languages supports standard text based formats as part of their standard library, usually at least JSON and XML
  • You can write schemas: schemas allows you to unambiguously specify what your file format should look like
  • Schemas enable interoperability: if you're developing a file format that's used by multiple parties, a schema allows different parties to ensure that they're producing files that can be correctly read by all other implementations
  • Schemas can automatically be turned into validating parsers: and into an API client library without you needing to write a single line of code yourself in the target platform
  • They're more secure: especially if you have a schema/validating parser, there's much less chance of getting data that you don't expect and causing security issues like buffer overflow, arbitrary code execution, etc. The history book is full of issues caused by poorly specified binary file format causing security issues, these issues are nearly non existent when you use text-based format.
  • Speed and size is usually of minimal concern for data interchange: your bottleneck is going to be the network, not file parsing. Once you downloaded the data, you can extract the data and store the important bits in a database, indexed and in a form that you can work with efficiently in the local system
  • They're plain text: so Unix tools knows how to work with them to some degree, you can use grep and sed to search and script one off manipulations easily
4

I suspect this question is posed more as rhetoric than from genuine puzzlement.

First off it must be acknowledged that binary formats are long-established and a perfectly legitimate option for saving data.

Though probably not what you had in mind, image formats are an obvious example where binary remains king - there is no breakdown of image data into human-readable scan lines or other algorithmic elements.

But equally, text-based formats were also long-established before XML and JSON became popular, particularly CSV for tabular data, and for example INI files (on Windows) for application configuration.

The primary advantage of binary formats for bulk data in the past was that they were compact (mainly because there are often no delimiters between fields, or other metadata - structural and descriptive information is stored almost entirely in the application or source code), and they are fast to parse because they are well-integrated with the application which uses them. The application is essentially written as a custom parser, probably exploiting assumptions built into the language or platform.

However speed and size for file formats and file-based applications have receded as concerns nowadays. What tends to be more important is durability and flexibility.

More standardised representations of data, more standardised layouts, the storage of descriptive metadata alongside the data, these are all recognised as things which tend to preserve the ability to read and write the data in the long-term, and ease the ability of programmers to read and manipulate the data without access to (or detailed familiarity with) the original source code or development environment.

Indeed, as other answers say, they can also be a boon for debugging and manual data recovery.

3
  • "Though probably not what you had in mind, image formats are an obvious example where binary remains king" except for when it comes to formats like SVG, of course! – Jasmijn Dec 13 '20 at 1:33
  • @Jasmijn: or netpbm images, but that probably fails the "widely used" criteria. – Lie Ryan Dec 13 '20 at 10:35
  • 1
    @Jasmijn, exactly so! I was thinking of photographics as opposed to vector graphics, but the print and publishing industry has a long history of standardised file formats with extensive metadata, mainly because their operational scale has never allowed them the luxury of assuming that their data will be handled end-to-end by one dedicated application on one piece of hardware, and indexed and organised casually by a small number of users. – Steve Dec 13 '20 at 11:20
3

You also miss a important point: Not all of us are using C/C++.

In other languages (like Java, JavaScript, ...) you can't easily save or read a struct.
And if you want to share these files between programs with different languages, than you need a standardized format.

In JavaScript or Java it's very hard (or almost impossible) to read a not aligned (packed) struct, saved by C. Because they don't working with bits and bytes. Java-Devs have not to worry about endian, 32 vs. 64bit, aligned/packed or not.

3

I should add a very pertinent war story. Way back in time Borland released Delphi 1.0 with the screen design files saved in binary format. This was a minor disaster.

  • Delphi was not itself bug free. It would regularly corrupt the file. It helped if you had backups or a very understanding boss.

  • Delphi supported third party libraries that could add to the layout file. This probably meant that that corruption could happen over which Borland had little control.

  • Renaming a screen component or changing it's class was a pain. All you could do was delete the component, add the replacement and wire up all the properties again. Refactoring was heavily impacted.

To their credit, Borland fixed all this by changing the layout file to a text file (I think) xml and all those problems went away.

Ever since, I have stuck with text files.

1

Apart from the obvious advantages that text is human readable, sometimes human fixable, and sometimes diffable, independent of endianness, and fully Unicode compatible:

  1. Specifically for JSON, in the environments that I use I can translate a file or a byte array containing JSON into a generic data structure containing arrays, dictionaries, and various primitives, with a single call. The integrity of the JSON data will be checked. And I don't need to know anything about the contents for this. So I have a single checkpoint where I know either "all my data is valid JSON" or "my data is not valid JSON".

  2. JSON describes its contents. That makes it relatively easy to read data from older or newer versions of your software with a bit of effort. With home-made binary data this is hard.

1

There are many good answers already but I just wanted to mention some important bits that are lacking in many answers.

  • Size is not a problem in many JSON protocols also because it is or can be zipped on the fly, either it fits on a couple of IP packet so there is zero gain on saving some spare bytes (that's the case for most JSON protocols) either you ship large data and you can gain ~80% of the original size if you just zip it.
  • Some binary standardized protocols exist, sometimes for storage, sometimes for efficient unpacking compared to zipped JSON(although that's quite fast). There is life outside JSON and XML. Check for example MesssagePack or BSON.
1

Sounds to me like you're using C or C++ here. In that case, here's my cautionary tale.

Twenty-odd years ago, someone [else!] wrote an Application in [MFC] C++ and used exactly this technique to save files from that Application. Only that Application ever needed to use these files, so that was fine.
This application was hugely successful.

Fast-forward twenty years:

  • Users have saved away tens of thousands of files from this Application.
  • The technology is seen as "old" and "obsolete".
  • Very few people are left who know how to work with it and even less want to get their hands dirty looking after this "ancient monstrosity".
  • Management would love to replace the application with something else ...
    ... but what about all those binary files that the Users have saved away and still rely on to run their business?

Someone is going to have [re-]invent a way to read those file into some other Application.

How much easier (less expensive and less Risky) would this exercise be if only the original Developer had put in the time and effort to save the data in a more portable format ...

0

The other reason (I guess) is that XML/JSON are human readable. It sometimes matters. For example JSON is widely used as a configuration file format so it is supposed to be written/edited by humans. Your question kinda sounds like "why there's a hammer if there's a shovel" ;) Binary and non-binary formats are created for different purposes. As other people mentioned here not every one uses C/C++ (e.g. me). It is much simpler to work with XML/JSON using Python than binary (without a standard).

Python parse/load JSON:

import json
parsed = json.loads(string_json)

With binary, I find it a lot easier. I can use fread(data, sizeof(struct DataStruct), 1, fileptr) to read, or fwrite to write data. I find that a lot easier than finding and using someone else's bloated library to parse and encode data in JSON or XML.

In some other languages these libraries are widely used. There's no need to do parsing "manually" in Python for example.

Since you're just reading the data directly, there is no parsing involved, making write and especially read operations a lot faster.

Reading by a machine: yes. By a human: no. No human can get it at all.

Since it's binary, instead of text based, the files are a lot smaller.

Smaller: yes. Human readable: no. You can't understand a binary file without an auxiliary program. You won't save config in a binary file, yeah? ;)

So sometimes you need a binary file. Sometimes you need a text file. Your C++ code is a text file so you can read and write it being a human. But it is compiled to machine code in order to be ran more easily and fast by a machine.

1
  • I see the analogy that JSON and XML is like uncompiled binary – Block of Diamond Feb 17 at 2:57
0

Efficient... How?

There are many ways for something to be efficient:

  • Total Storage used
  • Speed of reading
  • complexity of code performing the reading
  • adding new fields
  • removing old fields
  • saving a different length string/blob
  • network communication
  • error detection/correction
  • ease of implementation by other programmers

and more to boot...

Text is Binary

Unless my understanding of how computers work is fundamentally flawed, text files are binary files.

What you have is a much more nuanced world:

  • Some Files contain fixed length data-structures repeated one after another: Record based Files
  • Some files contain variable length blocks whose contents are sometimes different to the previous or next block, but form a chain of blocks: Block based Files
  • Some files contain data in some form of graph with records pointing to where related data is within the file: Structure based Files
  • Some files contain self describing data which follows a minimum set of conventions: Formal Language based Files
  • Some files contain semi-structured data which is generally freeform with a few areas of formality to ease automated processing: Natural Language based Files

And then there are the files that nest different sorts of file within demarked regions. eg: a record based file may nest a structured file within it logically. A Block based file may have blocks whose contents are Formal Language based.

Needless to say it doesn't make a lot of sense to talk about text files without knowing what kind of text file it is. It being human ledgable without a piece of paper and a calculator is neither here nor there.

They could be:

  • Natural Language based: like an SMTP file.
  • Formal Language based: like JSON or XML
  • Block Based: Like some variants of Key Value Pair or CSV
  • Record Based: Like some variants of CSV or Key Value Pair
7
  • Text ist not binary. Binary-files are not human-readble (you need a hex-editor to read or edit them). But text-files can be read by humans. – akop Dec 13 '20 at 10:20
  • 3
    @akop At the file level all data is simply a collection of bits - thus all files are binary. What you call a "text file" is simply your tool of choice taking those bits and mapping them to glyphs that you are able to interpret as text. And the mapping function is not universal, EG EBCDIC and ASCII encoded files are totally different. – Peter M Dec 16 '20 at 14:44
  • @PeterM if this definition is true, than are all files binary. But luckily is this not the case. Files are human readable, when a normal encoder can transform the bits and bytes to ASCII (or familiar). On the other side: A executable (as example) does not contains any human-readble bytes. You can't type cat file.exe to look what the program do. – akop Dec 16 '20 at 15:56
  • 1
    @akop Your "cat" example is false equivalence. Look at how a character is displayed on a screen: 1. A binary value is read from a file. 2. That value is used as an index into a font file that points to the glyph data. 3. The glyph data is rendered into the display buffer via a graphical process. 4. Your brain recognizes the image in the display buffer as a character. This process is the same whether you use "cat" or "hexdump" to display the contents of a file - only the mapping of the binary data to the glyph changes. – Peter M Dec 16 '20 at 16:46
  • 1
    @akop which is why i pointed out that it is actually more nuanced than that. Record orientated, block orientated, semi-structured, etc... The efficiency of each is directly proportional to its usage. Record is horrendous for variant data, unstructured is hard for automating meaningfully. – Kain0_0 Jan 4 at 4:05
0

The mentioned benefits are moot.

  • In any scenario where text based files are used, performance is not an issue.
  • Easy access on the programmer level should not be an issue, you would be talking to an object model rather than mess will low level persistency logic. Persistency should be encapsulated anyway, no matter how it is implemented.

Text based formats provide structure. They are human readable and writable using standard editing tools. Mind that it is often not just you, the programmer, who needs to deal with those files. "This is easy for me" hardly ever cuts it in software development. I could have answered you in Dutch because that would have been easy for me but it probably would not have been very helpful.

4
  • Um... no, parsing is a step, even if it's hidden behind JSON library functions in high level languages – Block of Diamond Feb 17 at 2:55
  • @Block of Diamond It is a step, no argument there. What is your point? – Martin Maat Feb 17 at 13:57
  • Each extra step will mean that it will take more time, so I disagree that my mentioned benefits are invalid – Block of Diamond Feb 19 at 21:12
  • My software often exchanges data in heavily encrypted JSON. The encryption is needed. The time for parsing JSON is tiny compared to the crypto. producing and parsing JSON is done with standard functions. – gnasher729 Feb 22 at 8:12
0

The answer is rather simple: you should use text when interoperability with tools that are known to require text or where you know you will be operating with other tools and you don’t of a shared non-text format.

If you want to have an image that is going to edited or analyzed by other tools, you’ll choose a binary format best represents your needs. You know what format you want and what tools can support it. If on the other hand you have a bunch of non-image, but non-text data that can be represented as text, and you want to let other tools deal with it, you’ll use a textual format that allows them to extract the data and examine it.

Simple example, a sql database table with a json or xml column. The database understands various base binary types: dates, times, numbers, even binary data, but if you want to query the last, you have to decode it yourself. It’s extremely difficult to find a row based upon partial binary content, particularly if the data is in a variable format. On the other hand, even if the sql engine doesn’t allow decoding of the textual format into a more binary format, it can still be queried as raw text. Finding rows with Zarg: 37 will be possible.

And of course the same thing applies if you’re not using a database to store the data: if you want a sum of zargs (whatever that may be) you don’t need to understand the rest of the format.

Also, having had to reverse engineer a couple of file formats in the past, I can say it would have been a lot easier if the native format hadn’t been binary. Not that I think it would have been logical for them to do so, but having an export to a textual representations would have been terrific.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.