Why do or should programmers save data in text based formats like JSON or XML instead of binary?

Question

I see a lot of reasons to use binary over text-based formats.

• With binary, I find it a lot easier. I can use fread(data, sizeof(struct DataStruct), 1, fileptr) to read, or fwrite to write data. I find that a lot easier than finding and using someone else's bloated library to parse and encode data in JSON or XML.
• Since you're just reading the data directly, there is no parsing involved, making write and especially read operations a lot faster.
• Since it's binary, instead of text based, the files are a lot smaller.

However, despite all of that, text-based formats are widely used. Why are JSON, XML, OBJ models, and other text-based formats so popular? What other factors did I not consider that makes these inefficient text-based formats desirable?

Answer 1

JSON and XML are less efficient than pure binary, but what you aren't considering is that they (JSON/XML, and some other formats like YAML) are standardised, while binary isn't - different systems, running in different environments or on different hardware, will encode what would be identical JSON or XML as non-identical binary.

That makes transmitting data between different systems less reliable and is a large factor in why JSON is a common format for data exchange in, say, HTTP requests. If you serialise data to a "plaintext"-like format its much easier to work with as a receiver.

Another consideration is also human readability, which may or may not be important depending on the field you work in, but being able to read and inspect stored data easily can help in debugging.

Answer 2

• They're human readable: you can read and write them in plain text editor
• They're standardized: which means there are a wealth of off the shelf tooling available to work with them
• They're easy: you don't need to know whether your system is big or small endian, you don't need to know how strings are encoded in-memory in your language+platform combination (in-memory strings usually uses weird encoding like UTF-16/UTF-32 or some non standard extension of ASCII codepage because they are fixed width encoding, but these are wasteful for data storage/exchange which is almost always in UTF-8 as they produce smaller files), you don't need to know the memory alignment requirement of struct in your programming language+platform
• They're in standard libraries: most modern languages supports standard text based formats as part of their standard library, usually at least JSON and XML
• You can write schemas: schemas allows you to unambiguously specify what your file format should look like
• Schemas enable interoperability: if you're developing a file format that's used by multiple parties, a schema allows different parties to ensure that they're producing files that can be correctly read by all other implementations
• Schemas can automatically be turned into validating parsers: and into an API client library without you needing to write a single line of code yourself in the target platform
• They're more secure: especially if you have a schema/validating parser, there's much less chance of getting data that you don't expect and causing security issues like buffer overflow, arbitrary code execution, etc. The history book is full of issues caused by poorly specified binary file format causing security issues, these issues are nearly non existent when you use text-based format.
• Speed and size is usually of minimal concern for data interchange: your bottleneck is going to be the network, not file parsing. Once you downloaded the data, you can extract the data and store the important bits in a database, indexed and in a form that you can work with efficiently in the local system
• They're plain text: so Unix tools knows how to work with them to some degree, you can use grep and sed to search and script one off manipulations easily

Answer 3

I suspect this question is posed more as rhetoric than from genuine puzzlement.

First off it must be acknowledged that binary formats are long-established and a perfectly legitimate option for saving data.

Though probably not what you had in mind, image formats are an obvious example where binary remains king - there is no breakdown of image data into human-readable scan lines or other algorithmic elements.

But equally, text-based formats were also long-established before XML and JSON became popular, particularly CSV for tabular data, and for example INI files (on Windows) for application configuration.

The primary advantage of binary formats for bulk data in the past was that they were compact (mainly because there are often no delimiters between fields, or other metadata - structural and descriptive information is stored almost entirely in the application or source code), and they are fast to parse because they are well-integrated with the application which uses them. The application is essentially written as a custom parser, probably exploiting assumptions built into the language or platform.

However speed and size for file formats and file-based applications have receded as concerns nowadays. What tends to be more important is durability and flexibility.

More standardised representations of data, more standardised layouts, the storage of descriptive metadata alongside the data, these are all recognised as things which tend to preserve the ability to read and write the data in the long-term, and ease the ability of programmers to read and manipulate the data without access to (or detailed familiarity with) the original source code or development environment.

Indeed, as other answers say, they can also be a boon for debugging and manual data recovery.

Answer 4

You also miss a important point: Not all of us are using C/C++.

In other languages (like Java, JavaScript, ...) you can't easily save or read a struct.
And if you want to share these files between programs with different languages, than you need a standardized format.

In JavaScript or Java it's very hard (or almost impossible) to read a not aligned (packed) struct, saved by C. Because they don't working with bits and bytes. Java-Devs have not to worry about endian, 32 vs. 64bit, aligned/packed or not.

Answer 5

I should add a very pertinent war story. Way back in time Borland released Delphi 1.0 with the screen design files saved in binary format. This was a minor disaster.

• Delphi was not itself bug free. It would regularly corrupt the file. It helped if you had backups or a very understanding boss.

• Delphi supported third party libraries that could add to the layout file. This probably meant that that corruption could happen over which Borland had little control.

• Renaming a screen component or changing it's class was a pain. All you could do was delete the component, add the replacement and wire up all the properties again. Refactoring was heavily impacted.

To their credit, Borland fixed all this by changing the layout file to a text file (I think) xml and all those problems went away.

Ever since, I have stuck with text files.

Answer 6

Apart from the obvious advantages that text is human readable, sometimes human fixable, and sometimes diffable, independent of endianness, and fully Unicode compatible:

1. Specifically for JSON, in the environments that I use I can translate a file or a byte array containing JSON into a generic data structure containing arrays, dictionaries, and various primitives, with a single call. The integrity of the JSON data will be checked. And I don't need to know anything about the contents for this. So I have a single checkpoint where I know either "all my data is valid JSON" or "my data is not valid JSON".

2. JSON describes its contents. That makes it relatively easy to read data from older or newer versions of your software with a bit of effort. With home-made binary data this is hard.

Answer 7

There are many good answers already but I just wanted to mention some important bits that are lacking in many answers.

• Size is not a problem in many JSON protocols also because it is or can be zipped on the fly, either it fits on a couple of IP packet so there is zero gain on saving some spare bytes (that's the case for most JSON protocols) either you ship large data and you can gain ~80% of the original size if you just zip it.
• Some binary standardized protocols exist, sometimes for storage, sometimes for efficient unpacking compared to zipped JSON(although that's quite fast). There is life outside JSON and XML. Check for example MesssagePack or BSON.

Answer 8

Sounds to me like you're using C or C++ here. In that case, here's my cautionary tale.

Twenty-odd years ago, someone [else!] wrote an Application in [MFC] C++ and used exactly this technique to save files from that Application. Only that Application ever needed to use these files, so that was fine.
This application was hugely successful.

Fast-forward twenty years:

• Users have saved away tens of thousands of files from this Application.
• The technology is seen as "old" and "obsolete".
• Very few people are left who know how to work with it and even less want to get their hands dirty looking after this "ancient monstrosity".
• Management would love to replace the application with something else ...
  ... but what about all those binary files that the Users have saved away and still rely on to run their business?

Someone is going to have [re-]invent a way to read those file into some other Application.

How much easier (less expensive and less Risky) would this exercise be if only the original Developer had put in the time and effort to save the data in a more portable format ...

Answer 9

The other reason (I guess) is that XML/JSON are human readable. It sometimes matters. For example JSON is widely used as a configuration file format so it is supposed to be written/edited by humans. Your question kinda sounds like "why there's a hammer if there's a shovel" ;) Binary and non-binary formats are created for different purposes. As other people mentioned here not every one uses C/C++ (e.g. me). It is much simpler to work with XML/JSON using Python than binary (without a standard).

Python parse/load JSON:

import json
parsed = json.loads(string_json)

With binary, I find it a lot easier. I can use fread(data, sizeof(struct DataStruct), 1, fileptr) to read, or fwrite to write data. I find that a lot easier than finding and using someone else's bloated library to parse and encode data in JSON or XML.

In some other languages these libraries are widely used. There's no need to do parsing "manually" in Python for example.

Since you're just reading the data directly, there is no parsing involved, making write and especially read operations a lot faster.

Reading by a machine: yes. By a human: no. No human can get it at all.

Since it's binary, instead of text based, the files are a lot smaller.

Smaller: yes. Human readable: no. You can't understand a binary file without an auxiliary program. You won't save config in a binary file, yeah? ;)

So sometimes you need a binary file. Sometimes you need a text file. Your C++ code is a text file so you can read and write it being a human. But it is compiled to machine code in order to be ran more easily and fast by a machine.

Answer 10

Efficient... How?

There are many ways for something to be efficient:

• Total Storage used
• Speed of reading
• complexity of code performing the reading
• adding new fields
• removing old fields
• saving a different length string/blob
• network communication
• error detection/correction
• ease of implementation by other programmers

and more to boot...

Text is Binary

Unless my understanding of how computers work is fundamentally flawed, text files are binary files.

What you have is a much more nuanced world:

• Some Files contain fixed length data-structures repeated one after another: Record based Files
• Some files contain variable length blocks whose contents are sometimes different to the previous or next block, but form a chain of blocks: Block based Files
• Some files contain data in some form of graph with records pointing to where related data is within the file: Structure based Files
• Some files contain self describing data which follows a minimum set of conventions: Formal Language based Files
• Some files contain semi-structured data which is generally freeform with a few areas of formality to ease automated processing: Natural Language based Files

And then there are the files that nest different sorts of file within demarked regions. eg: a record based file may nest a structured file within it logically. A Block based file may have blocks whose contents are Formal Language based.

Needless to say it doesn't make a lot of sense to talk about text files without knowing what kind of text file it is. It being human ledgable without a piece of paper and a calculator is neither here nor there.

They could be:

• Natural Language based: like an SMTP file.
• Formal Language based: like JSON or XML
• Block Based: Like some variants of Key Value Pair or CSV
• Record Based: Like some variants of CSV or Key Value Pair

Answer 11

The mentioned benefits are moot.

• In any scenario where text based files are used, performance is not an issue.
• Easy access on the programmer level should not be an issue, you would be talking to an object model rather than mess will low level persistency logic. Persistency should be encapsulated anyway, no matter how it is implemented.

Text based formats provide structure. They are human readable and writable using standard editing tools. Mind that it is often not just you, the programmer, who needs to deal with those files. "This is easy for me" hardly ever cuts it in software development. I could have answered you in Dutch because that would have been easy for me but it probably would not have been very helpful.

Answer 12

The answer is rather simple: you should use text when interoperability with tools that are known to require text or where you know you will be operating with other tools and you don’t of a shared non-text format.

If you want to have an image that is going to edited or analyzed by other tools, you’ll choose a binary format best represents your needs. You know what format you want and what tools can support it. If on the other hand you have a bunch of non-image, but non-text data that can be represented as text, and you want to let other tools deal with it, you’ll use a textual format that allows them to extract the data and examine it.

Simple example, a sql database table with a json or xml column. The database understands various base binary types: dates, times, numbers, even binary data, but if you want to query the last, you have to decode it yourself. It’s extremely difficult to find a row based upon partial binary content, particularly if the data is in a variable format. On the other hand, even if the sql engine doesn’t allow decoding of the textual format into a more binary format, it can still be queried as raw text. Finding rows with Zarg: 37 will be possible.

And of course the same thing applies if you’re not using a database to store the data: if you want a sum of zargs (whatever that may be) you don’t need to understand the rest of the format.

Also, having had to reverse engineer a couple of file formats in the past, I can say it would have been a lot easier if the native format hadn’t been binary. Not that I think it would have been logical for them to do so, but having an export to a textual representations would have been terrific.