Why do most log files use plain text rather than a binary format?

Question

Logging is something that is necessary but is (relatively) rarely used. As such it can be made much more compact in terms of storage.

For example the data most commonly logged like ip, date, time and other data that can be represented as an integer is being stored as text.

If logging was stored as binary data, a lot of space could be preserved thus requiring less rotation and increasing disk lifespan, especially with SSDs where writes are limited.

Some may say that it is such a minor issue that it does not really matter, but taking in consideration the effort needed to build such mechanism it makes no sense not to. Anyone can make this for like two days in his spare time, why don't people do this?

Answer 1

systemd famously stores its log files in binary format. The main issues I have heard with it are:

1. if the log gets corrupted it's hard to recover as it needs specialist tooling
2. they are not human readable, so you can't use standard tools such as vi, grep, tail etc to analyse them

The main reason for using a binary format (to my knowledge) was that it was deemed easier for creating indices etc i.e. to treat it more like a database file.

I would argue that the disk space advantage is relatively small (and diminishing) in practice. If you want to store large amounts of logging then zipping rolled logs is really quite efficient.

On balance, the advantages of tooling and familiarity probably would err on the side of text logging in most cases.

Answer 2

Why do most log files use plain text rather than a binary format?

Search for the word "text" in the Unix philosophy Wikipedia article, for example you'll find statements like:

McIlroy, then head of the Bell Labs CSRC (Computing Sciences Research Center), and inventor of the Unix pipe,[9] summarized the Unix philosophy as follows:[10]

This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.

Or for example, from Basics of the Unix Philosophy,

Rule of Composition: Design programs to be connected with other programs.

It's hard to avoid programming overcomplicated monoliths if none of your programs can talk to each other.

Unix tradition strongly encourages writing programs that read and write simple, textual, stream-oriented, device-independent formats. Under classic Unix, as many programs as possible are written as simple filters, which take a simple text stream on input and process it into another simple text stream on output.

Despite popular mythology, this practice is favored not because Unix programmers hate graphical user interfaces. It's because if you don't write programs that accept and emit simple text streams, it's much more difficult to hook the programs together.

Text streams are to Unix tools as messages are to objects in an object-oriented setting. The simplicity of the text-stream interface enforces the encapsulation of the tools. More elaborate forms of inter-process communication, such as remote procedure calls, show a tendency to involve programs with each others' internals too much.

Anyone can make this for like two days in his spare time, why don't people do this?

Storing the log file in binary is only the beginning (and trivial). You'd then need to write tools to:

• Display the whole log file (edit)
• Display the end of the log, without reading the beginning of it (tail -f)
• Search for stuff in the file (grep)
• Filter to only display selected/interesting stuff (using an arbitrarily complicated filter expression)
• Email the log to someone else who doesn't have your log-file-decoder-software
• Copy-and-paste a fragment of the log file
• Read the log file while the program (which creates the log file) is still being developed and debugged
• Read log files from old versions of the software (which are deployed on customer sites and running).

Obviously software can and does use binary file formats too (e.g. for relational databases) but it's not worthwhile (in a YAGNI sense), usually not worth doing, for log files.

Answer 3

There are a lot of debatable presumptions here.

Logging has been an integral part of (almost) every job I've had. It is essential if you want any sort of visibility on the health of your applications. I doubt that it is a "fringe" use; most organizations I've been involved with consider logs very important.

Storing logs as binary means you must decode them before you can read them. Text logs have the virtue of simplicity and ease of use. If you're contemplating the binary route, you might as well store logs in a database instead, where you can interrogate them and statistically analyze them.

SSD's are more reliable than HDD's nowadays, and the arguments against lots of writes are largely moot. If you're really worried about it, store your logs on an ordinary HDD.

Answer 4

Log files are a critical part of any serious application: if the logging in the app is any good, then they let you see which key events have happened and when; what errors have occurred; and general application health that goes beyond whatever monitoring has been designed in. It's common to hear about a problem, check the application's built-in diagnostics (pop open its web console or use a diagnostic tool like JMX), and then resort to checking the log files.

If you use a non-text format, then you are immediately faced with a hurdle: how do you read the binary logs? With the log-reading tool, which isn't on your production servers! Or it is, but oh dear, we've added a new field and this is the old reader. Didn't we test this? Yes, but nobody deployed it here. Meanwhile, your screen is starting to light up with users pinging you.

Or perhaps this isn't your app, but you are doing support and you think you know it's this other system, and WTF? the logs are in a binary format? Ok, start reading wiki pages, and where do you start? Now I've copied them across to my local machine, but - they're corrupted? Have I done some kind of non-binary transfer? Or is the log-reading tool messed up?

In short, text-reading tools are cross-platform and ubiquitous, and logs are often long-lived and sometimes need to be read in a hurry. If you invent a binary format, then you are cut off from a whole world of well-understood and easy-to-use tools. Serious loss of functionality just when you need it.

Most logging environments strike a compromise: keep the current logs readable and present, and compress the older ones. That means you get the benefit of the compression - more so, in fact, because a binary format wouldn't shrink the log messages. At the same time, you can use less and grep and so on.

So, what possible benefits might arise from using binary? A small amount of space efficiency - increasingly unimportant. Fewer (or smaller) writes? Well, maybe - actually, the number of writes will relate to the number of disk-commits, so if log-lines are significantly smaller than the disk blocksize, then an SSD would be assigning new blocks over and over anyway. So, binary is an appropriate choice if:

• you are writing huge amounts of structured data
• the logs have to be created particularly quickly
• you are unlikely to need to analyze them under "support conditions"

but this is sounding less like application logging; these are output files or activity records. Putting them in a file is probably only one step away from writing them to a database.

EDIT

I think there's a general confusion here between "program logs" (as per logging frameworks) vs "records" (as in access logs, login records etc). I suspect the question relates most closely to the latter, and in that case the issue is far less well-defined. It's perfectly acceptable for a message-record or activity log to be in a compact format, especially as it's likely to be well-defined and used for analysis rather than troubleshooting. Tools that do this include tcpdump and the Unix system monitor sar. Program logs on the other hand tend to be much more ad hoc.

Answer 5

An example of a somewhat binary log is wide-spread: the Windows event log. On the pro side, this allows log messages to be quite wordy (and thus hopefully helpful) at virtually no cost, possibly something like

Warning: The queue of foobars to do has grown by 517 items over the last 90 seconds. If this happens about once per day, there is nothing to worry about. If it happens more often or in rapid succession, you may want to check the amount of RAM available to the foobar application. If it occurs together with event 12345, however, you seem to be using an obsolete database and you better call support at +1-555-12345 in order to prevent data loss.

The main part of this message exists only once as a resource installed with the application. However, if this resource is not installed correctly (for example, because meanwhile a newer version has been installed that no longer supports this obsolete message), all you see in the event log is a standard message that is just fancy wording for

Dunno, something with "517" and "90".

and no longer helpful in any way.

Answer 6

TL;DR: Size doesn't really matter, but convenience of use does

First of all, whilst comparing the respective advantages of text and binary formats for short-term log storage is an important question, the size does not really matter. The two reasons for this are:

1. Logs are highly redundant information that will compress very well: in my experience it is not rare to see compressed log files whose size is 5% or less of the size of the original file. Consequently, using a text or a binary format should not have any measurable impact on the long-time storage of logs.

2. Whatever format we choose, logs will quickly fill a server disk if we do not implement a “log files sink” that compresses and sends log files to a long-term storage platform. Using a binary format could slow this a bit but even a change by a factor 10 would not matter that much.

Text versus binary log formats

The promise of Unix systems is that, if we learn to use the standard toolset working on text files structured in lines – such as grep, sort, join, sed and awk – we will be able to use them to quickly assemble prototypes performing any job we want, albeit slowly and crudely. Once the prototype has demonstrated its usefulness, we can choose to turn it in a really engineered software to gain performance or add other useful features. This is, at least in my understanding, the essence of the Unix philosophy.

To put it another way, if we likely need to perform treatments and analyses we cannot figure out by today, if we do not know who should implement this analysis, etc. then we are in the stage where prototypes should be used and text formats for logs are probably optimal. If we need to repeatedly perform a small set of well-identified treatments, then we are in the situation where we should engineer a perennial software system to perform this analyse and binary or structured formats for logs, such as relational databases, are likely to be optimal.

(Some time ago, I wrote a blog post about this.)

Answer 7

The two main questions you would want to ask before choosing between text and binary are:

• Who is my audience?
• What content do I need to convey?

A common opinion is that the audience of a log message is a human being. This is obviously not a perfect assumption, because there are plenty of log crawling scripts out there, but it is a common one. In this case, it makes sense to convey the information in a medium which humans are comfortable with. Text has a long standing tradition of being this medium.

As for content, consider that a binary log must have a well defined format. The format must be well defined enough for other people to write software which operates on those logs. Some logs are quite well structured (your question lists several). Other logs need the ability to convey content in a less-well-defined natural language form. Such natural language cases are a poor match for binary formats.

For the logs which could be well described in binary, you have to make a choice. Because text works for everyone, it is often seen as the default choice. If you log your results in text, people can work with your logs. It's been proven thousands of times. Binary files are trickier. As a result, it may be that developers output text simply because everyone knows what that's going to behave like.

Answer 8

Log files are in text format because they can be easily read using any type of text editor or by displaying the contents via console command.

However, some log files are in binary format if there is a lot of data. For example, the product I am working on stores a maximum of 15000 records. In order to store the records in the least amount of room, they are stored in binary. However, a special application must be written to view the records or convert them to a format that can be used (e.g. spreadsheets).

In summary, not all log files are in textual format. Textual format has an advantage that custom tools are not needed to view the content. Where there is a lot of data, the file may be in binary format. The binary format will need a (custom) application to read the data and display in a human readable format. More data can be packed into a binary format. Whether to use textual format or binary format is a decision based on the amount of data and ease of viewing the contents.

Answer 9

In embedded systems where I might not have an output channel available during run-time, the application can't afford the speed hit imposed by the logging, or logging would alter or mask the effect I'm trying to record, I've often resorted to stuffing binary data into an array or a ring buffer, and either printf()ing it at the end of the test run or dumping it raw and writing an interpreter to print it as readable. Either way, I want to end up with readable data.

In systems with more resources, why invent schemes to optimize what doesn't need optimizing?

Answer 10

Log files are intended to aid debugging of issues. Typically, hard drive space is much cheaper than engineering time. Log files use text because there are many tools for working with text (such as tail -f). Even HTTP uses plain-text (see also why don't we send binary around instead of text on http).

Additionally, it's cheaper to develop a plain-text logging system and verify that it works, easier to debug if it goes wrong, and easier to recover any useful information in case the system fails and corrupts part of the log.

Answer 11

A corrupted text file is still readable around the corrupted part. A corrupted binary file maybe restorable, but it also might not be. Even if it is restorable, it would require quite a bit more work. The other reason is that a binary logging format makes it less likely that during a rush to create a "temporary fix" (aka "the most permanent of all fixes") the logging solution will get used instead of something which can be created quicker.

Answer 12

Back in my mainframe days, we used a custom-designed binary log format. The main reason wasn't to save space, it was because we wanted the log to occupy finite space by overwriting old entries with new ones; the last thing we wanted was to be unable to diagnose problems caused by disks becoming full (in 1980 disk space used to cost $1000/Mb, so people didn't buy more than they needed).

Now I still like the idea of a circular log file, and if operating systems offered such a beast I would use it without hesitation. But binary was a bad idea. You really don't want to have to waste time finding the right commands for deciphering a log file when you've got a critical problem to solve.

Answer 13

We count on unit testing for attaining and maintaining the robustness of our software. (Most of our code runs in a server, headless; post-operation analysis of log files is a key strategy.). Nearly every class in our implementation does some logging. An important part of our unit testing is the use of 'mock' loggers that are used when unit testing. A unit test creates a mock logger and provides it to the item being tested. It then (when useful/appropriate) analyses what got logged (especially errors and warnings). Using a text-based log format makes this much easier for much the same reasons that analyses performed on 'real' logs: there are more tools at your disposal that are quick to use and adapt.

Answer 14

Historically, Logs were official, hand-written and sequential records of events. When machinery became capable of recording events, these were written to a hard-copy output device such as a teletype printer, which produced a permanent sequential record but which could only process text, and occasionally ring a BELL ...