81

Logging is something that is necessary but is (relatively) rarely used. As such it can be made much more compact in terms of storage.

For example the data most commonly logged like ip, date, time and other data that can be represented as an integer is being stored as text.

If logging was stored as binary data, a lot of space could be preserved thus requiring less rotation and increasing disk lifespan, especially with SSDs where writes are limited.

Some may say that it is such a minor issue that it does not really matter, but taking in consideration the effort needed to build such mechanism it makes no sense not to. Anyone can make this for like two days in his spare time, why don't people do this?

  • 20
    I would challenge your assertion that people don't do this. Many do. Some don't, sure, but plenty do. – Servy Oct 4 '16 at 15:05
  • 55
  • 44
    > If logging was stored as binary data, a lot of space could be preserved Well, old logs are typically compressed. – leonbloy Oct 4 '16 at 17:10
  • 89
    Reading a text log on a machine that's halfway broken might be a huge advantage over needing a binary to analyze it. – tofro Oct 4 '16 at 18:30
  • 23
    After months of modifications to get the algorithm executed on the large cluster properly, we still couldn't see much of a performance gain, but when we changed to storing the log files in binary files? Holy cow, we never dared to dream that the performance could be at that level. How plausible is that kind of story? – null Oct 4 '16 at 19:13

14 Answers 14

164

systemd famously stores its log files in binary format. The main issues I have heard with it are:

  1. if the log gets corrupted it's hard to recover as it needs specialist tooling
  2. they are not human readable, so you can't use standard tools such as vi, grep, tail etc to analyse them

The main reason for using a binary format (to my knowledge) was that it was deemed easier for creating indices etc i.e. to treat it more like a database file.

I would argue that the disk space advantage is relatively small (and diminishing) in practice. If you want to store large amounts of logging then zipping rolled logs is really quite efficient.

On balance, the advantages of tooling and familiarity probably would err on the side of text logging in most cases.

  • 3
    Good point. I was immediately thinking of systemd too. The even more important part here is that your application doesn't have to know how the log data is stored. It can be provided as a system service. – 5gon12eder Oct 4 '16 at 15:30
  • 97
    "famously", more like "infamously" – whatsisname Oct 4 '16 at 18:18
  • 4
    pf (firewall) also logs in binary, specifically to tcpdump format – Neil McGuigan Oct 4 '16 at 19:39
  • 3
    @Hatshepsut Rolled logs: the log output writes to one file, say myapp.log until midnight, and then moves that file to myapp.log.1, and starts writing to a new myapp.log file. And the old myapp.log.1 gets moved to myapp.log.2, and so on, they all roll along. Thus, myapp.log is always the current one. Or they may switch when a certain size is reached. Maybe they put the date/time in the filename. Many logging frameworks support this sort of thing out of the box. – SusanW Oct 4 '16 at 23:45
  • 13
    @Hatshepsut The term rotating is also used from what I am aware. – George D Oct 5 '16 at 0:14
90

Why do most log files use plain text rather than a binary format?

Search for the word "text" in the Unix philosophy Wikipedia article, for example you'll find statements like:

McIlroy, then head of the Bell Labs CSRC (Computing Sciences Research Center), and inventor of the Unix pipe,[9] summarized the Unix philosophy as follows:[10]

This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.

Or for example, from Basics of the Unix Philosophy,

Rule of Composition: Design programs to be connected with other programs.

It's hard to avoid programming overcomplicated monoliths if none of your programs can talk to each other.

Unix tradition strongly encourages writing programs that read and write simple, textual, stream-oriented, device-independent formats. Under classic Unix, as many programs as possible are written as simple filters, which take a simple text stream on input and process it into another simple text stream on output.

Despite popular mythology, this practice is favored not because Unix programmers hate graphical user interfaces. It's because if you don't write programs that accept and emit simple text streams, it's much more difficult to hook the programs together.

Text streams are to Unix tools as messages are to objects in an object-oriented setting. The simplicity of the text-stream interface enforces the encapsulation of the tools. More elaborate forms of inter-process communication, such as remote procedure calls, show a tendency to involve programs with each others' internals too much.

Anyone can make this for like two days in his spare time, why don't people do this?

Storing the log file in binary is only the beginning (and trivial). You'd then need to write tools to:

  • Display the whole log file (edit)
  • Display the end of the log, without reading the beginning of it (tail -f)
  • Search for stuff in the file (grep)
  • Filter to only display selected/interesting stuff (using an arbitrarily complicated filter expression)
  • Email the log to someone else who doesn't have your log-file-decoder-software
  • Copy-and-paste a fragment of the log file
  • Read the log file while the program (which creates the log file) is still being developed and debugged
  • Read log files from old versions of the software (which are deployed on customer sites and running).

Obviously software can and does use binary file formats too (e.g. for relational databases) but it's not worthwhile (in a YAGNI sense), usually not worth doing, for log files.

  • 24
    Don't forget documentation! I wrote a binary message recorder for a system a few years ago, which logged incoming requests for regression/replay. Now, the only way to understand these awful files is to look at the code that read/writes them, and yet other teams use them and ask questions about them. Horrible things. – SusanW Oct 4 '16 at 22:10
  • 2
    To be fair, storing your log in a SQLite DB combined with basic query tools for reading would provide all those features you mention out of the box. ;) – jpmc26 Oct 6 '16 at 17:43
  • 3
    @jpmc26 Yes you can read the log file as long as you can, somehow, convert it to a text format... – ChrisW Oct 6 '16 at 18:06
  • 1
    as said in other comments: text files could be compressed easily and efficient. But the compression does not need to be in the 'data'. The compression could be done in the file system. so you can use the plain text for all tools and have no wasted disk space. – Bernd Wilke πφ Oct 7 '16 at 6:53
  • 2
    @JefréN. If I run tail -f on a multi-gigabyte log file, it skips to the end of the file (using 'seek' without 'read') and then reads-and-displays just the end of the file. It doesn't need to decompress/decode the whole file. – ChrisW Oct 11 '16 at 0:06
49

There are a lot of debatable presumptions here.

Logging has been an integral part of (almost) every job I've had. It is essential if you want any sort of visibility on the health of your applications. I doubt that it is a "fringe" use; most organizations I've been involved with consider logs very important.

Storing logs as binary means you must decode them before you can read them. Text logs have the virtue of simplicity and ease of use. If you're contemplating the binary route, you might as well store logs in a database instead, where you can interrogate them and statistically analyze them.

SSD's are more reliable than HDD's nowadays, and the arguments against lots of writes are largely moot. If you're really worried about it, store your logs on an ordinary HDD.

  • 19
    "you might as well store logs in a database, where you can interrogate them and statistically analyze them." At a previous job, we had a custom tool that imports our (text-based) logs into a database for exactly this purpose. – Mason Wheeler Oct 4 '16 at 15:16
  • 5
    I thin what OP meant by _"SSD where writes are limited" is the fact that in SSD have a limited write/erase cycles and writing too much on a sector diminished the service life of the device. She didn't mean that writes are lost. – Tulains Córdova Oct 4 '16 at 15:20
  • 4
    @TulainsCórdova: Yes, I knew what she meant. – Robert Harvey Oct 4 '16 at 15:25
  • 2
    @DocSalvager: I didn't assert otherwise. – Robert Harvey Oct 4 '16 at 21:25
  • 2
    @TulainsCórdova - the limits of SSD write cycles are generally very high these days. Even low-cost consumer grade SSDs have manufacturer warranties on write cycles that run into the high hundreds of times the size of the device, and MTBFs that will cover you for writing thousands of times the capacity of the device. And in a commercial setting you should be using higher end devices that have much larger write cycle limits and should be replacing them on at least a 5 year cycle so unless you're writing > 10% storage capacity per day, I don't think there's anything to worry about. – Jules Oct 7 '16 at 14:18
36

Log files are a critical part of any serious application: if the logging in the app is any good, then they let you see which key events have happened and when; what errors have occurred; and general application health that goes beyond whatever monitoring has been designed in. It's common to hear about a problem, check the application's built-in diagnostics (pop open its web console or use a diagnostic tool like JMX), and then resort to checking the log files.

If you use a non-text format, then you are immediately faced with a hurdle: how do you read the binary logs? With the log-reading tool, which isn't on your production servers! Or it is, but oh dear, we've added a new field and this is the old reader. Didn't we test this? Yes, but nobody deployed it here. Meanwhile, your screen is starting to light up with users pinging you.

Or perhaps this isn't your app, but you are doing support and you think you know it's this other system, and WTF? the logs are in a binary format? Ok, start reading wiki pages, and where do you start? Now I've copied them across to my local machine, but - they're corrupted? Have I done some kind of non-binary transfer? Or is the log-reading tool messed up?

In short, text-reading tools are cross-platform and ubiquitous, and logs are often long-lived and sometimes need to be read in a hurry. If you invent a binary format, then you are cut off from a whole world of well-understood and easy-to-use tools. Serious loss of functionality just when you need it.

Most logging environments strike a compromise: keep the current logs readable and present, and compress the older ones. That means you get the benefit of the compression - more so, in fact, because a binary format wouldn't shrink the log messages. At the same time, you can use less and grep and so on.

So, what possible benefits might arise from using binary? A small amount of space efficiency - increasingly unimportant. Fewer (or smaller) writes? Well, maybe - actually, the number of writes will relate to the number of disk-commits, so if log-lines are significantly smaller than the disk blocksize, then an SSD would be assigning new blocks over and over anyway. So, binary is an appropriate choice if:

  • you are writing huge amounts of structured data
  • the logs have to be created particularly quickly
  • you are unlikely to need to analyze them under "support conditions"

but this is sounding less like application logging; these are output files or activity records. Putting them in a file is probably only one step away from writing them to a database.

EDIT

I think there's a general confusion here between "program logs" (as per logging frameworks) vs "records" (as in access logs, login records etc). I suspect the question relates most closely to the latter, and in that case the issue is far less well-defined. It's perfectly acceptable for a message-record or activity log to be in a compact format, especially as it's likely to be well-defined and used for analysis rather than troubleshooting. Tools that do this include tcpdump and the Unix system monitor sar. Program logs on the other hand tend to be much more ad hoc.

  • 1
    Even Unix /var/log/utmp / wtmp are binary. They record who's currently logged in on which tty (so they don't just grow), but they are a form of logging. (And it's useful to be able to parse them cheaply, since various common commands like who do just that.) – Peter Cordes Oct 5 '16 at 3:34
  • 1
    @PeterCordes Very true. Again, well-defined data. structured records. And of course, speed and size at all scales were vital considerations back in those days. – SusanW Oct 5 '16 at 9:15
9

An example of a somewhat binary log is wide-spread: the Windows event log. On the pro side, this allows log messages to be quite wordy (and thus hopefully helpful) at virtually no cost, possibly something like

Warning: The queue of foobars to do has grown by 517 items over the last 90 seconds. If this happens about once per day, there is nothing to worry about. If it happens more often or in rapid succession, you may want to check the amount of RAM available to the foobar application. If it occurs together with event 12345, however, you seem to be using an obsolete database and you better call support at +1-555-12345 in order to prevent data loss.

The main part of this message exists only once as a resource installed with the application. However, if this resource is not installed correctly (for example, because meanwhile a newer version has been installed that no longer supports this obsolete message), all you see in the event log is a standard message that is just fancy wording for

Dunno, something with "517" and "90".

and no longer helpful in any way.

  • 9
    Not to mention that finding something in the Windows event log can be a nightmare. It certainly makes me long for a simple text file. – Michael Hampton Oct 6 '16 at 4:55
  • 4
    Wait. Did you want to see two (or more) log entries simultaneously? Well too bad. – Eric Towers Oct 7 '16 at 8:52
  • 2
    My answer was going to be "Windows event logs, enough said." – Craig Oct 8 '16 at 5:43
  • My experience of missing resources for the Event Viewer has been with tools that don't have resources to install, but in that case, AFAIR, there's still a line of actual info from the reporting program, at the bottom, after Windows finishes its 'the resource may be missing or corrupted" spiel. – underscore_d Oct 8 '16 at 12:25
5

The two main questions you would want to ask before choosing between text and binary are:

  • Who is my audience?
  • What content do I need to convey?

A common opinion is that the audience of a log message is a human being. This is obviously not a perfect assumption, because there are plenty of log crawling scripts out there, but it is a common one. In this case, it makes sense to convey the information in a medium which humans are comfortable with. Text has a long standing tradition of being this medium.

As for content, consider that a binary log must have a well defined format. The format must be well defined enough for other people to write software which operates on those logs. Some logs are quite well structured (your question lists several). Other logs need the ability to convey content in a less-well-defined natural language form. Such natural language cases are a poor match for binary formats.

For the logs which could be well described in binary, you have to make a choice. Because text works for everyone, it is often seen as the default choice. If you log your results in text, people can work with your logs. It's been proven thousands of times. Binary files are trickier. As a result, it may be that developers output text simply because everyone knows what that's going to behave like.

5

TL;DR: Size doesn't really matter, but convenience of use does

First of all, whilst comparing the respective advantages of text and binary formats for short-term log storage is an important question, the size does not really matter. The two reasons for this are:

  1. Logs are highly redundant information that will compress very well: in my experience it is not rare to see compressed log files whose size is 5% or less of the size of the original file. Consequently, using a text or a binary format should not have any measurable impact on the long-time storage of logs.

  2. Whatever format we choose, logs will quickly fill a server disk if we do not implement a “log files sink” that compresses and sends log files to a long-term storage platform. Using a binary format could slow this a bit but even a change by a factor 10 would not matter that much.

Text versus binary log formats

The promise of Unix systems is that, if we learn to use the standard toolset working on text files structured in lines – such as grep, sort, join, sed and awk – we will be able to use them to quickly assemble prototypes performing any job we want, albeit slowly and crudely. Once the prototype has demonstrated its usefulness, we can choose to turn it in a really engineered software to gain performance or add other useful features. This is, at least in my understanding, the essence of the Unix philosophy.

To put it another way, if we likely need to perform treatments and analyses we cannot figure out by today, if we do not know who should implement this analysis, etc. then we are in the stage where prototypes should be used and text formats for logs are probably optimal. If we need to repeatedly perform a small set of well-identified treatments, then we are in the situation where we should engineer a perennial software system to perform this analyse and binary or structured formats for logs, such as relational databases, are likely to be optimal.

(Some time ago, I wrote a blog post about this.)

4

Log files are in text format because they can be easily read using any type of text editor or by displaying the contents via console command.

However, some log files are in binary format if there is a lot of data. For example, the product I am working on stores a maximum of 15000 records. In order to store the records in the least amount of room, they are stored in binary. However, a special application must be written to view the records or convert them to a format that can be used (e.g. spreadsheets).

In summary, not all log files are in textual format. Textual format has an advantage that custom tools are not needed to view the content. Where there is a lot of data, the file may be in binary format. The binary format will need a (custom) application to read the data and display in a human readable format. More data can be packed into a binary format. Whether to use textual format or binary format is a decision based on the amount of data and ease of viewing the contents.

3

In embedded systems where I might not have an output channel available during run-time, the application can't afford the speed hit imposed by the logging, or logging would alter or mask the effect I'm trying to record, I've often resorted to stuffing binary data into an array or a ring buffer, and either printf()ing it at the end of the test run or dumping it raw and writing an interpreter to print it as readable. Either way, I want to end up with readable data.

In systems with more resources, why invent schemes to optimize what doesn't need optimizing?

  • 1
    Similarly, when trying to log in real-time from an embedded device to a PC over a 9,600 baud serial port, it is often advisable to compress data or use a binary format, to prevent overflows. – Mawg Oct 6 '16 at 17:34
3

Log files are intended to aid debugging of issues. Typically, hard drive space is much cheaper than engineering time. Log files use text because there are many tools for working with text (such as tail -f). Even HTTP uses plain-text (see also why don't we send binary around instead of text on http).

Additionally, it's cheaper to develop a plain-text logging system and verify that it works, easier to debug if it goes wrong, and easier to recover any useful information in case the system fails and corrupts part of the log.

  • 2
    Since it was brought up by someone else, I wanted to point out that HTTP/2 (look out!) allows for binary, bi-directional, multiplexed communications. Any developers who fancy themselves elite should go learn it real quick and then ask themselves why it didn't happen sooner. – Shaun Wilson Oct 7 '16 at 4:28
3

A corrupted text file is still readable around the corrupted part. A corrupted binary file maybe restorable, but it also might not be. Even if it is restorable, it would require quite a bit more work. The other reason is that a binary logging format makes it less likely that during a rush to create a "temporary fix" (aka "the most permanent of all fixes") the logging solution will get used instead of something which can be created quicker.

2

We count on unit testing for attaining and maintaining the robustness of our software. (Most of our code runs in a server, headless; post-operation analysis of log files is a key strategy.). Nearly every class in our implementation does some logging. An important part of our unit testing is the use of 'mock' loggers that are used when unit testing. A unit test creates a mock logger and provides it to the item being tested. It then (when useful/appropriate) analyses what got logged (especially errors and warnings). Using a text-based log format makes this much easier for much the same reasons that analyses performed on 'real' logs: there are more tools at your disposal that are quick to use and adapt.

  • 2
    although someone else downvoted, i would like to point out this kind of answer provides value still, it shows that text-based logs can be made useful at even the worst levels of the practice in ways your average programmer doesn't actually care, but should. +1 – Shaun Wilson Oct 7 '16 at 4:30
  • Thanks for the support comment. I try to provide info that I think will be useful to at least some people. It's what I want and expect when I go to SO. – Art Swri Oct 7 '16 at 15:35
2

Historically, Logs were official, hand-written and sequential records of events. When machinery became capable of recording events, these were written to a hard-copy output device such as a teletype printer, which produced a permanent sequential record but which could only process text, and occasionally ring a BELL ...

2

Back in my mainframe days, we used a custom-designed binary log format. The main reason wasn't to save space, it was because we wanted the log to occupy finite space by overwriting old entries with new ones; the last thing we wanted was to be unable to diagnose problems caused by disks becoming full (in 1980 disk space used to cost $1000/Mb, so people didn't buy more than they needed).

Now I still like the idea of a circular log file, and if operating systems offered such a beast I would use it without hesitation. But binary was a bad idea. You really don't want to have to waste time finding the right commands for deciphering a log file when you've got a critical problem to solve.

protected by maple_shaft Oct 5 '16 at 11:49

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.