2 Slightly more idiomatic English
source | link

TL;DR: Size doesn't really matter while, but convenience of use does

First of all, whilewhilst comparing the respective advantages of text and binary formats for short-term log storage is an important question, the size does not really matter. The two reasons for this are, that:

  1. Logs are highly redundant information that will becompress very well compressed,: in my experience it is is not rare to see compressed log files whose size is 5% or less of the size of the original file. Consequently, using a text or a binary format should not have any measurable impact on the long-time storage of logs.

  2. Whatever format we choose, logs will quickly fill a server disk if we do not implement a “log files sink” that compresses and sendsends log files to a long-term storage platform. Using a binary format could slow this a bit but even a change by a factor 10 would not matter that much.

Text versus binary log formats

The promise of Unix systems is that, if we learn to use the standard toolset working on text files structured in lines – such as grep, sort, join, sed and awk – we will be able to use them to quickly assemble prototypes performing any job we want, yetalbeit slowly and crudely. Once the prototype has demonstrated its usefulness, we can choose to turn it in a really engineered software to gain performance or add other useful features. This is, at least in my understanding, the essence of the Unix philosophy.

To put it another way, if we likely need to perform treatments and analyses we cannot figure out by today, if we do not know who should implement this analyseanalysis, etc. then we are in the stage where prototypes should be used and text formats for logs are probably optimal. If we need to repeatedly perform a small set of well-identified treatments, then we are in the situation where we should engineer a perennial software system to perform this analyse and binary or structured formats for logs, such as relational databases, are likely to be optimal.

(For a longSome time ago, I have writtenwrote a blog post about this.)

TL;DR: Size doesn't really matter while convenience of use does

First of all, while comparing the respective advantages of text and binary formats for short-term log storage is an important question, the size does not really matter. The two reasons for this are, that

  1. Logs are highly redundant information that will be very well compressed, in my experience it is is not rare to see compressed log files whose size is 5% or less of the size of the original file. Consequently, using a text or a binary format should not have any measurable impact on the long-time storage of logs.

  2. Whatever format we choose, logs will quickly fill a server disk if we do not implement a “log files sink” that compresses and send log files to a long-term storage platform. Using a binary format could slow this a bit but even a change by a factor 10 would not matter that much.

Text versus binary log formats

The promise of Unix systems is that, if we learn to use the standard toolset working on text files structured in lines – such as grep, sort, join, sed and awk – we will be able to use them to quickly assemble prototypes performing any job we want, yet slowly and crudely. Once the prototype has demonstrated its usefulness, we can choose to turn it in a really engineered software to gain performance or add other useful features. This is, at least in my understanding, the essence of the Unix philosophy.

To put it another way, if we likely need to perform treatments and analyses we cannot figure out by today, if we do not know who should implement this analyse, etc. then we are in the stage where prototypes should be used and text formats for logs are probably optimal. If we need to repeatedly perform a small set of well-identified treatments, then we are in the situation where we should engineer a perennial software system to perform this analyse and binary or structured formats for logs, such as relational databases, are likely to be optimal.

(For a long time, I have written a blog post about this.)

TL;DR: Size doesn't really matter, but convenience of use does

First of all, whilst comparing the respective advantages of text and binary formats for short-term log storage is an important question, the size does not really matter. The two reasons for this are:

  1. Logs are highly redundant information that will compress very well: in my experience it is not rare to see compressed log files whose size is 5% or less of the size of the original file. Consequently, using a text or a binary format should not have any measurable impact on the long-time storage of logs.

  2. Whatever format we choose, logs will quickly fill a server disk if we do not implement a “log files sink” that compresses and sends log files to a long-term storage platform. Using a binary format could slow this a bit but even a change by a factor 10 would not matter that much.

Text versus binary log formats

The promise of Unix systems is that, if we learn to use the standard toolset working on text files structured in lines – such as grep, sort, join, sed and awk – we will be able to use them to quickly assemble prototypes performing any job we want, albeit slowly and crudely. Once the prototype has demonstrated its usefulness, we can choose to turn it in a really engineered software to gain performance or add other useful features. This is, at least in my understanding, the essence of the Unix philosophy.

To put it another way, if we likely need to perform treatments and analyses we cannot figure out by today, if we do not know who should implement this analysis, etc. then we are in the stage where prototypes should be used and text formats for logs are probably optimal. If we need to repeatedly perform a small set of well-identified treatments, then we are in the situation where we should engineer a perennial software system to perform this analyse and binary or structured formats for logs, such as relational databases, are likely to be optimal.

(Some time ago, I wrote a blog post about this.)

1
source | link

TL;DR: Size doesn't really matter while convenience of use does

First of all, while comparing the respective advantages of text and binary formats for short-term log storage is an important question, the size does not really matter. The two reasons for this are, that

  1. Logs are highly redundant information that will be very well compressed, in my experience it is is not rare to see compressed log files whose size is 5% or less of the size of the original file. Consequently, using a text or a binary format should not have any measurable impact on the long-time storage of logs.

  2. Whatever format we choose, logs will quickly fill a server disk if we do not implement a “log files sink” that compresses and send log files to a long-term storage platform. Using a binary format could slow this a bit but even a change by a factor 10 would not matter that much.

Text versus binary log formats

The promise of Unix systems is that, if we learn to use the standard toolset working on text files structured in lines – such as grep, sort, join, sed and awk – we will be able to use them to quickly assemble prototypes performing any job we want, yet slowly and crudely. Once the prototype has demonstrated its usefulness, we can choose to turn it in a really engineered software to gain performance or add other useful features. This is, at least in my understanding, the essence of the Unix philosophy.

To put it another way, if we likely need to perform treatments and analyses we cannot figure out by today, if we do not know who should implement this analyse, etc. then we are in the stage where prototypes should be used and text formats for logs are probably optimal. If we need to repeatedly perform a small set of well-identified treatments, then we are in the situation where we should engineer a perennial software system to perform this analyse and binary or structured formats for logs, such as relational databases, are likely to be optimal.

(For a long time, I have written a blog post about this.)