Create machine-readable logfiles

Question

I'm currently developing a new device, which produces (text-)logfiles. There will be many devices of this type and I want to analyse the logfiles (for error-detection and statistics).

Until now, the log analysis was a side-product, logfiles of other devices were primary for peoples who tried to debug, so the information were human-readable ("fulltext"). With my automatic loganalysis-tool, I analysed these logfiles by searching for multiple keywords, which indicated interesting lines of the logfile. So the analysis was very intense, because I had to check one line for multiple keywords, and it's possible, that at the end, a line is irrelevant for my analysis. In case that the line is relevant for me, I sometimes need to parse the line costly by cutting out information at different positions in the line. Altogether, the analysis of these logfiles is very slow.

With the new device, I like to implement more machine-readable logfiles (but still human-readable), what enables faster and easier analysis, also during implementation/extension of my log-analysis tool. I'm just wondering, whats the best practice? My first idea is, to use a "trigger char" in a log message. We are in Germany, so I choosed the Dollar-sign, because we don't use it regularly. When I find this trigger-char, I know that I have to analyse this line (and I avoid scanning for multiple keywords). In addition, I thought that the most common case is a key-value-pair, which can contain the most important information of a log message. Just in case, that I have multiple values (e.g. a list), I need also a machine-readble version of this. So my next idea was, to combine the trigger-char with a JSON-object. In the standard case, this only includes one key-value-pair and the loganalysis-tool will parse this for performance-reasons with simple string-operations and don't use a JSON-parser-library. But for the case that there is a list, I'll create a log message with a JSON-Array and parse this with a JSON-lib (in my case json-simple). On events, I'll insert a boolean (the "true" is not neccessary, just to stay JSON-conform).

So at the moment, a line looks like this:

05.01.2018 11:11:23: No new APN needed. ${"currentAPN":"m2m-net.sa.t-mobile"}
05.01.2018 11:11:51: can't open gpio 969. ${"openGPIO969failed":true}

Can I improve my concept in any way?

Side-note: I'm using a self-written Java-parser to analyse the logfiles of the older devices, which I would extend for the new project. Because of the very special log-message-"formats" (which grew by time), something like Logstash didn't work. Nevertheless, with ElasticSearch and Kibana I use the rest of the ELK-stack for my log-analysis and -visualization.

Side-note 2: The new device is a a kiosk-device with Android, so only one app running. We have full control over the system (device-owner) and also most parts of the hardware.

/edit: extended by event-message as mentioned in first comment /edit 2: new side-note

Answer 1

I've have had a lot of success using a "JSON-lines" format, where each line in the log file is a JSON object, e.g.

{"timestamp": "2018-01-05T11:11:23.000Z","Message": "No new APN needed.","currentAPN": "m2m-net.sa.t-mobile"}
{"timestamp": "2018-01-05T11:11:51.000Z","Message": "can't open gpio 969.","openGPIO969failed": true}

This format lends itself nicely to being indexed using log aggregation tools (we use ELK), while still being relatively human readable. There are also a bunch of libraries which write this format (at least there are in the .Net ecosystem) - I'm a big fan of Serilog at the moment.

Answer 2

You could log to some sqlite database (or find some specialized logging library, there are many of them).

You could decide for a more structured format of your log file. For example, use JSON, YAML, S-expressions, perhaps prefixed with some numerical code.

You should define and document, if your log file is textual, its file format, perhaps by specifying it in some EBNF notation.

Altogether, the analysis of these logfiles is very slow.

If your format syntax is well designed and regular (e.g. LL(1) grammar), parsing it should be quite fast.

For instance, if you use JSON notation for some parts of your log file, you could be sure that TAB characters are never used (since in JSON you'll encode them as \t), then the tab could be used as some field separator.

Try to use existing formats when possible (from your industry; perhaps there are some already). Otherwise, define and document well some open format (that even your competitors would want to use). If possible, make your logging functions into some open source software library.