How do we sequence logs from different services in a system?

Question

So, if I am designing a system as a set of different roles (client, app-server, backend services 1 to n). Each of these roles will have their own logging mechanism. I can aggregate all the logs at one location, but how can I sequence the logs from different sources, to show an interleaved log?

I can make my own strategy. Timestamps can help but will not be 100% accurate. What is normally done in the industry?

Answer 1

Normally you view the logs independently. Too much log is a real problem. There is almost no reason to view an interleaved log. If you really need to log an "action" through all the services have them write to a central log

syslog is a great example

apache writes to one log mysql to another, and there's usually never a reason to interleave them. Your structure is not that different.

If you really want to interleave a log then your going to have to do so using "something central" i.e. syslog, one file, a database, etc, but you should treat the log as FIFO and not as ordered in any specific way.

Answer 2

At my current company we have a similar setup: public facing websites, a few API's for partners, and a whole bunch of background tasks and services. One thing I pushed for when I joined the team several years back was to log everything to one database table. Skeptical?

• Yes, the DBA was too. I convinced him :)
• Yes, we had to be careful of hampering the performance of our production DB with frequent (average ~4 per second) inserts of medium sized log records (average 100~2000 bytes each) -- we setup the table as leanly as possible and indexed it well to speed up queries.
• Yes, we had to setup an archive/deletion job to clean up old records (just like a good sysadmin would do on a server.)

The advantages are huge:

• The DB enforces a sequential ordering of log entries by a primary auto-increment integer key, so there is a canonical ordering to the list of log records, and of course a datetime field as well. (This addresses your specific question.)
• We defined our own schema that categorized records by system component, subcomponent, and even method name (via reflection and stack tracing in our logging methods).
• We can query the log according to specific patterns using SQL rather than grepping and concatenating a bunch of text files strewn about 10 different servers in 100 different folders.

Having worked with this logging setup for a few years, I would not willingly go back to logging to distributed text files, unless the system I was working on was very small.