So, if I am designing a system as a set of different roles (client, app-server, backend services 1 to n). Each of these roles will have their own logging mechanism. I can aggregate all the logs at one location, but how can I sequence the logs from different sources, to show an interleaved log?

I can make my own strategy. Timestamps can help but will not be 100% accurate. What is normally done in the industry?


Normally you view the logs independently. Too much log is a real problem. There is almost no reason to view an interleaved log. If you really need to log an "action" through all the services have them write to a central log

syslog is a great example

apache writes to one log mysql to another, and there's usually never a reason to interleave them. Your structure is not that different.

If you really want to interleave a log then your going to have to do so using "something central" i.e. syslog, one file, a database, etc, but you should treat the log as FIFO and not as ordered in any specific way.

|improve this answer|||||

At my current company we have a similar setup: public facing websites, a few API's for partners, and a whole bunch of background tasks and services. One thing I pushed for when I joined the team several years back was to log everything to one database table. Skeptical?

  • Yes, the DBA was too. I convinced him :)
  • Yes, we had to be careful of hampering the performance of our production DB with frequent (average ~4 per second) inserts of medium sized log records (average 100~2000 bytes each) -- we setup the table as leanly as possible and indexed it well to speed up queries.
  • Yes, we had to setup an archive/deletion job to clean up old records (just like a good sysadmin would do on a server.)

The advantages are huge:

  • The DB enforces a sequential ordering of log entries by a primary auto-increment integer key, so there is a canonical ordering to the list of log records, and of course a datetime field as well. (This addresses your specific question.)
  • We defined our own schema that categorized records by system component, subcomponent, and even method name (via reflection and stack tracing in our logging methods).
  • We can query the log according to specific patterns using SQL rather than grepping and concatenating a bunch of text files strewn about 10 different servers in 100 different folders.

Having worked with this logging setup for a few years, I would not willingly go back to logging to distributed text files, unless the system I was working on was very small.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.