We are internally debating in our team if we should continue to run our production boxes with logs turned off completely and just log errors and exceptions with log rotations.

I want to know how everyone else logs info in their production environment and any particular strategy/tool that has worked well for you.

Thanks Isaq

  • 1
    If you have no logging at all, how do you work out what is wrong when something breaks currently? – Matthew Scharley May 5 '11 at 6:06
  • Why would you want to turn off logs, first of all ?? – Heandel May 5 '11 at 6:22
  • Currently we turn the logs on when we find an issue to troubleshoot. – Anonymous May 5 '11 at 6:23
  • 6
    What particular type of logging are you talking about? that is a vague term, really – Marc Gravell May 5 '11 at 11:03
  • The absolutely crucible thing is whether all issues you may encounter are reproducible later with loggin enabled. – user1249 May 5 '11 at 11:24

If you are confident that you'll be able to reproduce any error that occurs then by all means turn off the logs. However, it's rarely as simple as that and rare errors are often those that are the most difficult to reproduce. Having the logs will help you here.

If you are turning them off because they take up a lot of disk space then you should be looking at why they are so large. What exactly are you logging? Do you need to keep logs of everything or can you be more selective.

Another alternative is to rotate the logs more quickly so that each file covers a shorter time period and you delete archive the logs sooner as well.

| improve this answer | |

You will always need a heart-beat log to record the ups and downs of your application, and ensure that your application still alive.

You will also most likely want a transaction log allowing you to see the absolute basics of the application data flow.

You will most likely NOT want a full debug log running, as it may give quite an overhead.

| improve this answer | |

My current approach is to have production servers with logs turned on, but with certain level of verbosity. That verbosity level is finetuned to contain messages denoting significant actions within application. When some error is reported, those significant actions are helpful in figuring out context of the error. The benefit of such context is that it is easier to exactly understand the nature of the error and provide a real fix. Sometimes without precise sequence of events you won't even be able to pinpoint error in your code (stack traces don't necessarily point to the root cause of the problem).

We can also switch verbosity to so called "debug level", where every tiny meaningful detail is being logged. This is useful when we can't find an error on our part, but the application is not behaving correctly. It is activated for short periods of time and as you might expect, generates a lot of data to chew on.

We don't pay any attention to log sizes. Since logs are only used to track errors, they are removed after 14 days. This period is long enough for somebody to notice and report an error and for our team to check log files and provide fix.

| improve this answer | |

I think that you simply need a better logging scheme.

If there is a performance issue:

  • Buy a faster HDD, ideally dedicated to logging
  • Send the traces, in UDP or similar, to a server whose sole role is logging, you may split the logs onto several servers if the volume is that high
  • Use macros for logging, and strip the lower levels from the code in Release mode

If it's a storage issue:

  • Archive! We compress logs on the fly. The files are rotated often (50 MB uncompressed) and bear a timestamp so that they can be organized and searched
  • Move! Only keep the most recent files in the current directory, move the older files onto a cheaper disk
  • Remove! Log files are not meant to live forever.

In both cases, you might want to make an audit of your logs, it may be that your logging practices are not adapted to the environment you're evolving in.

For example, where I work we have servers that process a few hundreds of transactions per second. The logging scheme is adapted:

  • for a regular transaction (where everything went fine), you have a single line of log (at most two, one for the beginning and another for the end), summing up (minimally) what the transaction was about
  • for an erroneous transaction, you get a line (tagged ERROR) for each error that occurred (if several)
  • when we need to debug, this volume is multiplied by about 10 to 20
  • in local (development), this volume is multiplied by about 100 to 200 (the supplementary traces are removed by the preprocessor in Release mode)

Some of our most verbose applications spew 50MB files twice per second (which is only ~15MB once compressed). Their logs are kept for 14 days still.

When a client calls you, asking why the transaction that he did 3 days ago at about 14h or maybe 15h (local time for him, of course) failed, you just cannot answer without logs. And no he cannot reproduce the issue, the data have changed since then.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy