What's the best way to manage error logging for exceptions?

Question

Introduction

If an error occurs on a website or system, it is of course useful to log it, and show the user a polite message with a reference code for the error.

And if you have lots of systems, you don't want this information dotted around - it is good to have a single centralised place for it.

At the simplest level, all that's needed is an incrementing id and a serialized dump of the error details. (And possibly the "centralised place" being an email inbox.)

At the other end of the spectrum is perhaps a fully normalised database that also allows you to press a button and see a graph of errors per day, or identifying what the most common type of error on system X is, whether server A has more database connection errors than server B, and so on.

What I'm referring to here is logging code-level errors/exceptions by a remote system - not "human-based" issue tracking, such as done with Jira,Trac,etc.

Questions

I'm looking for thoughts from developers who have used this type of system, specifically with regards to:

• What are essential features you couldn't do without?
• What are good to have features that really save you time?
• What features might seem a good idea, but aren't actually that useful?

For example, I'd say a "show duplicates" function that identifies multiple occurrence of an error (without worrying about 'unimportant' details that might differ) is pretty essential.
A button to "create an issue in [Jira/etc] for this error" sounds like a good time-saver.

Just to re-iterate, what I'm after is practical experiences from people that have used such systems, preferably backed-up with why a feature is awesome/terrible.
(If you're going to theorise anyway, at the very least mark your answer as such.)

Answer 1

I have been in a project where with logged client errors using Microsoft Enterprise library. All exception where send to our mail box. In the mail subject we added hash code of serialized error for avoiding duplicated messages. One could of course store serialized messages in database and so on.

I recommend you to check out Microsoft Enterprise library and Log4Net.

Some Features of Log4Net

• Support for multiple frameworks
• Output to multiple logging targets
• Hierarchical logging architecture
• XML Configuration
• Dynamic Configuration
• Logging Context
• Proven architecture
• Modular and extensible design • High performance with flexibility

Answer 2

In the case of database applications, some kind of ID (like <TABLE>:<PrimaryKeyID>) that allows you to track the records in the database related to the scope where the exception was catched.

I've done it with Oracle and PL/SQL, recording the ID in a database table within the aplication, from the exception handler.

Answer 3

Much of what you describe (ie. the logging specific parts) are implemented in enterprise library as Amir Rezaei has noted. Everything else seems to be more of the analytics part (ie. what to do with logs afterwards).

In my case, I created some small apps and sql scripts that made some things easier. Here's some of the things that I really liked:

• Grouping same-errors together (ie. 100 users all experienced the same bug around the same time is 1 bug report with a note of how many occurrences there were)
• Auto-filing a ticket in the case tracker (never managed to make this 'at the click of a button' but always wanted to)
• Username of the user of the software (not just the machine, which is available with most loggers). In some cases, the automated user accounts caused problems whereas in others, specific users were the cause of problems. "I need to watch Mike do some work, he keeps causing a specific error."
• "User Actions" - I had a global stack that would keep a trace of every actionable click/button press as the user did it and had that tacked on to error logs. Reproducing the error was often a case of walking through that trace and performing the same steps as the user (I had hoped to build a CodedUI test generator that would parse the trace and perform the steps automatically, but never did)

Answer 4

Sometimes, the log information is just too voluminous to be stored on disk. One approach I have seen is to write your logging entries to a firehose (in, say, perl) something like this:

# Create socket.
my $sock = IO::Socket::INET->new(
    Proto       => 'udp',
    PeerAddr    => $bcastaddr,
    Broadcast   => 1,
) or die "Can't create socket ($bcastaddr): $!";

while (<>) {
    chomp;
    unless (/File\ does\ not\ exist:/) {
        $sock->send("$eventtype:$_") or warn "Can't send: $!";
    }
}

then an analyst can grep out what s/he wants to look at.

Answer 5

Here's some things I've learned from error monitoring in our applications:

• Being able to tail a rolling log file (I generally use log4net/log4j for logging in applications and BareTail to follow the log) is really useful for being able to check on the current health of a system
• For seeing when problems were introduced and the rate at which problems occur it's nice to have them in a database with timestamps to that you can run reports.
• The ability to send email/sms/voice alerts is super helpful in making sure systems stays up, but you have to have the ability to easily customize what types of errors that alert you. If you're getting 800 error emails a day your bound to miss the "Oh no the data center is on fire" one.

I've had great results for log4net because it makes it really easy to log to multiple places and make changes to the logging configuration easy as well.

Answer 6

elmah is an open source error logging system for ASP.NET apps and can be added to an existing system (using NuGet http://nuget.codeplex.com/) quickly and easily. It supports various backends and notification functions.

I don't know of anyone who's added it to a desktop app as it runs as a website but there's nothing preventing you from running it as a service and posting your exceptions to it through the web.

http://code.google.com/p/elmah/

ELMAH (Error Logging Modules and Handlers) is an application-wide error logging facility that is completely pluggable. It can be dynamically added to a running ASP.NET web application, or even all ASP.NET web applications on a machine, without any need for re-compilation or re-deployment.

Once ELMAH has been dropped into a running web application and configured appropriately, you get the following facilities without changing a single line of your code:

• Logging of nearly all unhandled exceptions.
• A web page to remotely view the entire log of recoded exceptions.
• A web page to remotely view the full details of any one logged exception, including colored stack traces.
• In many cases, you can review the original yellow screen of death that ASP.NET generated for a given exception, even with customErrors mode turned off.
• An e-mail notification of each error at the time it occurs.
• An RSS feed of the last 15 errors from the log...