Strategies to analyze collected exceptions

Question

We want to add error feedback to our application. I had a look at existing solutions (e.g. raygun.io), but these work "in the cloud", which is a no-go for us:

• most installations of our application are offline, so we will need to export the collected logs to a file and then send it per email.
• I don't feel comfortable uploading exception details to a third party server because of privacy concerns. (some of our clients would probably kill us).

Now, just logging these exceptions is not that hard. I have attached handlers to AppDomain.UnhandledException and AppDomain.FirstChanceException and then write them out to a small SQLite DB.

But the question for me is how to best analyze them. Analyzing them by hand would cost too much time, so we need to at least be able to group them into "buckets" and order them by number of occurrences, etc.

I do have some ideas in mind where I compare callstacks and messages, etc, but I feel like someone else MUST have done the same already.

Do you know of good strategies to classify exceptions and group them?

Answer 1

Collecting exceptions

When collecting exceptions, be careful with two things:

• What will happen if an exception occurs, and while reporting it, the reporting mechanism throws another exception?

  The worst case is to start reporting the new exception, which may trigger a new one, resulting in thousands of new exceptions thrown in a loop. You absolutely need to avoid that.

• What if you can't report an exception?

  Sending an exception to a server may fail because the server is not responding. Storing an exception to Windows Events may fail because of the lack of permissions. Saving the exception to a local database may fail because the database is, for some reason, down. Even appending the exception to a text file can be problematic: you may simply be out of space or lack the permissions.

  Usually, you need to have two or three ways to log the exceptions, doing a fallback from the one which failed to the one which should be more reliable (say report to a server; database; simple text file).

  It is essential, once the original problem which prevented the exceptions from being logged properly is solved, to be able to retrieve the exceptions from all those sources and to analyze them carefully. For example, how do you collect the exceptions which were stored in a text file when both the network and the database were down?

When it comes to sensitive data, be sure to encrypt the exceptions when sending them to a third-party server (or even your servers), in order to prevent access to stack trace to unauthorized persons. Make sure that developers never include sensitive information in the messages of the exceptions. For instance, AccessDeniedException shouldn't have as message:

Failed to authenticate the user John: the password "sihjdfogdhf" is invalid.

Among the information you collect, make sure you have:

• The exact version of the application. If you don't have the version, debugging an exception may quickly become a nightmare.

• The information about the environment. An application which throws an exception on Windows XP won't necessarily throw it on Windows 8.1. An application which throws an exception with FAT32 system may work well with NTFS. An application may behave strangely because there is less than 1 MB of disk space left, or because the RAM is full and the OS has to swap everything, leading to slow response rates and lots of timeout exceptions.

• Something which identifies the customer (if appropriate). This makes it possible to contact the customer to ask for more information.

Analyzing exceptions

The most important step is to deduplicate information about the exceptions you collect. Many exceptions will be similar, but not identical: it is crucial to identify how close they are and to group them. The goal is to be able to focus on exceptions which are the most frequent: if you have an exception which affected one user twice for the past year, and an exception which affected thousands of users thousands of times for the same period of time, the first exception may not be your top priority.

You probably won't get the grouping right from scratch; instead, study the exceptions you already gathered, try a grouping approach and see how it's working, then modify it until you get what you need. The grouping is tricky, because there are practically no absolute duplicates, and because the gap between what makes it a duplicate and what doesn't is fuzzy.

For instance:

• If you receive two AccessDeniedExceptions reported at login window button OnClick event, one concerning John, another one concerning Mary, those may be grouped, despite the different message strings.

• On the other hand, there may be another AccessDeniedException concerning Mary which happens somewhere else in the application, and which shouldn't be grouped with the first two.

• Now there is an AccessDeniedException thrown by a different revision of your app: the stack trace is different, but the exception is still the same than the first two.

• etc.

When grouping, make sure you keep the individual information as well. For example, the report may tell that there were 94 AccessDeniedExceptions thrown for the past day for users John, Mary and 14 others in versions 4.0.17 and 4.0.18 with the given stack traces by customers 1, 2, 3 and 4.

Answer 2

I have no detailed solution for this but my approach would be to make sub classes of the exceptions you catch and store + analyse and implement some information about the exception type.

public enum ExceptionDanger
{
    Low,
    Middle,
    SecurityCritical
    // And so on
}

Now we add this to the derived exception class:

public class MonitoredException : Exception
{
    public readonly ExceptionDanger danger;

    public MonitoredException(string message, ExceptionDanger danger)
        : base(message)
    {
        danger = danger;
    }
}

And in the AppDomain.FirstChanceException event subscriber we use the information to build a new log info (same for AppDomain.UnhandledException):

static void FirstChanceHandler(object source, FirstChanceExceptionEventArgs e)
{
    var monitoredException = e.Exception as MonitoredException;
    if(monitoredException != null)
    {
        var exDanger = monitoredException.danger;
        var exType = monitoredException.GetType().Name;
        var exTimestamp = DateTime.Now;

        // Build a package like a struct or so and exDanger, exType and exTimestamp to it
    }
}

Now when the server receives the package (that consists out of exDanger and exType) the data can be used for storing how many errors of this type were thrown and if you add a user ID to the package you can specify which users threw the exceptions. The ExceptionDanger enum is to sort the importance of the exception serverside.

I hope this gave you tips or at least an idea on how to sort and analyse exceptions.