Tracing and Logging per item instead of per application

Question

I'm building a service in .NET that process "things" in the background and I'd like to trace the execution of these things per item, not per application.

Most logging frameworks (I'm familiar with System.Diagnostics.TraceSource and NLog) work in a way that you generate messages for everything and decide in a configuration file what and where to store using filters.

But in this case, I may be processing tens of thousands of items at any given time, and I just need to log information for one of them. The amount of log information I generate per item is huge, and it is not just a string with some parameters, so I'd rather avoid unnecessary processing and IO.

I'm already using NLog for application exceptions and things like that, so for this specific use I don't require log levels (info, errors, warnings, exceptions). All entries are "information entries" like what data is being generated or which decision was taken by the steps, so we can analyze what is going on for specific items in production.

One last note is that this information will not be stored on a file, but will be buffered in memory and sent via web sockets to a listening service. If the connection is not up, we just throw away the data.

I already have some flags per item I process in a context object, like:

interface IItemContext {
    Guid ItemID { get; }
    bool StepXLoggingEnabled { get; }
    bool StepYLoggingEnabled { get; }
    bool StepZLoggingEnabled { get; }
}

My current take on this is to have a bunch of extension methods on the Context object that decide if logging should happen and extract the information needed and delegate to another service.

internal static class ItemContextLogExtensions {
    public static void LogStepX(this IItemContext context, string data1, Foo data2) {
        if (context.StepXLoggingEnabled) {

            // do some work to extract/generate information relevant to my logging
            var entry = new LogEntry {
                Message = "StepX ...",
                Details = ""
            };

            SomeLogService.Current.Log(entry);
        }
    }
}

This way I can control when the log needs to be generated and can delegate to NLog during development if needed.

I keep thinking about just using NLog or TraceSource directly in the IItemContext, but while I don't want to reinvent the wheel, I wonder if there is any sense in trying to use them for this at all. It seems to me that in this case, these frameworks it will end up being just a glorified concurrent queue with a lot of methods I don't have use for.

What do you think? Is my take a good solution? Am I missing something? What could be other options to achieve this?

And am I missing some feature from NLog or some other .NET logging library that is specifically geared toward this kind of logging that can be enabled per item instead of per application?

Answer 1

This screams out for a Log4Net custom appender. You could write your own, have it check the active item to determine whether to log or not, and log to a per-item memory buffer.