Concurrency pattern of logger in multithreaded application

Question

The context: We are working on a multi-threaded (Linux-C) application that follows a pipeline model.

Each module has a private thread and encapsulated objects which do processing of data; and each stage has a standard form of exchanging data with next unit.

The application is free from memory leak and is threadsafe using locks at the point where they exchange data. Total number of threads is about 15- and each thread can have from 1 to 4 objects. Making about 25 - 30 odd objects which all have some critical logging to do.

Most discussion I have seen about different levels as in Log4J and it's other translations. The real big questions is about how the overall logging should really happen?

One approach is all local logging does fprintf to stderr. The stderr is redirected to some file. This approach is very bad when logs become too big.

If all object instantiate their individual loggers - (about 30-40 of them) there will be too many files. And unlike above, one won't have the idea of true order of events. Timestamping is one possibility - but it is still a mess to collate.

If there is a single global logger (singleton) pattern - it indirectly blocks so many threads while one is busy putting up logs. This is unacceptable when processing of the threads are heavy.

So what should be the ideal way to structure the logging objects? What are some of the best practices in actual large scale applications?

I would also love to learn from some of the real designs of large scale applications to get inspirations from!

======

EDIT:

Based on both the answers here is the question I am now left with :

What is the best practice about assigning loggers (logging queues) to the object: should they call some global_api() or should the logger be assigned to them in the constructor. When the objects are in some deep hierarchy this later approach becomes tedious. If they are calling up some global_api() it's kind of coupling with the Application, so trying to use this object in other application throws this dependency. Is there a cleaner way for this?

Answer 1

an acceptable way it to use the singleton logger which delegates the actual logging to its own thread

you can then use any efficient producer-consumer solution (like a non-blocking linked list based on the atomic CaS) to gather the log messages without worrying that it is an implicit global lock

the log call will then first filter and build the log message and then pass it to the consumer, the consumer will then grab it and write it out (and free the resources of the individual message)

Answer 2

Ratchet Freak's answer is what I initially thought of too.

An alternate method might be to give each of your modules its own producer-consumer queue and then have your logger mechanism scan these queues on its own thread.

This might end up being more flexible because you can switch out how many loggers you use -- you might have one for everything, one for each module, or divide the modules into groups and have one for each group.

Edit: Elaboration

(don't mind my C -- that's what you said you're coding in, right?)

So this idea is having a producer-consumer queue/list for each of your modules. Such a queue would probably be something like this:

enum LogItemType {INFO, ERROR};

struct LogItem
{
    LogItemType type;
    char *msg;
};

struct LogQueue {...}; // implementation details -- holds an array/list of LogItem

bool queueLogItem(log_queue *, log_item *);
bool queueHasItems(log_queue *);
bool queueIsFull(log_queue *);
LogItem *dequeueLogItem(log_queue *);

Each module will need to either initialize its own such queue or be passed one by the initialization code that sets up the threads etc. The init code should probably keep references to all the queues:

LogQueue *module_queues = {module_1_queue_ptr, module_2_queue_ptr, ... , module_N_queue_ptr};

setModuleLoggingQueue(module1, module_1_queue_ptr);
// .
// .
// .

Inside the modules you'd have them create LogItems and queue them for each message.

LogItem *item = malloc(sizeof(LogItem));
item->type = INFO;
item->msg = malloc(MSG_SIZE)
memcpy("MSG", item->msg);
queueLogItem(module_queue, item);

Then, you'd have one or more consumers of the queues which would take the messages and actually do the log writing in a main loop like so:

void loggingThreadBody()
{
    while (true)
    {
        for (i = 0; i < N; i++)
        {
            if (queueHasItems(module_queues[i]))
                writeLogItem(dequeueLogItem(module_queues[i]));
        }

        threadSleep(200);
    }
}

or something like that.

This would be flexible in allowing you to have different consumers of the queues, e.g.:

// For one logger:

LogQueue *module_queues = {module_1_queue_ptr, module_2_queue_ptr, ... , module_N_queue_ptr};

initLoggerThread(module_queues);


// -OR-
// For multiple loggers:

LogQueue *group1_queues = {module_1_queue_ptr, ..., module_4_queue_ptr};
LogQueue *group2_queues = {module_5_queue_ptr, ... , module_Nmin7_queue_ptr};
LogQueue *group3_queues = {module_Nmin7_queue_ptr, ... , module_N_queue_ptr};

initLoggerThread(group1_queues);
initLoggerThread(group2_queues);
initLoggerThread(group3_queues);

Note: I'm guessing you'd want to allocate the memory for a log message at queue time and deallocate it at consumption time (assuming your messages will contain dynamic content).

Another Edit:

Forgot to mention: if you're expecting a lot of activity on your module threads you might see if you can do the log writes asynchronously so things don't block.

Yet Another Edit:

You'll probably also want to put a timestamp as part of the LogItem; with the logger thread(s) going through the queues sequentially the log statements might get out of order from when they occurred chronologically in the modules.

And Yet Another Edit:

With this setup it would be fairly easy to pass all the modules the same queue and just having the consumer look at that one queue, which would take you back to Ratchet Freak's answer.

Geeze, will you stop editing!?:

Also you could have one queue for each group of modules and have a logger for each queue.

Ok, I'll stop now.