These are radically different approaches, each with its own sets of pros and cons, which you will most likely see panning out at a later development stage:
I have thought of two ways :-
- Create workers that are only for logging and use the ZeroMQ IPC transport
- Use Multiprocessing with a Queue
One way you could try is to have an additional logging-worker, as in approach 1. You could let your workers log to a memcache logging cluster, and the logging worker monitors the current resource load and upon deceeding a given resource load parameter, the worker logs to an IOPs limited device (e.g. harddisk) .
I also like Jonathan's approach with the caveat that I too mostly use Python 2.x, and that you would likely have to setup your own logging backend to really push the performance-envelope.
Correct me if I am wrong, but my take is that you are doing some really data-intensive task, with storage IOPs being your bottleneck.
A convenient way still would be to let the broker do the
brokerage logging - in the form as described- with all the disadvantages of a central broker instance.
For instance if the broker is in such high demand that it never gets some breathing room to write the memcached logs back to storage, you would need to take another approach.
You may ultimately end up with a brokerless model. That is with the workers managing their work among themselves. In a simple example, through a Distributed round-robin algorithm.