2

I have a problem to solve that's very much like a thread pool, and I was hoping to hear some strategies or find some resources to information on managing the size of the pool.

Let's say I have the following:


public interface IWorkerBee : IDisposable
{
    Task Configure(WorkerBeeConfiguration config);
    Task<WorkItemResult> DoWork(WorkItemData workData);
}

public class WorkerBeeConfiguration 
    : Equ.MemberwiseEquatable<WorkerBeeConfiguration>
{ 
    /* initialization stuff */ 
}

public class WorkItemData
{
    public string CacheKey { get; set; }
    public TimeSpan ExpireAfter { get; set; }
    /* other per-work-item stuff */
}

public class WorkItemResult { }

Here are some facts about the system:

  • 50 to 100 WorkItemData's per second are submitted for processing.
  • Those are spread across a few hundred unique WorkerBeeConfiguration's.
  • IWorkerBee.Configure() is expensive - it takes several seconds and creates a new System.Diagnostics.Process.
  • If the result of IWorkerBee.DoWork() is not in the cache, it takes 200-300ms to create and involves inter-process communication with the IWorkerBee's child Process.
  • IWorkerBee can be re-used after it is configured, but DoWork is single-threaded and can only process one work item at a time.
  • It is ok to be aggressive with expanding a pool and keeping worker instances for a long time. System memory is the main constraint. Low latency and high throughput are necessary.
  • WorkItemData is provided by a large system that can take data from dozens of different sources. This code runs within the same process and can consume 10-20GB of memory for cached data. The servers should have at least twice as much memory as what the cache consumes for data.
  • The distribution of WorkItemData's to WorkerBeeConfiguration's varies over time with end-user usage and the addition and removal of available WorkerBeeConfiguration's. It's not uncommon for a large user to come online and noticeably change the distribution. I might know from historical logs that WorkerBeeConfiguration-A accounts for 3-4% of WorkItemData's and WorkerBeeConfiguration-B, C, D & E each account for 1-2%, but I don't want to rely on that information for sizing - I want it to be self-adjusting.

An IWorkerBee implementation would look like this:

public class WorkerBee : IWorkerBee
{
    private readonly AsyncLock Lock = new AsyncLock();
    private readonly ICache Cache;
    private System.Diagnostics.Process WorkerProcess; // each process commits 20-30MB of private, unshared working set memory

    // This is expensive - both creating and configuring the process - about a second each
    public async Task Configure(WorkerBeeConfiguration config)
    {
        using (await Lock.LockAsync()) {
            if (WorkerProcess == null)
                WorkerProcess = await CreateProcess(config);

            await ConfigureProcess(WorkerProcess, config);
        }
    }

    private Task<System.Diagnostics.Process> CreateProcess(WorkerBeeConfiguration config) 
        => TaskConstants<System.Diagnostics.Process>.Default;

    private Task ConfigureProcess(System.Diagnostics.Process process, WorkerBeeConfiguration config)
        => Task.CompletedTask;

    public void Dispose() { WorkerProcess.Dispose(); }

    // Only one item can be processed at a time. Each item takes 200-300 ms.
    public Task<WorkItemResult> DoWork(WorkItemData workData)
        => Cache.GetOrSetAsync(
            workData.CacheKey,
            workData.ExpireAfter,
            async () => {
                using (await Lock.LockAsync())
                    return await DoWorkInner(workData);
            });

    private Task<WorkItemResult> DoWorkInner(WorkItemData workData)
        => TaskConstants<WorkItemResult>.Default; // inter-process communication to perform work
}

I want to keep a pool hot for each unique WorkerBeeConfiguration in use (I doubt it will be necessary to completely destroy pools that fall out of use, but a more complete implementation would do that):

public class WorkerBeeProvider
{
    private readonly ConcurrentDictionary<WorkerBeeConfiguration, Task<IWorkerBee>> Colony
        = new ConcurrentDictionary<WorkerBeeConfiguration, Task<IWorkerBee>>();

    public Task<IWorkerBee> GetWorkerBee(WorkerBeeConfiguration config)
        => Colony.GetOrAdd(config, CreateAndConfigureWorkerBeePool);

    static private async Task<IWorkerBee> CreateAndConfigureWorkerBeePool(WorkerBeeConfiguration config)
    {
        var hive = new WorkerBeePool();
        await hive.Configure(config);
        return hive;
    }
}

Here's a simple pool implementation, but I haven't addressed resizing in response to demand:

public interface INextBeeStrategy
{
    Task<IWorkerBee> GetNextBee(List<IWorkerBee> hive);
}

public class WorkerBeePool : IWorkerBee
{
    private List<IWorkerBee> Hive;
    private INextBeeStrategy NextBee;

    public WorkerBeePool(int initialSize = 1, INextBeeStrategy nextBeeStrategy = null)
    {
        NextBee = nextBeeStrategy ?? new RoundRobin();
        Hive = new List<IWorkerBee>(initialSize);
        for (var i = 0; i < initialSize; i++)
            Hive.Add(new WorkerBee());
    }

    public Task Configure(WorkerBeeConfiguration config)
        => Task.WhenAll(Hive.Select(h => h.Configure(config)));

    public void Dispose()
    {
        foreach (var bee in Hive)
            bee.Dispose();
    }

    public async Task<WorkItemResult> DoWork(WorkItemData workData)
        => await (await NextBee.GetNextBee(Hive)).DoWork(workData);


    private class RoundRobin : INextBeeStrategy
    {
        private int _counter = -1;

        public Task<IWorkerBee> GetNextBee(List<IWorkerBee> hive)
            => Task.FromResult(hive[Interlocked.Increment(ref _counter) % hive.Count]);
    }
}

So, how might I go about making WorkerBeePool expand with demand, rather than queue up requests too far, without exhausting system resources and staying in balance with a few hundred other pools?

2

It sounds like you may have to do a little trial and error here, but a strategy that I would try is performing a bit of statistical analysis on the work that needs to be performed and using that information to increase/decrease number of threads as required.

In other words, you should have a thread which does nothing than occasionally (sleep often) poll the current number of items in your queue to process. If this number is higher than the previous number polled, consider bumping the number of active workers up one. If the number of items in your queue to process is lowering, consider lowering the number of active workers by one. While the number of workers should remain the same, simply when a worker finishes a job, you assign it to another.

By lower, I mean set the ideal number of worker threads down by one, and rather than create more workers, simply wait until one finishes naturally, and then let it become inactive. Should this number increase again before a job is finished, then by raising the number of worker threads, you've only returned to your previous number of worker threads and nothing changes.

Probably a good metric of whether or not you should be spawning a new worker thread is if the number of jobs increases or decreases by 10% with respect to the last time you created/destroyed a worker. This percentage of course is up to you to decide.

The factors which change here are:

  • The maximum number of workers allowed. Generally the performance advantage of having more threads than the number of CPU cores is zilch, so strongly consider setting it to the number of CPU cores. I wouldn't consider the "analysis" thread in this count, seeing how most of its time will be spent sleeping. This maximum number of workers should also take into consideration the amount of memory required for each thread to work, so if anything, perform a minimum between the max allowable by memory and the number of CPU cores.
  • Amount of time for each poll. This determines how quickly your analysis can change the number of worker threads, but it can also get rather taxing on the system if its too low.
  • Tolerance percentage. I said 10%, but perhaps 5% would be better suited if you want to respond to queue changes quickly. Keep in mind that a low tolerance may also mean constantly creating/destroying workers which is expensive, so there should be a good balance here.
  • Starting workers. Call this the "minimum" workers allowed if you will. How many threads would you want to start with?

With some testing, you'll probably achieve a good dynamic thread pool which increases on demand and decreases when demand is satisfied. The only thing that makes this particularly tricky is the time required to create/destroy a thread. If you find that this is particularly expensive, add a higher tolerance and that should do the trick. If not, consider boosting the number of threads created/destroyed each time to 2 rather than 1.

Good luck!

  • 3
    A good technique to avoid starting and stopping threads too frequently is to set min/max utilization thresholds for the pool. E.g. shut down idle threads if utilization drops below 60% and start new threads if utilization exceeds 90%. A more involved technique would track the rate of requests and try to ensure that the pool is large enough to handle requests with a particular probability, e.g. a 99% probability and then use a Poisson distribution to figure out the necessary pool size. – amon Jan 21 at 9:57
  • @Neil: regarding CPU cores, the work is mainly CPU-based, but can end up waiting on I/O. One of the biggest points I get mentally stuck on is how do I know when adding more instances no longer helps.. Also, they are processes, not threads. The queue should respond quickly to load pressure - this work must be done in the context of a live HTTP request and must have a reasonable response time. If one work item takes 250ms, having, say, more than 4 items in a queue waiting is not very acceptable. – quentin-starin Jan 21 at 16:16
  • @quentin-starin If a bottleneck is reached, then by definition no amount of threads/processes is going to make something faster, unless that bottleneck is CPU performance. If you're having I/O issues because of heavy disk access, you may consider setting up a RAID 0 (disk load operations can be done in parallel). For what concerns quick load pressure, have many starting threads. If that doesn't do the trick, don't even bother with the dynamic thread pool, simply start off with maximum threads. What are these processes? Like instances of tomcat? – Neil Jan 23 at 7:20
  • @Neil the processes are chromium. Each unique combination of page content (HTML/JS/etc) and viewport resolution would have its own pool of processes. Maximum threads, or peak number of items simultaneously needing processing by a single pool, could be up to 100.. multiplied by up to 300 pools. It sounds like a stretch to just go with max threads, but I suppose I can't say it wouldn't work until I try. I'm also still experimenting with the effects of various chromium process models and options. – quentin-starin Jan 24 at 4:06

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.