There is a series of procedures, each of which falls into one of two general categories.

Most of these are of category 1: it would be best to execute them one at a time - the overhead associated with threading or even using a thread pool negates any gains.

However there are a few of category 2: it would be best to move these into a concurrency scenario, like a different thread, a task, etc. These procedures, unlike the others, do justify the overhead.

The series is made up of an unknown number and unknown order of both categories, other than the fact that most will usually be of category 1.

What are the known approaches to solving this problem? Here is a stab at some approaches, but I'm curious if there are others or if there is a comprehensive survey of the problem.

  1. Start all procedures sequentially, but determine mid-process if it should be moved to a separate concurrent process.
  2. Have a 'look-ahead' process running simultaneously that decides what category the procedure belongs to. It is referred to when a procedure is ready to be started.

Both of these ideas have a tradeoff (as I'm sure any approach will): the added intelligence costs something that adds to the overhead.

  • You want the code itself to determine whether it should run multithreaded or not? Mar 15 '19 at 16:29
  • @RobertHarvey I guess I'm thrown off by the question. If you could help me understand, what would be an example of having something besides the code itself determine? Mar 15 '19 at 16:33
  • Generally speaking, you make that decision yourself. You figure out what portions of the code will benefit from multithreading under heavy load, and you make them run multithreaded unconditionally. If you do it right, the overhead imposed by enabling concurrency is minimal, and it will still perform adequately under light load. Mar 15 '19 at 16:40
  • Beyond that, your strategy is going to depend heavily on what your specific needs are. There are frameworks like Akka that will scale up as your processing needs increase, but that's an Actor Model, a different approach than multithreading. Mar 15 '19 at 16:42
  • 1
    How much of a performance hit do you take on a single-threaded "series" if you just assume that it needs to be multithreaded and process it accordingly? Mar 15 '19 at 17:43

Depending on your specific needs, you should find an off-the-shelf solution that already performs this "automatic scaling" for you. These exist in abundance.

For example, Akka and Akka.net do this via an Actor model and a robust message-passing mechanism. You spin up as many actors as you need, and add more hardware as your needs grow.

Writing reliable concurrent programs using solely threads, mutexes and semaphores is hard. If you're using a threading model, take advantage of modern concurrency abstractions that take some of the guesswork out of threading and provide load balancing for free.

For example, Parallel.Foreach will only spin up as many threads as it needs, and you can limit the number of threads it uses to some maximum value.


I think you are conflating concurrency and multithreading. Most languages have a concept that provides concurrency without the huge overhead of allocating a separate thread for each task. Some of the names I've seen that concept go by are greenlets, fibers, futures, and promises. Their overhead is generally barely more than a virtual function call. I would try that before trying to munge a thread-only model to fit your needs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.