There is a series of procedures, each of which falls into one of two general categories.
Most of these are of category 1: it would be best to execute them one at a time - the overhead associated with threading or even using a thread pool negates any gains.
However there are a few of category 2: it would be best to move these into a concurrency scenario, like a different thread, a task, etc. These procedures, unlike the others, do justify the overhead.
The series is made up of an unknown number and unknown order of both categories, other than the fact that most will usually be of category 1.
What are the known approaches to solving this problem? Here is a stab at some approaches, but I'm curious if there are others or if there is a comprehensive survey of the problem.
- Start all procedures sequentially, but determine mid-process if it should be moved to a separate concurrent process.
- Have a 'look-ahead' process running simultaneously that decides what category the procedure belongs to. It is referred to when a procedure is ready to be started.
Both of these ideas have a tradeoff (as I'm sure any approach will): the added intelligence costs something that adds to the overhead.