My goal is to understand how to handle jobs coming from the client side at very high frequency each job is either CPU-intensive or I/O intensive, but both types of jobs are continuously arriving at my server.

For example if ThreadPoolExecutor is configured with 50 threads and request frequency is 100 req/sec and each CPU-intensive job takes 2sec to complete and each I/O job is querying a data base and with 100 req/second first 50 jobs are CPU-intensive and remaining are mixture of CPU & I/O intensive. Then in this scenario all 50 threads woulb be busy executing CPU-intensive jobs and remaining clients have to wait a lot. I want to solve this problem by using distributed thread pool executor so that CPU-intensive jobs take advantage of cluster and go to other nodes for execution and my server just execute I/O intensive jobs. how can I design a server in this scenario?

  • Have a read on Lambda Architecture. It is relatively new so it may be the chance that not every university will teach it in a CS degree program. However it is very popular and taking over older paradigms in terms of being able to solve actual industry data processing requirements.
    – rwong
    Jan 7, 2015 at 16:53
  • Are these requests coming in continuously? If 50 CPU-intensive requests, each of which requires 2 seconds to complete, are coming in every second, by my reckoning you need 100 processor cores just to keep up. Jan 7, 2015 at 18:06
  • In any event, the architecture doesn't seem especially difficult. Just figure out what kind of request it is, and route it to a cluster that's optimized for that request type. Jan 7, 2015 at 18:07
  • 2
    In the future, please do not cross-post questions between Stack Exchange sites. If you posted on the wrong site or think that another site would be a better fit to get you helpful answers, flag your question for moderator review.
    – Thomas Owens
    Jan 7, 2015 at 19:58
  • 1
    @ahmadraza "How I would figure out" ... ask the job submitter? If the job submitter doesn't tell then what else can I do?
    – rwong
    Jan 7, 2015 at 20:34

1 Answer 1


In general, there is very little you can do anything with the information you gather from a task. The reason is this: once you start a task, you have to let it finish, no matter what.

  • You can't suspend it forever. Some other parts of code might be depending on the task result; suspending it means the dependent code will be waiting forever.
  • You can't kill it. The task might have been making changes in the file system or database; it might leave the system or datastore in an inconsistent state if you kill the task in the middle of execution.
  • For the same reason you can't kill it, you can't even move an executing task to another computer, or even to another thread.

Basically, once a task is executing, you have to let it finish, no matter much much resource it takes.

The only other remedial action you can take is that if you see a significant number of worker threads are:

  • Executing on some tasks;
  • Are not consuming much CPU time due to them being in the blocked state (by blocking I/O)

Then you can perhaps spin up a few more worker threads and give them new tasks, in the hope that some of them are CPU intensive and therefore recoup the "wasted CPU cycles".

Some thread pool designs simply try to opportunistically increase the number of worker threads (and have them run concurrently on tasks) until the point where they see diminishing marginal returns.


I can only give a rough overview of ideas. Feedback are welcome.

First, some definitions.

  • "CPU intensive" - tasks which, once started executing on a thread, will keep on executing on the CPU until finish, making continuous progress along the way, and rarely volunteer up for a "pause" (blocking operation).

  • "IO intensive" - tasks which, during its execution on a thread, tend to frequently request for a "pause" (entering sleep or a blocking operation) voluntarily. This "pause" typically occurs when the task is waiting for an external event to arrive, such as waiting for the data from reading a file, or from a database query.

  • "CPU wasting" - tasks which superficially look like the "CPU intensive" type, but is in fact "busy-spinning" - running a tight empty loop without making progress. It could be further divided down to "programming errors (software defect)" or "malicious workload".

Notice I use the word "volunteer" several times. This is because an operating system may preemptively suspend a thread as it context-switch to other threads so that the OS can support a total number of threads in far excess of the number of CPU cores. (Typical CPU cores are in single-digit or low ten's, but typical total number of threads in OS is in the hundreds to thousands.)

It is possible for a watchdog thread (with suitable administrative privileges) to query the OS for the CPU time consumed by each worker thread. This can be used to decide how "CPU intensive" a particular running task is.

Also, it might be possible for the watchdog thread to wake up with a regular interval to check what each worker thread is up to. If one wants to find out whether a thread spends most of its time being blocked (waiting for I/O), one will need to drill down to the reason of the thread's reason of suspension. As described before, one has to distinguish between threads that are preempted by the OS, versus threads that are blocked by its own IO operations. This is doable on some OSes, but may be impossible on some other OSes.

I have never used any of these techniques so I don't know how feasible or effective these techniques are.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.