4

I came across below statement at this blog

Computation intensive operations should use a number of threads lower than or equal to the number of cores, while IO intensive operations like copying files have no use for the CPU and can therefore use a higher number of threads

I am not getting the logic/reasoning behind why number of threads can be greater than cores IO intensive operation while should be lower in case of Computation intensive operations?

6

Threads compete for some resource. If more threads want the resource than you have resources available, they are just blocking each other and you have overhead from task switching. The question is, what kind of resource is involved here?

For CPU-bound threads the resource is usually one CPU core. In some cases cores share some resource, e.g. caches or FPUs. In particular Intel's Hyperthreading often doesn't help with CPU-bound threads. So you might stop seeing speedups long before you are “using” all your cores.

For I/O-bound threads, the question is what precise kind of I/O is involved here. If the contested resource is a hard disk drive, then more than one concurrent access will slow things down. If the contested resource is a network interface, then any number of threads can access it without any problems as long as the total bandwidth of the interface is not exceeded.

For I/O bound threads the number of CPU cores is completely irrelevant. In fact you don't need multiple threads to do parallel I/O: asynchronous I/O and event-based systems are entirely sufficient here.

12
  • When you say For CPU-bound threads I believe you mean thread which needs cpu cycles like some processing/calculation etc ? You also said For I/O bound threads the number of CPU cores is completely irrelevant Consider an example where I have four core CPU and I need to write eight separate files on disk. My earlier understanding was at a time only four threads parallely can write four files as there are four cores(basically cores work as infrastructure for running thread). But I believe you are saying 8 threads can parallely write eight separate files as they do not need any cores at all Jul 26 '18 at 13:21
  • I thought even IO operation like write/read need core as there is some processing involved here. Jul 26 '18 at 13:25
  • 1
    @amon "I'm saying that writing four or eight files at the same time won't be faster than writing one file at a time." - That's quite often wrong, even if you have only one hard disk. Giving more work to the disk controller may be advantageous as it can optimize disk seeks (on a HDD). Some time ago I achieved a speed-up of factor of nearly 3 using 3 threads (though it didn't scale any further).
    – maaartinus
    Jul 27 '18 at 2:44
  • 1
    @maaartinus You do have a point there. AFAIK: A hdd typically keeps a queue of pending operations. If the queue is empty the disk can't do useful work. So when accessing small files there's no drawback in using multiple threads to keep the drive busy. For large files this is not the case: disks love sequential access patterns , and any concurrent accesses will disrupt this pattern. These seek times are similar to increased context-switching overhead on a CPU. Does my understanding match your experience?
    – amon
    Jul 27 '18 at 11:04
  • 1
    @amon I didn't make tests with large files, but what you're saying makes sense. However, I'd hope that the controller is smart enough to handle a couple of requests efficiently.
    – maaartinus
    Jul 27 '18 at 14:36
1

The prescription about number of threads has nothing to do with the logic of your application. It may well make sense to have MANY more threads than cores in your application, regardless of whether they are compute bound, or I/O bound.

The reason for the prescription is that when you have more compute bound threads than cores, the extra compute bound threads will be waiting, and not making forward progress on their computation.

Consider having T=20 threads, and C=8 cores. Consider your run-q (the list of available threads to run). At most 8 can be running, and the rest will be sitting around in the queue waiting for their chance to run.

If a large percentage of your threads were I/O bound (or blocking on anything really) - then they won't be 'ready to run' anyhow, and won't affect your CPU utilization.

If you think of the job of your application is to keep all the CPU cores hot (running fully at 100%), there is no gain in CPU utilization by adding more compute intensive threads after you have at least enough (compute intensive) threads for each (logical) core.

NOTE - thats the thought process behind this theory (and probably the blogger). It's really not quite true. There are MANY more factors that come into account in deciding how to get the most compute out of your processor. One consideration is that it SLOWS DOWN each core to have all the cores operating. So EVEN up til the point where you have enough compute threads for each core, adding more compute threads can slow you down as much as speed you up.

And there is significant CPU overhead in managing the threads (and can be memory locality issues, depending on how you allocate/manage your threads).

ADVICE/SUMMARY So - personally - I'd advise worrying less about this sort of reasoning, and more about the logical structure of your application - what makes sense for it and how to make it work. Use thread pools. And make their sizes configurable.

My $0.02.

3
  • 1
    As you said Consider having T=20 threads, and C=8 cores. Consider your run-q (the list of available threads to run). At most 8 can be running, and the rest will be sitting around in the queue waiting for their chance to run. Is it true for IO operation also. Consider I need to write 16 separate file on disk, can 16 threads run in parallel or still max 8 threads can run in parallel depending on CPU cores ? Similarly Consider I need to open 16 separate web sites from my java program, can 16 threads run in parallel or still max 8 threads can run in parallel depending on CPU cores ? Jul 26 '18 at 13:53
  • The short answer is, if they are blocked in I/O they can all run in parallel. The slightly longer answer is, they aren't actually running. THREADs is a great programming model for software developers to use. But the operating systems use lower level and more efficeint means. The IO you do gets queued, and the threads get marked as 'blocked on io'. Logically, they are REMOVED from the RunQ (or marked as blocked for IO - amounts to the same thing). They are 'running' in the sense that their IO is till happening as fast as it can. But they aren't competing to be scheduled for the CPU. Jul 26 '18 at 13:57
  • If you are just blocking on network IO, you can start hundreds of threads per CPU you have available, and have them do web requests (blocking network IO) and you probably wont see your CPU max out. There IS alot of cost to doing this, and you should consider other paradigms if you care about efficiency (you can start a request and not block on its results for example - async requests). Jul 26 '18 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.