1

I've noticed my software severely degrades when the # of threads is substantially increased.

What I mean is that when I limit the # of threads, the performance is much better than when I just let them all run simultaneously.

My cpu is an i7-3940XM, so very fast for a mobile and still not too shabby compared to desktop i7s for an old processor. It is 4 core but has 8 logical cores. Windows 10.

The test case creates 65 threads and it takes almost 5 minutes to run. CPU is maxed out when this happens because the code is mostly all in-memory and the only resources it accesses somewhat frequently is a ram-disk.

But when I limit the # of threads that can run concurrently, performance drastically improves:

Threads means concurrent Threads in the image below, each time is for the same application that ran 65 total Threads, only the # of concurrent threads varied

enter image description here

So it seems that performance is best when the # of threads is close to the # of logical cores

The reason I'm posting though is I wonder if I need to investigate further if I have anything too "blocking" in my code, I don't really understand why when there is no cap on the # of simultaneous threads it slows down so dramatically.

Can anyone offer some thoughts?

update:

I did find some file write/read code I forgot about, and switched it off - so at 8 simultaneous threads it made no difference in time per thread but at 65 it dropped that down to 1.00 seconds avg per thread

Improve this question
6
  • 2
    Entire books have been written on multithreading: performance, pitfalls, optimization, etc. This is far too broad to be answerable. Even if you constrained the question, it would likely be a question of implementation anyway which is off-topic here.
    – user22815
    Commented Jun 9, 2016 at 18:48
  • 2
    If your thread is a long running computation, and there are no breaks in the processing, you are going to max out a processor (real or virtual) to complete the computation. Having more threads with long running computations will do the same thing until you run out of processors, which is why you are seeing a correlation between processing speed and the number of threads. Once your thread count exceeds the number of available processors, you are now having to context switch the threads in and out, which is a costly operation, slowing down your performance.
    – Adam Zuckerman
    Commented Jun 9, 2016 at 18:53
  • 1
    " I don't really understand why when there is no cap on the # of simultaneous threads it slows down so dramatically." Because other folks have different use cases than yours. You mention that you are doing very little IO. Other folks may need to do lots of blocking IO to different devices, so lots of threads might be useful.
    – Charles E. Grant
    Commented Jun 9, 2016 at 18:58
  • @CharlesE. Grant sure. that's why I mentioned it is a 100% CPU use-case, I'm sure things work a bit differently when the CPU is less taxed
    – ycomp
    Commented Jun 9, 2016 at 19:45
  • 1
    It's not clear in your example if each thread is doing the same amount of work across all the examples. For example does each thread always process 10,000 bytes of input, regardless of the number of threads? In that case, the test with 65 threads is doing 65x the work of the example with 1 thread, so it's no surprise at all that it takes longer, even with perfectly parallel code and no contention between the threads (you'd need 65 cores to have a chance at the same total latency).
    – BeeOnRope
    Commented Sep 19, 2016 at 16:48

2 Answers 2

Reset to default
5

Sounds like you're running into Context Switching issues. (The linked article talks about entire processes rather than threads, but the idea is similar) There is a very real cost incurred when a CPU switches from working on one thing to working on another.

As you've discovered, when the number of CPUs roughly matches the number of threads, the CPUs don't have to put down one bit of work to pick up and work on another one very often.

If you have "too many" threads, then the OS is going to try to make roughly equal progress on all of them at the same time. Since you don't have that many cores, it means that each core will pick up a thread, do a little work, save that work somewhere, pick up the next thread, and repeat. The "picking up" and "saving" adds up.

Threading is useful for keeping a UI alive and it can be very useful for I/O intensive work (where you spend a lot of time waiting for bits to arrive or depart). Once you're past "keep all the cores busy", it's not overly useful for speeding up CPU bound operations.

Improve this answer
4
  • ah, interesting .. I didn't know that it was expensive to switch threads - I'm sure this adding up adds to something because my loops are running many times in the program
    – ycomp
    Commented Jun 9, 2016 at 19:41
  • I'm not sure context switching overhead adds up to a 20-fold increase in execution time, however. My suspicion is simply that the program uses more memory when it has more threads, and is less efficient because of that. And at somewhere between 20 & 65 threads, it runs out of memory and/or available heap space and starts thrashing/running full GCs.
    – Jules
    Commented Jun 9, 2016 at 19:47
  • 1
    There is almost no way that context switching is causing this issue. I've looked at dozens of multithreading performance issues in the last decade, and in exactly zero of them has "context switching" alone been the primary issue. 20 years ago, when the cost of a context switch was high in relation to the work a thread could do in its quantum, and the cost of refilling caches as larger, it was a serious issue. Nowadays, even high levels of context switching are tolerable, as long as each thread is using its quantum.
    – BeeOnRope
    Commented Jun 9, 2016 at 21:54
  • High numbers of context switches are usually an indication of an application level synchronization issue instead.
    – BeeOnRope
    Commented Jun 9, 2016 at 21:54
2

As Dan Pichelman pointed out in his answer, it looks like you're running into large amounts of context switches. Here's what's going on in ASCII art form, on a simpler three core computer. I'll have context switches use one time period each, and do 24 time periods of work divided evenly among the threads. In order to avoid starving threads of runtime, there will be a context switch every three time periods to the next thread in the queue. If the queue is empty, the current thread will just keep running without a context switch.

With three threads (8 units of work per thread) we finish all the work at the end of the 9th time period:

Core  Time Period
        1  2  3  4  5  6  7  8  9
0     |CS|a0|a1|a2|a3|a4|a5|a6|a7|
1     |CS|b0|b1|b2|b3|b4|b5|b6|b7|
2     |CS|c0|c1|c2|c3|c4|c5|c6|c7|

With six threads (4 units of work per thread), things take a bit longer, thanks to all the context switching:

Core  Time Period
        1  2  3  4  5  6  7  8  9 10 11 12
0     |CS|a0|a1|a2|CS|d0|d1|d2|CS|a3|CS|d3|
1     |CS|b0|b1|b2|CS|e0|e1|e2|CS|b3|CS|e3|
2     |CS|c0|c1|c2|CS|f0|f1|f2|CS|c3|CS|f3|

And for completeness, here's just two threads (12 units of work per thread), taking longer yet with core 2 idle:

Core  Time Period
        1  2  3  4  5  6  7  8  9 10 11  12  13
0     |CS|a0|a1|a2|a3|a4|a5|a6|a7|a8|a9|a10|a11|
1     |CS|b0|b1|b2|b3|b4|b5|b6|b7|b8|b9|b10|b11|
2     |CS| i| i| i| i| i| i| i| i| i| i|  i|  i|

Note also that processors with hyperthreading or similar things, such as the two cores per module in some of AMD's processors have shared resources between threads on the same core, or on the same module. This could lead to one thread waiting for the other to finish using that resource. However, this does not appear to be a significant problem for you, if at all, since your 5 thread run (which should avoid most such conflicts) was slower than your 8 and 10 thread runs (which would be more likely to encounter such conflicts).

Improve this answer
5
  • is threading better implemented on core i series Intel CPUs than on AMD cpus? nice charts btw )
    – ycomp
    Commented Jun 9, 2016 at 20:52
  • @ycomp They do things differently. Basically, a core, as used by Intel (and, I think most everybody else) is capable of running a single thread without having to share any internal resources (upper level cache, memory, and IO devices are shared, but they're generally not considered part of a core anyway). Over in AMD land, a module is capable of running a single thread without sharing internal resources. Each module has two "cores" which, IIRC, share floating point resources. Those "cores" are less capable than Intel's cores, but more capable than Intel's hyperthreads. At least in theory.
    – 8bittree
    Commented Jun 9, 2016 at 21:08
  • 1
    Context switches take perahps 5000 nanoseconds, for threads within an application (the applicable case). Typical quantums are 10,000,000 nanoseconds or more, so the context switch overhead alone is not going to explain this regression. What could explain it is if the process is "thread swapping" because of synchronization - for e.g., due to blocking on a fair mutex.
    – BeeOnRope
    Commented Jun 9, 2016 at 23:05
  • @BeeOnRope Might be worth explaining that more in it's own answer.
    – 8bittree
    Commented Sep 16, 2016 at 20:04
  • Perhaps, but it would be kind of a "non-answer" as in: I don't think the cause is "context switches", but without more details I don't the real cause. In fact, frequent switching between threads is often the root cause of performance problems - but it almost always application driven switching - voluntary context switches - (e.g., lock contention), which is very different than involuntary context switching (i.e., OS time-slicing because the number of runnable threads is larger than the number of CPUs) which is I think that the two answers here are getting at.
    – BeeOnRope
    Commented Sep 19, 2016 at 16:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.