8

Not almost, but all modern CPUs have multiple cores, yet multithreading isn't really that common. Why to have these cores then? To execute several sequential programs at the same time? Well, when calculations are complex (rendering, compiling), the program is definitely made to use advantage of multiple cores. But for other tasks a single core is enough? I understand that multi-threading is hard to implement and has drawbacks if number of threads is less than expected. But not using these idle cores seems so irrational.

8
  • 4
    Clear case of YAGNI in most cases. So why bother and overcomplicate simple stuff? Jul 11, 2020 at 7:45
  • 7
    Maybe you can give an example of somewhere you think multithreading should be? Jul 11, 2020 at 7:52
  • 1
    Hard is relative. Adding in openMP is not per se difficult, but not every task benefits from parallelism. Sequential steps by their very nature cannot be parallelised if they depend on each other and depending on the task there are parallelisation limits. (The sweet spot for a code I wrote in the past seemed to be about 6 CPU cores.) GPU computing a good example here: A GPU can be faster or slower than the CPU for a task, depending on the type of task. See here for example: xcelerit.com/computing-benchmarks/insights/…
    – DetlevCM
    Jul 11, 2020 at 18:05
  • 1
    To add to @DetlevCM's remark about “not every task benefits from parallelism”: the maximal speedup can be calcualted by Amdahl's Law. For many programs there are few sections that can be truly parallelized.
    – amon
    Jul 11, 2020 at 19:18
  • 6
    No offense, but have you tried it? For a non-trivial project? Jul 11, 2020 at 20:45

9 Answers 9

29

The proliferation of multi-core CPUs is predominantly driven by supply, not by demand.

You're right that many programmers don't bother decomposing their systems so that they can profit from multiple threads of execution. Therefore the customer sees a benefit mainly from OS multiprogramming rather than program multi-thread execution.

The reason CPU vendors create more and more cores is that the traditional way of increasing performance - increasing clock speed - has run into fundamental limitations of physics (both quantum effects and thermal problems). To keep producing chips that can be credibly sold as offering more compute power than last year's chips, they put more and more independent cores into them, trusting that OS multiprogramming and increasing use of multi-threading will catch up and yield actual rather than just nominal gains.

But make no mistake, both designing and exploiting multi-core CPUs is a lot harder than just running the same code faster. Both programmers and chip manufacturers would very much like to just keep increasing the speed of their chips; the trends towards parallelization is largely a matter of necessity, not preference.

11
  • 2
    Relativistic effects?
    – Mark H
    Jul 11, 2020 at 19:37
  • 4
    @MarkH: Speed of light = speed of electricity = finite, meaning that propagation delay over long wires within one core are a problem for high clock speeds, as well as for power. Jul 11, 2020 at 19:40
  • 2
    Valid points, @PeterCordes, however the most relevant bottleneck isn't the distance inside the processor - the by far (!) most significant bottleneck is the simple truth that main memory is slow as farts and L1 cache a very scarce resource... Jul 11, 2020 at 20:20
  • 3
    Isn't it more quantum effects than relativity? You can't really scale down to single atom connections and separation, because you get quantum effects. I think signals don't currently move at light speed, which is why they're working on making light-based chips. (Light's pretty big compared to atoms though)
    – Mark
    Jul 11, 2020 at 21:03
  • 5
    Physicist here, Mark H. is right. Relativity theory (Special and General) deals with the slowing of time and the curvature of space. The finite speed of light can be dealt with in Maxwell's classical equations. PN junctions are very much Quantum Mechanics, that's the chief non-classical physics on any chip.
    – MSalters
    Jul 11, 2020 at 21:20
41

Why multithreading isn't everywhere?

Because …

I understand that multi-threading is hard to implement and has drawbacks if number of threads is less than expected.

2
23

Why multithreading isn't everywhere?

Frame challenge: but it is everywhere.

Let's see, let's name some platforms:

  • Desktops/laptops: one of the most common applications today is the browser. And to get a good performance modern browsers take every advantage they can get, including multithreading, GPUs, etc. At the very least every tab will get a separate thread. And many modern applications are also built in HTML with an embedded browser (for example Slack and Discord). Games, at least the bigger ones, also have embraced multithreading a long time ago.
  • Servers: This day and age most servers deal with HTTP requests; other technologies are pretty niche. And webservers scale up nicely to all the cores you have. Sure, every request will most likely run on a single thread, but multiple threads means you can process multiple requests in parallel. It's absolutely standard. The other common part - the database software - also scales well and any serious engine uses multiple threads.
  • Cell phones/tablets: browsers, again. But even without them there are still plenty of background tasks that repeatedly wake up and do a little something. Having multiple cores means that these background tasks affect your foreground app less and it seems more "snappy". Cell phone CPUs are pretty powerful already, but the low power usage requirement means that they're still slower compared to the desktops - and yet we use them perhaps even more extensively. Including for computation-heavy processes like games.

Long story short, if we went back to single-core CPUs, you'd notice it immediately. Modern operating systems have many processes working in parallel and reducing task switching gives serious benefits. Even if some programs individually have little benefits from multiple cores, the system as a whole almost always profits. That said, I suppose there is a limit on how many cores it makes sense to have for various systems. A cell phone with 64 cores will probably not be significantly faster than a cell phone with 32 cores.

8
  • Underestimated answer, thank you! I thought that browsing is a simple task that doesn’t benefit from multiple cores.
    – Martian
    Jul 11, 2020 at 18:17
  • @AlexeiSavitsky - Well, to be honest, I'm not an expert in browser technology so I don't know how much they've actually done. I know for sure that each tab gets their own process these days. But layout calculations are pretty complicated too and drawing is done with hardware acceleration, so there certainly are opportunities for multithreading.
    – Vilx-
    Jul 11, 2020 at 18:20
  • 1
    +1 Multithreaded software is very common. When performance matters, a serious software developer will usually at least consider using multithreading as part of an optimization, and modern programming frameworks make it relatively easy. Also, much of today's software is web based; such software is effectively multithreaded automatically, because web servers are multithreaded. You don't need to multithread your web software (although it can sometimes help for long running operations) if your web server is already using multithreading to more efficiently serve multiple concurrent users. Jul 11, 2020 at 20:21
  • I just checked my browser (Chrome). I have currently 1 tab open, but there are 7 chrome processes open (leftovers from previous tabs and the frame) and they each have anywhere between 6 and 32 threads. I don't know what they are used for, but it's obvious that Chrome does rely on multithreading.
    – Vilx-
    Jul 11, 2020 at 21:06
  • Correct. It's very unlikely to run software on a single thread operating system. The reason this might not get an appropriate amount of votes is because this community is biased towards programming, where users are delegated a single thread from multiples that have been previously split.
    – TZubiri
    Jul 11, 2020 at 21:39
11

I want to emphasize a point you made that multithreading is hard to implement. Some problems are naturally broken into independent parts that are easily parallelizable ("embarrassingly parallel") so we may easily use multithreading and other parallel techniques such as vector instructions, distributed systems, etc. It may be as easy as using #pragma omp parallel for on a for loop. Maybe even the compiler will automatically vectorize your loop.

Many problems are not so easy and require great care in the interaction between different parts so that they operate in the correct order and don't accidentally break each other's functioning. This is often implemented with locks which usually block execution and may introduce deadlock, but more exotic lock-free algorithms exist. Even then, shared resource contention can lead to problems like resource starvation. See Wikipedia article on concurrency for a broad overview that applies very much to multithreading.

Multithreaded code is much harder to get right and much harder to debug. There may be race conditions that only show up 1/100 times the program is executed. It is much easier for a programmer to reason correctly about a program that (appears to) execute in order than multiple threads executing with any number of different memory access orderings. Behind the scenes, a single processor or compiler may reorder instructions in a way that is normally hidden to the programmer, but breaks if multithreading is introduced.

SEI CERT has a list of rules that should be followed when implementing concurrency. All of these should be taken into account by the programmer to have correct code, and then the programmer must also consider performance. If not followed, severe security vulnerabilities may follow.

1
  • Thanks for a more in-depth explanation, another good answer.
    – Martian
    Jul 11, 2020 at 21:29
7

Software falls in two categories: Fast enough, and not fast enough. If it’s fast enough there is no point in making it run faster with multi threading. Whether there are 15 unused cores doesn’t matter if it’s fast enough without using them.

If it’s not fast enough, people will try to use more cores. (But careful. If my single purpose software runs in 8 days on a single core, and it takes me 3 days to make it use all 8 cores and run in one day, then letting the computer run for eight days is a lot cheaper than paying me for three days of work). Some problems are “embarrassingly parallel”. For example solving the same equation with 1000 different values for some parameter. Or compiling 1,000 source files.

Some problems are hard to improve by multi threading. They will come last.

2
  • 2
    The business value argument is not to be neglected indeed. Also, if you can't scale directly from 1 to 24 threads, but - say - to only 3 and your data isn't well aligned, false sharing might just make your code take 12 days to run now. Not a purely hypothetical scenario. Jul 11, 2020 at 20:17
  • This is an important answer. A lot of programs used every day are not speed limited at all, like my word processor or terminal.
    – qwr
    Jul 11, 2020 at 21:44
3

I have experimented with multi-threading, it is not easy to gain an increase in performance, because the cost of setting up a new thread to carry out a task tends to be quite high - so high that it may not be worth the cost in typical situations.

That said, for tasks which involve intensive computations, there may be gains to be had. I found that it was worthwhile to use a second thread to perform LZ77 compression when implementing RFC 1951.

I doubt there is any significant cost to having multiple cores - so there is nothing irrational about modern processors having the capability, even if it is typically under-utilised.

6
  • 1
    Interesting! Indeed, it's not easy to transform increased multithreading into increased performance. Running more threads, even with multithreaded cores, reduces slightly the speed of each individual thread, while at the same time it can increase significantly the global throughput (see experiment here on SO). But the global throughput increases only if the processing task can be parallelized, which is not obvious for many algorithms that rely on sequential flow (example on SE).
    – Christophe
    Jul 11, 2020 at 12:58
  • You may want to try thread pools in the future. The overhead of multi threading can be reduced significantly if done carefully. The main bottleneck is interdependency between threads. The more you can isolate the work of each thread, the more leverage you'll get.
    – Brent
    Jul 11, 2020 at 15:55
  • @Brent The code does use the system thread pool ( System.Threading.ThreadPool.QueueUserWorkItem) however the time to queue a task for execution is still quite significant, typically several micro-seconds. This means a task that takes roughly this time or less ( and modern processors can do a lot of work in this time ) is not a good candidate for parallel execution. Jul 11, 2020 at 17:21
  • 3
    Not just intensive computations. GUI applications are inherently multithreaded (unless they do no serious work, or freeze the UI whenever they're doing it). So are applications which read external data streams (video, sensor data, stock prices…), or which respond to incoming requests (web servers…).
    – gidds
    Jul 11, 2020 at 17:55
  • You have a rather limited idea of what constitutes "intensive computations" :-) Really intensive computations may have threads that run for hours, days, or even longer, so the setup time is inconsequential.
    – jamesqf
    Jul 11, 2020 at 20:45
3

Vilx- is right, it is everywhere. But first let's separate cores and threads. Having more cores is just a technical detail that allows multi-threaded programs to run faster. Programmers do not "utilize cores", they apply multi-threading and they do not deal with cores at all. Cores are hidden from application programmers, they only deal with threads. And you can create a multi-threaded application on a single core processor just fine and it could be just as useful and effective as it would be on a multi-core processor.

There are basically two reasons to use multiple threads:

  • To get the work done faster.
  • To keep the UI (or some other task) responsive.

Your typical data entry application may have no use for multiple threads because there is just one task and data cannot be processed before the user submits it. When he does submit, he will be interested in the results (if any) so there is nothing that can be done in parallel.

If the submit starts a lengthy search operation however, the user may in the meantime want to do other stuff or start another search, and check back for the results later. Or cancel the search. Then you want to use more than one thread.

You can rest assured that multi-threading will be applied if it makes sense to do so. You may not always be aware of it when you use an application, but you probably would notice it if multi-threading was not applied in a scenario that calls for it.

2
  • When im making advanced calculations based on big dataset i wil split the dataset into serveral smaller, based on how many cores the computer have and run a thread for each dataset. In this way i wil use 100% of the CPU. So at least for me and the software i make, its important to know how many cores the system got.
    – Mr Zach
    Jul 11, 2020 at 21:24
  • Number of hardware threads is related to number of cores. On Intel processors with hyper-threading, this is twice the number of cores.
    – qwr
    Jul 11, 2020 at 21:42
2

An anecdotal data-point: 20+ years ago, when MPI (=message passing interface) was first a widely known thing, many people experimented with rewriting various computation-intense mathematical things for "parallel computation". (Yes, I know this is different from multi-threading at the OS level, but in a way its amplified aspects are easier to understand, since the bench-marking is in some ways easier.)

I vividly recall that one project reported that, after several months of work, their parallelized version ran half as fast (rather than far worse!) as the non-parallelized version. :)

Yes, they were able to identify the bottlenecks, etc., for this, in terms of the algorithm involved. Again, in many ways such analysis is simpler than understanding what an OS is doing! :)

1
  • 1
    I have a similar experience with a linear algebra library I used. I was amazed that it was able to utilize 100% of all my CPUs. Impressive! But profiling the application revealed that most of its time was spent in spinlocks and inter-thread communication overhead. Forcing it into single-threaded mode made the program twice as fast…
    – amon
    Jul 12, 2020 at 8:17
1

https://en.m.wikipedia.org/wiki/Multithreading_(computer_architecture)

Disadvantages

Multiple threads can interfere with each other when sharing hardware resources such as caches or translation lookaside buffers (TLBs). As a result, execution times of a single thread are not improved and can be degraded, even when only one thread is executing, due to lower frequencies or additional pipeline stages that are necessary to accommodate thread-switching hardware.

Overall efficiency varies; Intel claims up to 30% improvement with its Hyper-Threading Technology,[1] while a synthetic program just performing a loop of non-optimized dependent floating-point operations actually gains a 100% speed improvement when run in parallel. On the other hand, hand-tuned assembly language programs using MMX or AltiVec extensions and performing data prefetches (as a good video encoder might) do not suffer from cache misses or idle computing resources. Such programs therefore do not benefit from hardware multithreading and can indeed see degraded performance due to contention for shared resources.

From the software standpoint, hardware support for multithreading is more visible to software, requiring more changes to both application programs and operating systems than multiprocessing. Hardware techniques used to support multithreading often parallel the software techniques used for computer multitasking. Thread scheduling is also a major problem in multithreading.

2
  • Your first paragraph's disadvantages apply to multiple software threads sharing one physical CPU core (by context switching or by SMT). This question is about having at least enough threads to benefit from multi-core CPUs that have multiple separate physical cores. I think that whole article is about techniques for making one core look like multiple logical cores, like SMT or switch-on-cache-miss or other CPU-architecture things, not so much about software multi-threading. Modern CPUs are already paying the cost of supporting SMT (e.g. Hyperthreading), the question is how to take advantage Jul 11, 2020 at 19:51
  • Yes, hyperthreading gains most with inefficient code. Either with code where it is just difficult to use all the available hardware resources, or code compiled by a not very good compiler.
    – gnasher729
    Jul 11, 2020 at 23:03

Not the answer you're looking for? Browse other questions tagged or ask your own question.