2
  • Why do the compile times not vary significantly between different era CPUs, even though disk (NVMe vs. HDD) and CPU benchmarks vary significantly in performance?
  • Why does disabling hyperthreading affect performance significantly with the Ryzen CPU?

Over the past few months I have seen some different machines all running Linux with CPUs that varied from about 9 years since release and some recently released. The newer CPUs received much higher benchmark numbers from cpubenchmark.net. Details below including compile times for the Linux 4.4.176 kernel using Ubuntu 18.04.

To put it simply, CPUs that scored multiple times faster at cpubenchmark.net most certainly did not decrease compile times by the same factor. In fact, sometimes meager improvements were seen.

What would be the bottleneck to change or fix? The Ryzen machine has all the latest hardware gadgets. Or is this a case of synthetic benchmarks vs. reality?

This question touches on the topic of benchmarking by compile time comparison, but does not explore the (lack of) variability in build times seen.

  • Recent Ryzen 2950X system
    • Single Thread Rating: 2208
    • Disk: NVMe
    • Source: Linux kernel, 4.4.176, compiled with .config from Ubuntu Xenial
    • Invocation: make -j32
    • Compile time: 25997u 4910s 17:25 wall time
  • Same system, with hyperthreading (or whatever AMD calls it) turned off
    • Source: Linux kernel 4.4.176, same .config file
    • Invocation: make -j16
    • Compile time: 10561u 1796s 13:44 wall time

That is an over 20% reduction in compile time amounting to a difference of 3 minutes 41 seconds, from 17:25 to 13:44. Just by disabling the hyperthreading feature.

New Data, 2019-03-08

Compile time actually decreased monotonically, but not linearly, until make -j12, taking 12:46 with a dozen processes. The compile time then increased slowly and monotonically out to the last run of make -j24. Hyperthreading was off for these tests.

Whatever the bottleneck is, is hit after 12 parallel threads.

New Data - 2019-03-27

After tuning the X.M.P. memory profile, but still with single channel access, the times decreased until a minimum was reached of 10:08.5 at 16 threads; as many threads as cores (no HT enabled). With one exception the compile times increased slowly to 10:41 at 32 threads.

Once dual channel memory was enabled, times decreased to 6:47.5 at 17 threads. After this point, the timings wobbled up and down with another minimum of 6:46.6 at 20 threads. This is probably a fluke and likely not a definitive point at which a minimum point is reached. Only a maximum of 24 threads was tested in this case.

It seems that memory is a huge factor and the bottleneck for this processor. Tossing more processing threads at the problem once memory is configured optimally does not seem to help or hinder much.

Conclusion

  • Memory is a major potential bottleneck for this CPU and must be configured properly to take advantage of the processor.

  • The old rule of as many threads as processors seems to hold.

  • Hyperthreading was not tested beyond the first tests, so no conclusions can be drawn in that area, other than that the default configuration with single channel memory probably starves the hyperthreaded cores for memory accesses.

Improve this question
8
  • You're telling your make program to run the same number of jobs in parallel as your processor has effective cores. What happens if the compiler can exploit concurrency? Specifically, what happens to the cache? Or are those the best speeds you got with different levels of parallelism on the make command line?
    – Ed Grimm
    Feb 27, 2019 at 4:54
  • Here is a primer on what NVMe is, and how much faster than the HDD in the older system is, for those wondering how slow the disk is: enterprisestorageforum.com/storage-hardware/…
    – casualcoder
    Feb 28, 2019 at 0:42
  • @grimm, I can try different levels of parallelism. It might take a couple of days, will report back then. Running make -j1 might not be worth the time to do it, though. :)
    – casualcoder
    Feb 28, 2019 at 0:45
  • @grimm, ran the compile jobs again and posted a summary of the results.
    – casualcoder
    Mar 9, 2019 at 2:47
  • It would be my guess if you turn hyperthreading back on, and tested from -j10 to -j24, you would probably find with hyperthreading you can get a bit better performance than without it, but the optimal performance will be somewhere below -j24. Though it probably only makes sense to go a few iterations after it starts getting slower, rather than all the way to 24. If the bottleneck you hit at 12 is CPU (basically, the compiler itself is able to make enough use of parallelism to use roughly a third of a processor), the knee will be higher than 12, otherwise still 12.
    – Ed Grimm
    Mar 9, 2019 at 3:06

3 Answers 3

Reset to default
7

Just off the top of my head:

  1. CPU performance is not the only thing that affects compile times.

  2. Benchmarks do not always take into account performance factors that affect real programs.

  3. Cache affinity can be a larger factor than CPU speed.

  4. Compilers don't always exploit concurrency mechanisms effectively.

  5. In general, programs like compilers can be written in such a way that they exploit (or fail to exploit) specific CPU characteristics.

  6. Concurrency mechanisms have overhead. If the software is not getting any benefit from the concurrency offered, the result is a net negative.

In short, it's complicated. A highly complex task like compilation is not always well-modeled by benchmarks. You have to consider other real-world factors that affect overall performance, like seek times on hard disks and the manner in which the code is written.

For a simple, but rather dramatic example of this, see here.

Improve this answer
1
  • Thank you for your answer. It was possible to double the speed of the machine by configuring the memory properly. So, the bottleneck seemed to be the memory subsystem. This system is about 3.6x faster than an old Intel i7, 4 core machine which took 24:22 to do the same compile job. This is roughly in line with the benchmark numbers on cpubenchmark.net .
    – casualcoder
    Mar 28, 2019 at 2:07
2

This turns out to be more of a hardware problem than a software engineering issue, perhaps.

There is a memory setting in modern BIOS' called X.M.P. This setting is by default at the lowest common denominator for what memory the mainboard supports. To take full advantage of the memory installed, one has to go into the BIOS and set the X.M.P. profile. There seems to be only one profile available in this case besides the default setting.

The other issue is setting dual channel memory access vs. single channel memory access. That will take a moment with the manual to determine how to install the memory to enable this setting.

A test build with some different software took more than 20% off the build time with the better X.M.P. setting vs. the previous default setting. So there is some hope for better performance to come.

Once tuned, I will report back with the final numbers for comparison.

2019-03-27 Conclusions

Memory was the major bottleneck for this CPU. See question for more details and conclusions.

This question has not garnered much interest, so this will likely be the last entry unless someone specifically asks a question.

Improve this answer
1

Compiling a large project may involve reading hundreds of source files, and writing hundreds of intermediate (object) files. Then reading all those intermediate files again to compile the finished executable or library.

Often, the biggest bottleneck isn't the CPU but the disk subsystem (the disk drive itself and the disk controller).

Multicore or hyperthreaded CPUs can confuse the system further by sending multiple simultaneous read or write requests to the disk subsystem, each asking for different files. If the disk has a high seek time, then this could even slow down performance when compared with a single core CPU.

The impact of disk performance can be particularly acute when the files are on a remote server, and mapped as a drive on the local machine.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.