How can single thread execution further increase since frequency stagnates?

Question

What are things that newer CPU can do to speed up single thread execution?

• multiple registers? (can compilers always benefit from it?)

• SIMD? (do compilers use SIMD without code annotation?)

• does a larger cache always speed up execution speed?

• Is branch prediction faster with more transistors?

What other things have been increasing execution speed?

A lot of "common algorithms" cannot be parallelized or use concurrency, but with newer CPUs, it still "feels" like single execution is faster than the increase in frequency.

How can CPU designers go around frequency limitations to increase single thread speed?

I don't really follow CPU designs, it looks quite complicated.

Answer 1

An addition to the question was: Can CPU speed be increased for one thread only?

Clock speed is mostly limited by heat and power consumption. If you run two cores at half speed, you use much less power than with one core at full speed, so you want to use more cores.

But if you use a single thread only, that produces one eighth of the heat of eight cores at the same speed. If your code can’t use multiple cores, you can run that core at a higher speed. You can even move the thread to a different core say every second, so the heat goes to different places in the chip, allowing even more power usage = clock speed without the processor getting too hot.

So it’s easier to get more work done with a single thread. It is still much better (less heat, less power) to use multiple threads.

Do bigger caches produce more speed? Depends. First, with the same technology access to a larger cache can take longer. So it is a requirement to have better technology.

Some software is designed to use caches in an optimal way. If the processor has a bigger cache but the software makes incorrect assumptions, it might not use the bigger cache and it doesn’t help. You get the biggest benefit if the old processor didn’t have enough cache for a task, and now it is enough. For instruction caches, you can likely not influence the amount of cache needed. So either the smaller cache was big enough or you win.

Answer 2

1. Single thread performance is still increasing. For example about 50% between M1 and M4.

2. Anything that is slowed down noticeably by using a single thread only, get used to using multiple threads. In most environments this is a very, very simple task.

3. Easy improvements are larger caches, more cache levels, better branch prediction, more execution units, bigger buffers for OOO execution. Sometimes execution times for complex instructions (division and square root) can be improved.

4. Improving the instruction set and ISA is possible, but requires at least recompiling. Like more registers, more and/or larger vector registers, adding instructions that are often used by a compiler, efficient atomic instructions etc.

Currently it is much easier to add more cores than increasing single threaded performance. For example there are ARM server chips with 64 or 128 not very fast cores, perfect to handle requests by dozens and dozens of users simultaneously at lowest power usage. The same work can be done much faster and with less energy using multiple cores. This will happen until a point is reached where more threads cannot be used in a useful way in many cases.

Answer 3

multiple registers? (can compilers always benefit from it?)

Adding registers is an architectural change that require recompilation, so is quite rare. But it was done in the x86 -> x64 migration, and the APX proposal might increase the register count in the future. It also includes various other improvements that should give some benefit.

SIMD? (do compilers use SIMD without code annotation?)

They could, but automatic vectorization is rather difficult to do well without manual intervention, and many compilers do not bother.

does a larger cache always speed up execution speed?

No. Latency is also a very important, and larger caches often result in higher latency. The working set and access patterns of the application is also important. If everything fits in the cache already there will be no benefit.

Is branch prediction faster with more transistors?

A higher transistor budget can help improve branch prediction and therefore performance. But some branches are almost impossible to predict, and some can be predicted well with even a simple predictor. So results will vary.

How can CPU designers go around frequency limitations to increase single thread speed?

The vast majority of improvements are a result of improved parallelization, from implicit instruction level parallelism to the massive explicit parallelism of GPUs. Most workloads have plenty of possible parallelism to be exploited, both implicitly and explicitly.

This will likely continue, with CPUs keeping more and more instructions in flight, as well as improvements on the software side to better use the available resources. It will also favor software that can scale well. Much of the advancement in AI has been enabled by the availability fairly cheap GPUs as an example.