In software programming, would it be possible to have both CPU and GPU loads at 100%?

Question

This is a general question on a subject I've found interesting as a gamer: CPU/GPU bottlenecks and programming. If I'm not mistaken, I've come to understand that both CPU and GPU calculate stuff, but that one is better in some calculations than the other due to the difference in architecture. For example, cracking hashes or cryptocurrency mining seems way more efficient on GPUs than on CPUs.

So I've wondered: is having a GPU at 100% load while the CPU is at 50% (for example) inevitable?

Or, more precisely: Can some calculations that are normally done by the GPU be done by the CPU if the first one is at 100% load, so that both reach a 100% load?

I've searched a bit about the subject, but have come back quite empty-handed. I think and hope this has its place in this subsection and am open to any documentation or lecture you might give me!

Answer 1

Theoretically yes, but practically it's rarely worth it.

Both CPUs and GPUs are turing-complete, so any algorithm which can be calculated by one can also be calculated by the other. The question is how fast and how convenient.

While the GPU excels at doing the same simple calculations on many data-points of a large dataset, the CPU is better at more complex algorithms with lots of branching. With most problems the performance difference between CPU and GPU implementations is huge. That means using one to take work from the other when it is stalling would not really lead to a notable increase in performance.

However, the price you have to pay for this is that you need to program everything twice, once for the CPU and once for the GPU. That's more than twice as much work because you will also have to implement the switching and synchronization logic. That logic is extremely difficult to test, because its behavior depends on the current load. Expect very obscure and impossible to reproduce bugs from this stunt.

Answer 2

It is not related to game programming. Some scientific code can also use both the GPU and the CPU.

With careful -and painful- programming, e.g. by using OpenCL or CUDA, you could load both your GPU and your CPU near 100%. Very probably you'll need to write different pieces of code for the GPU (so called "kernel" code) and for the CPU, and some boring glue code (notably to send into the GPU the compiled kernel code).

However, the code would be complex, and you probably need to tune it to the particular hardware you are running on, in particular because data transmission between GPU & CPU is costly.

Read more about heterogeneous computing.

See also OpenACC, supported by recent versions of GCC (e.g. GCC 6 in june 2016)

Answer 3

From a supercomputing viewpoint it is better not to think in CPU/GPU load in percentage but rather determine how many operations your problem at hand needs and then compare that to the peak performance of the system.

If you get 100% CPU utilization it does not necessarily mean that you get all the performance out of the system. CPUs can often do multiple different things at the same time, say a division and an addition. If you can start the division early, it can possibly be overlapped with the addition. Your desktop CPU most likely has an out of order unit which will reorder the statements in order to benefit from such overlaps. Or if you have the following program:

if (expr1)
    expr2;
else
    expr3;

A reordering CPU will try to compute the three expressions at the same time and then throw away the result of one of them. This makes it faster overall. If you have some blocker in your program and you cannot reorder, then you are utilizing less lanes in the CPU, but it will probably still show 100%.

Then you have SIMD features in the CPUs which are vector operations. It is like GPGPU-light in the sense that you usually only have four or eight operations at the same time, GPUs do like 32 or 64. Still you have to use that to crank out the FLOPS.

Stuff like false sharing can lead so a heavy synchronization cost which usually shows up as kernel load in Linux. The CPU is completely used but you do not have much useful throughput.

I have done some programming on an IBM Blue Gene/Q machine. It is has many hierarchy levels (schematic of outdated Blue Gene/L) and is therefore hard to program efficiently. You will have to use the full hierarchy down to SIMD and SMT (Intel calls this HyperThreading) in order to get the performance out.

And then the network often limits you. Therefore it turns out that it is faster in (wall clock) time to compute things at multiple CPUs at the same time instead of communicating it over the network. This will put more load on the CPUs and make the program run faster. But the actual program throughput is not as good as it seems from the raw numbers.

If you add GPUs to the mix, it will become even harder to orchestrate this whole thing to yield performance. That will be one of the things I'll start to do in my Lattice QCD Master Thesis in a couple months.

Answer 4

You might be interested in checking out the Servo browser engine being developed at Mozilla Research, and more specifically its Web Render (video).

While shifting a task from CPU to GPU dynamically might be impractical, as mentioned in other answers (notably @Philip's), it can be practical to study the load of CPU/GPU on typical workloads in advance and switch some tasks to the generally less loaded one.

In the case of Web Render, the novelty is that traditionally browsers do most of their rendering work on the CPU (ie, the CPU is used to compute which objects to display, where to cut, etc...). The GPU is normally better at it... except that not all usecases are trivial to implement (partial culling, shadows, ... and text).

An initial version of Web Render proved highly successful in the performance increase, but did no try to address the issue of text rendering (and had a few other limitations). Mozilla Research is now working on a second version which is intended to have fewer limitations, and notably to support text rendering.

The goal, of course, is to off-load as much as possible of the rendering process to the GPU, leaving the CPU free to execute Javascript, update the DOM, and all the other tasks.

So, while not as extreme as your suggestion, it does go into the direction of designing a computation strategy with both CPU and GPU in mind.

Answer 5

One real world example is the open source LuxRender rendering engine, which is capable of fully loading a CPU and GPU at the same time. In addition, it can load multiple GPUs at the same time and can also distribute across multiple computers.

LuxRender uses OpenCL to facilitate this, although builds without OpenCL also exist.

This is practical because the algorithms that LuxRender uses are highly parallelizable. The most common algorithm LuxRender uses is path tracing, where many individual light paths may be computed independently of each other—an ideal situation for GPU computing and one which requires no complex synchronization between compute nodes. However, limitations of GPUs (lower amounts of memory, lack of support for some complex rendering features, and general lack of availability to some artists) ensure that CPU support is still essential.

Answer 6

With a focus on games (since you mentioned it specifically in your post), there are some ways you can balance the load. One example is "skinning", i.e. animating a model. For each frame to be rendered, you have to generate the transformation matrices for each frame of animation and apply it to the vertices of the model to transform it into the pose it needs to be in. You also have to interpolate frames to get smooth movement, unless you want your animation to look like the original Quake (i.e. jerky).

In this situation, you could do it either on the CPU and upload the results to the GPU for rendering, or do the calculation and rendering on the GPU. I believe nowadays it is done on the GPU (known as "hardware skinning"): it makes sense to do so given you have relatively simple calculations that have to be performed thousands of times over, and each vertex can be calculated concurrently since the result of vertex A has no bearing on the result of vertex B.

In theory however, you could dynamically switch between doing it on the CPU or GPU depending on how overloaded the GPU and CPU are.

The main blocker to doing this across all calculations however is that the CPU and GPU have different strengths and weaknesses. Massively parallel jobs are better done on the GPU, while intensive linear tasks with branching are better done on the CPU. Only a few jobs could realistically be done on both without a serious performance hit.

Overall, the major issue with GPU programming (at least with OpenGL and DirectX 11 and under) is that you have little control over how the GPU interprets your shader code. Branching within a shader is risky because if you accidentally create a dependency between calculations, then the GPU may decide to start rendering your pixels one-by-one, turning 60fps to 10fps in an instant despite the actual data to be rendered being identical.

Answer 7

Yes, it's certainly possible.

Any computation that a CPU can do, a GPU can also do, and vice versa.

But it's uncommon because:

• Engineering complexity While it is possible to run the same code on a CPU and GPU (e.g. CUDA), the processors have different abilities and performance characteristics. One is MIMD; the other, SIMD. What is fast on one is slow on the other (e.g. branching), so you make need to write separate code to maximize performance.

• Cost efficiency GPUs are in aggregate a lot more powerful than CPUs. The whole idea of GPUs is to use cheaper, slower, but more numerous processors to perform computations far faster than CPUs could for the same cost. GPUs are more efficient cost-wise by one or two orders of magnitude.

If you get your algorithm to run on GPUs, it just makes more sense to optimize for those, and add as many as you need.