From a supercomputing viewpoint it is better not to think in CPU/GPU load in percentage but rather determine how many operations your problem at hand needs and then compare that to the peak performance of the system.
If you get 100% CPU utilization it does not necessarily mean that you get all the performance out of the system. CPUs can often do multiple different things at the same time, say a division and an addition. If you can start the division early, it can possibly be overlapped with the addition. Your desktop CPU most likely has an out of order unit which will reorder the statements in order to benefit from such overlaps. Or if you have the following program:
A reordering CPU will try to compute the three expressions at the same time and then throw away the result of one of them. This makes it faster overall. If you have some blocker in your program and you cannot reorder, then you are utilizing less lanes in the CPU, but it will probably still show 100%.
Then you have SIMD features in the CPUs which are vector operations. It is like GPGPU-light in the sense that you usually only have four or eight operations at the same time, GPUs do like 32 or 64. Still you have to use that to crank out the FLOPS.
Stuff like false sharing can lead so a heavy synchronization cost which usually shows up as kernel load in Linux. The CPU is completely used but you do not have much useful throughput.
I have done some programming on an IBM Blue Gene/Q machine. It is has many hierarchy levels (schematic of outdated Blue Gene/L) and is therefore hard to program efficiently. You will have to use the full hierarchy down to SIMD and SMT (Intel calls this HyperThreading) in order to get the performance out.
And then the network often limits you. Therefore it turns out that it is faster in (wall clock) time to compute things at multiple CPUs at the same time instead of communicating it over the network. This will put more load on the CPUs and make the program run faster. But the actual program throughput is not as good as it seems from the raw numbers.
If you add GPUs to the mix, it will become even harder to orchestrate this whole thing to yield performance. That will be one of the things I'll start to do in my Lattice QCD Master Thesis in a couple months.