On the GPU, each thread has access to "shared" or "local" memory, which is analogous to cache on the CPU. So instead of just caching the most recent page, I can tell my program which pieces of memory will be accessed most frequently and manually keep those in cache. My question is: why do CPU designers not allow an analogous operation? I.e. why can't I say to the CPU, "OK, the nodes of this tree aren't on the same page in memory, but I need to access them a lot, so malloc them in the cache for me"?

  • 3
    Because GPU local memory isn't a cache. It's a kind of non-uniform accessible memory. If your CPU had a non-uniform memory access architecture, that would have operations for determining what kind of memory to store what data on, too.
    – Jules
    Commented Aug 27, 2016 at 22:36
  • 1
    @Jules: Smells like an answer! Commented Aug 28, 2016 at 0:18
  • @jules I see now my question wasn't clear, feel free to edit it... What I'm asking is, why are non-uniform access architectures not widespread? And instead caching is the norm. When I said "cache" I meant "faster memory", not "cached memory" Commented Aug 28, 2016 at 0:21

2 Answers 2


This kind of memory management: telling the cpu (in advance) what content is frequently accessed, is really hard to do for a wide array of programming problems, where data structures involve pointers and such.

Yet it is (by comparison) easier to do for certain parallel sliced algorithms, such as found in the graphics domain. In the graphics domain, you're dealing with large chunks of contiguous (numeric) data and vastly fewer pointers.

So, modern CPUs opt to do their cache management automatically, using multi-level caches that ultimately ends with disc-based memory. Each level of the cache is noticing how often some cached portion of memory is used, and uses that information when it decides to evict something from that level of the cache. Each level has a different "page" size (called line size on the upper levels).

So, there's virtually no way for a programmer to inform the CPU of what to keep and what to evict, because of the combination of multi-level and varying cache line/page sizes at each level. Ok, so that's bad enough, but now, throw in that the same program wants to run on multiple different cpus of different performance (where much of that performance difference comes by increasing cache sizes, number of cache levels, etc..), and, then this becomes an intractable problem for the programmer dealing with general purpose algorithms and data structures.

What programmer can do, then, instead of informing the cpu what to keep/evict, is attempt to co-locate related items (e.g. A and B) so that across all the possible variations of cpus and multi-level caches, if A is in the cache, then so is B. (There are other things programmers can attempt to keep programs cache friendly, you can google "cache friendly" data structure or algorithms.)

Another difference is that the GPU memory is separated from the CPU memory, so programming the GPU necessarily involves moving memory back and forth. Whereas the CPU has cache misses and page faults that automatically load memory that is not close to the CPU, the GPU (historically) doesn't have these mechanisms and the programmers of the GPU have to constantly instruct the GPU to copy memory back and forth between GPU memory and CPU memory. This has been and is ever increasingly a problem as we use GPUs for more problem solving, so eventually we'll see more and more hardware breaking down the barrier between CPU memory and GPU memory, resulting in unification at higher levels of the cache hierarchy.

  • Agreed that generalized programming models (as opposed to the vectorized, SIMD restriction on the GPU) require multi-way associative caches as opposed to manually managed scratchpad SRAM. Note though that we’re contemplating an Actor-like generalized programming model which can enable the compiler to tell the caches when to evict and when to do a cache-to-cache transfer. Commented Jul 27, 2018 at 7:14

There are many reasons why CPUs use a cache rather than simply a block of faster memory. But I'd say that the biggest one is this.

Caches were invented primarily as a means of optimization for memory accesses. And that could only work if it can happen transparently, to programs that have already been written and compiled. Therefore, forcing a program to manually do caching would not be helpful.

Caching was designed to be transparent. It allows programs to benefit from different cache sizes and architectures without having to be explicitly written for them. What you want is something less than transparent, something that only really works well if you program for a specific architecture.

Which is why you do sometimes see this sort of thing... in embedded CPUs. Specialized embedded CPUs have been known to provide access to a small amount of fast SRAM, in some cases by sacrificing a percentage of the data cache. This is done precisely because they are embedded; the code written for these circumstances is specialized and not meant to be cross-platform.

Also, to correct some misinformation:

On the GPU, each thread has access to "shared" or "local" memory, which is analogous to cache on the CPU.

No, it's not analogous to a CPU cache. At all. When GPUs perform memory accesses, they usually do so through caches, just like CPUs do.

Workgroup-local memory is not meant to be a cache. It primarily exists for fast intercommunication between invocations within a workgroup. That's why all threads in a workgroup get to see the same local variable declarations. Sure, you can divide it up and use it as additional per-thread memory. But that's not its main purpose.

  • You mention that GPUs use caches as well... I'm pretty sure shared memory isn't cached, actually (global is), I could be wrong... anyway, my objection is that transparency need not suffer. CPUs could provide two types of memory and provide caches for both. Then programs that don't exploit the faster memory would just use the regular pointer space, no problem. Programs that want to use the faster memory could use special malloc/free functions for two separate spaces. Caching would still happen regardless. Commented Aug 28, 2016 at 2:08
  • @RenéG: "I'm pretty sure shared memory isn't cached" I never said that it was. "Then programs that don't exploit the faster memory would just use the regular pointer space, no problem." From a CPU-designer's POV, there's a problem: you have a lot of hardware that is not being used. You could have put that silicon into stuff that existing programs would actually use. Furthermore, what's the point of caching small memory like that? The whole point is that it's fast memory; you only cache memory accesses if they're slow. Commented Aug 28, 2016 at 2:13
  • "hardware not being used" you misunderstand. Regular programs would use a pointer space that spans the two memories. Commented Aug 28, 2016 at 2:14
  • And ya, I meant you provide a cache for the slow memory as well as providing manual fast memory. Just like on the GPU. Commented Aug 28, 2016 at 2:15
  • 1
    @RenéG: I think you're misunderstanding my point. Programs that don't know about the special memory won't get any benefit, even with your "spans the two memories" approach. And since that is 100% of all programs that existed before this memory was introduced, you would have created hardware that does nothing. CPU makers hate making features that don't get used. Consider the slow adoption of SSE-style instructions. Commented Aug 28, 2016 at 2:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.