Technology

Wrote a longer reply to someone else, but briefly, yes, you are correct. Kinda.

Caches won’t help with bandwidth-bound compute (read: ”AI”) it the streamed dataset is significantly larger than the cache. A cache will only speed up repeated access to a limited set of data.

Could it work?

Yes, but it would require:

• A redesigned memory controller capable of tiering RAM (which would be more complex).
• OS-level support for dynamically assigning memory usage based on speed (Operating systems and applications assume all RAM operates at the same speed).
• Applications/libraries optimized to take advantage of this tiering.

Right now, the easiest solution for fast, high-bandwidth RAM is just to solder all of it.

Using it as cache would reduce total capacity as cache implies coherence, and treating it as ordinary swap would mean copying to main memory before you access it which is silly when you can access it directly. That is you'd want to write a couple of lines of kernel code to use it effectively but it's nowhere close to rocket science. Nowhere near as complicated as making proper use of NUMA architectures.