A ground breaking research idea - MacBook user’s…
What if we’re optimizing the wrong bottleneck?
I recently went down an 11 PM research rabbit hole around Mixture-of-Experts (MoE) and LLM inference.
A lot of MoE optimization assumes a traditional GPU setup: experts live in CPU RAM, get moved to GPU VRAM, and PCIe becomes the bottleneck. But what happens when that assumption disappears?
Apple Silicon uses Unified Memory, where CPU and GPU share the same memory pool. So maybe the interesting question isn’t: “How do we move experts faster?” but : “How should we optimize MoE when there’s no traditional CPU↔GPU memory transfer bottleneck?”
That leads to some surprisingly interesting questions around memory bandwidth, caching, expert dispatch, and GPU kernels. I can’t investigate it properly myself right now, I don’t have the hardware. So I’m putting the idea out there. If you have an M-series Max/Ultra Mac, you might have a research project sitting on your desk. 👀
Read the article…