u/hycrice

I’ve tested a number of ROCm-FPX models, including Qwen, DeepSeek, and many others. Some of them benchmark surprisingly well, reaching 30+ tok/s.

However, once I put them into real production workloads, I often find that they take significantly longer to complete the same task. A model may generate tokens quickly, but if it requires more reasoning steps, produces mistakes, or needs multiple attempts to reach the correct result, that raw token speed means very little.

So, to me, obsessing over quantization benchmarks and tok/s is often just a comfort drug—the numbers make you feel good, but what really matters is time-to-solution: how long it takes the model to actually finish the job correctly.

I've spent the last few sessions trying to make MiniMax-M3's sparse attention actually pay on my 2× Strix Halo (128 GB each, USB4 interconnect, llama.cpp Vulkan + RPC) cluster. Posting the negative result so the next person doesn't burn a week on the same path.

TL;DR: run M3 as a dense model. The sparsity buys you nothing in llama.cpp at ≤128k context, and the "obvious" optimization is actively counterproductive. This is a kernel-architecture limitation, not a bug anyone is about to fix with a small patch.

What we measured (M3, dual-node, Vulkan, needle tests passing throughout):

• Prefill: dense beats sparse at every depth we could test — 9k, 32k, 60k, 118k. Not close, either: dense ~142 t/s u/32k vs sparse ~56 t/s on our setup.
• Decode: sparse decays with depth too. The indexer re-reads its full index cache every token (O(N) per token), so there's no flat-TG rescue at long context either.
• The "just skip the masked tiles" idea is worse than doing nothing. llama.cpp's Vulkan FA already has mask-based tile-skip infrastructure (fa_mask_opt), and we verified — with shader read-backs and bitmap dumps — that it correctly detects and skips 30–50% of tiles with MSA's mask. FA time still got worse than with skipping disabled (+29% on the FA kernel). Reason: the FA is bandwidth-bound, and concurrent workgroups share L2 by streaming K/V in near-lockstep. Sparse per-row skipping desynchronizes that walk, so the same K/V gets re-fetched — you save ALU you had to spare and pay DRAM you didn't. Forced uniform skips (same tiles for every workgroup) save exactly proportional time, which is how we isolated the cause.
• The paper's speedups are real but don't transfer. MiniMax's 14×/7.6× numbers are at 1M context on H800s with their co-designed KV-outer CUDA kernel. llama.cpp's Q-outer FA + a mask is structurally the wrong shape for this — you'd need a ground-up KV-outer kernel (reverse sparse index, gather-based, two-phase LSE combine) to see wins, and our estimate is it only overtakes dense around ~128k even done perfectly.

What actually helped on this hardware, in case you have M3 running:

• q8_0 KV cache (the one genuinely useful thing to come out of our MSA work — f16 caps ~256k context, q8 reaches 380k+)
• Bigger ubatch (-ub 1536): +8% prefill at ~9k depth, parity at 32k
• EAGLE3 speculative decoding: +25% decode on code workloads, ~nothing on chat (don't believe the 2.8× vLLM numbers transfer — vLLM's eager-mode baseline is much slower than llama.cpp's, so spec decode has more overhead to amortize there)