I’ve been tuning Qwen3.8-Flash-Next 125B MoE on a workstation with only 32 GB VRAM + 64 GB RAM.

The result I found most interesting: in a real agent workload, decode stayed around 28 tok/s while context grew from 116K to 148K.

This is a llama.cpp fork based on upstream "9e7171624" with Qwen4Exp support plus a few local changes.

Hardware

• Intel Core Ultra 5 225, 10 cores
• 62 GiB DDR5-6200
• RTX PRO 4500 Blackwell 32 GB, SM120
• PCIe 4.0 NVMe
• Ubuntu 24.04
• CUDA 13.4.59
• built for "sm_120a"

Model

Qwen3.8-Flash-Next, "UD-Q3_K_XL"

• ~90 GB GGUF
• 125B total / ~6B active per token
• 48 layers
• 512 experts, 10 routed + 1 shared
• QSA indexer, 2048-token budget
• additional 51B N-gram / PLE table: 26.8 GiB
• MTP sidecar available but disabled

Placement

-dev CUDA0 -ngl all -ncmoe 32

Dense/GDN/QSA/router/shared stay on GPU.

Of the 48 routed-expert layers:

• 16 on GPU
• 32 on CPU

PLE is disk-backed:

-ot per_layer_token_embd=CPU --load-mode mmap --lazy-mode on

Other relevant settings:

-c 240000 -fa on -ctk f16 -ctv f16 -t 6 -tb 6 --fit on --fit-target 2048

Steady state is roughly:

• 31.3 GiB VRAM
• 43 GiB RSS
• effectively no process swap on weights

Real workload numbers

19 minutes of agent traffic:

• 35 requests
• 23K generated tokens
• context: 116K → 148K

Token-weighted decode:

27.9 tok/s

By context:

• 115–125K: 27.7
• 125–135K: 28.5
• 135–148K: 27.5

So decode is essentially flat across another 32K of context.

Real prompt chunks prefill around 150 tok/s, peak 201.

Prefix cache hit rate is ~99%, with only ~370 new prompt tokens/request at 130K+ context.

CUDA graphs were reused 22,590 times for 23,053 generated tokens.

CPU usage is around 5.8 cores, so the CPU expert layers are doing real work.

PLE disk traffic was only 5 GB over 52 minutes, so once warm, the SSD is not the bottleneck.

What actually mattered

1. Disk-backed PLE

The 26.8 GiB PLE table stays mmap-backed and only rows that are needed get faulted in.

Without this, the model does not fit comfortably in 64 GB RAM.

In practice the I/O cost has been surprisingly small.

2. Static hybrid expert placement

This was the biggest win.

At 8K context:

• CPU-only: 42.6 pp / 6.3 tg
• hybrid: 250 pp / 32.4 tg

Roughly 5.9× prefill and 5.1× decode.

3. Forced cuBLAS

GGML_CUDA_FORCE_CUBLAS=ON

The SM120 MMQ path crashes on this model with "illegal CUDA memory access".

cuBLAS is stable.

4. Sparse QSA decode

This uses block-level Top-K plus a persistent block-key cache, with "n_kv_max = top_k".

This appears to be the main reason decode does not collapse at 140K+ context.

The dense reference path can still be restored with:

LLAMA_QWEN4_SPARSE_QSA=0

5. K-only indexer cache

Upstream PR #28330 saves about 0.75 GiB VRAM at 128K.

That headroom is what lets me keep F16 KV and CUDA graphs enabled.

6. F16 KV beats Q8_0 here

At 49K prompt / 65K context:

• F16: 496 pp / 38.1 tg
• Q8_0: 499 pp / 33.4 tg

Q8_0 saves ~808 MiB but costs about 12% decode, so I kept F16.

7. Faster CPU expert kernels

The 32 CPU expert layers only work because the CPU path is reasonably optimized:

• signed-VNNI Q8 "mul_mat": 1.36–1.39×
• IQ4_NL/Q8_0 two-token expert kernel + compact routing: 1.43×
• routing table: 20 MiB → 44 KiB
• fused AVX2 hyperconnection ops: 3–7× per op

Things that did not work

"--lazy-mode on-direct" 1.3 tok/s decode vs 31.6 on the same build. Hard no.

Persistent GPU hot-expert cache 15.6 GiB cache gave 21.6 tok/s decode vs 31.6 with static placement.

SM120 MMQ CUDA faults. Using cuBLAS.

MTP Disabled. The draft runs on CPU and the extra PCIe/RAM traffic outweighs the speculative-token gain on this machine.

Caveats

The sparse-QSA CUDA patch is still experimental and was explicitly marked unreviewed by its author.

I have not yet done full long-context correctness testing against the dense path.

These are also real workload numbers with warm caches, not a controlled benchmark suite, and there is no quality/PPL benchmark yet.

The individual speedups above come from different configurations, so they are not additive.

Question

At this point decode is basically flat through 148K context.

The current wall is VRAM: I only have about 604 MiB free at "-c 240000".

If you had another ~1 GiB to spend on this setup, would you use it to:

• move more expert layers from CPU → GPU, or
• keep it as KV / CUDA-graph / longer-context headroom?

Especially interested in results from similar hybrid MoE setups where the CPU expert path is already reasonably fast.

Humen (Me ;) TL;DR:

• Optimized CPU-only llama.cpp inference for Qwen3.8-Flash-Next UD-Q3_K_XL;
• System: Intel Core Ultra 225 with 64GB DDR5;
• RAM pinning (N-GRAM SSD offload hard coded) + IQ4 routing boosted Prefill from 23.6 to 43.9 t/s (+86%), while AVX2 hyperconnection fusion + MTP (Draft 3) boosted Generation from 5.4 to 7.8 t/s (+44%), reaching 11.3 t/s on pi agent workloads.

LLM long story:

Hey everyone,

I’ve been experimenting with CPU-only inference for Qwen3.8-Flash-Next (UD-Q3_K_XL) with N-GRAM SSD offloading. With some iterative help from OpenAI’s GPT Astra to write and refactor low-level kernels and memory management, I managed to significantly boost both Prefill and Generation performance on a pure CPU setup.

Hardware Specs

• CPU: Intel Core Ultra 225
• RAM: 64 GB DDR5-6200 (4x16 GB); real speed random read ~70 GB/s
• Storage: Fast NVMe SSD (used for N-GRAM offloading)

Key Optimizations & Benchmarks

Here is the breakdown of the changes made, why they helped, and their direct impact on inference speed:

Optimization	Why it helps	Prefill Speed	Generation Speed
Anonymous RAM Locking & SSD N-GRAM	Keeps base model weights pinned in RAM, preventing OS paging/re-reading, while streaming N-GRAM on demand from SSD.	23.61 → 38.27 t/s	5.42 → 5.86 t/s
Q8-Matmul with Direct Signed-VNNI	Reduces instruction count and avoids unnecessary register spills.	Isolated kernel 1.24–1.39× faster (no overall prefill gain in that run)	5.86 → 6.09 t/s
IQ4 Expert Kernel (2 Tokens & Compact Routing)	Reuses decoded weights, reduces routing overhead, and improves thread distribution across cores.	35.82 → 43.89 t/s (+22.5%)	No measurable change
Hyperconnection-Fusion (AVX2)	Fuses Gate, Stream-Averaging, and Residual ops to reduce intermediate tensor allocation and synchronization barriers.	Isolated kernel @ 512 tokens 3.18–3.65× faster	Single token 4.19–4.52× faster (overall run boost pending final isolation)
MTP & Draft Length Reduction (5 → 3)	Evaluates multiple tokens jointly. Shorter draft lengths reduce wasted compute cycles on rejected drafts.	No clear prefill gain from MTP	Draft 5 → 3: 6.20 → 7.78 t/s (+25.6%) Pi-Agent @ Draft 3: 11.30 t/s

Key Takeaways

1. Prefill Bottlenecks: The biggest gains for Prefill speed came from RAM pinning (eliminating OS page faults) and IQ4 routing optimizations.
2. Generation Bottlenecks: MTP tuned with a Draft Length of 3 yielded the highest throughput jump during token generation, reaching up to 11.30 t/s in specific agent workloads (Pi-Agent).