GPT-OSS-120B (MXFP4) resident on a Strix Halo iGPU, +2 more models in 96GB UMA — real tok/s, Vulkan not ROCm
Running a 3-model stack fully resident on an AMD Ryzen AI MAX+ 395 (Strix Halo) integrated GPU — no discrete card, no cloud. Sharing the setup and honest numbers because most "local LLM on an APU" posts skip the part where the vendor stack doesn't work.
The box: GMKtec EVO-X2, Ryzen AI MAX+ 395, Radeon 8060S iGPU (gfx1151), 128 GB unified RAM, Ubuntu 24.04 / 6.17 HWE kernel. BIOS UMA split set to 96 GiB VRAM / 32 GiB system.
Why Vulkan, not ROCm: ROCm/HIP is broken on gfx1151 — sentence-transformers hangs on HIP init, the compute stack won't come up. So the whole thing runs on llama.cpp's Vulkan backend (RADV). If you're on Strix Halo, save yourself the week: go Vulkan.
What's resident (~78 GiB of the 96):
- lm1 — GPT-OSS-120B, MXFP4 (~60 GiB) — the reasoner; ~5B active params/token (MoE)
- lm2 — Llama3.2-24B-A3B (i1-Q4_K_M) (~11 GiB) — creative/uncensored slot
- lm3 — Qwen2.5-Coder-7B (Q4_K_M) (~5 GiB) — FIM/autocomplete
- plus MiniLM + nomic embeddings on their own llama-server
Numbers (GPT-OSS-120B, MXFP4, Vulkan, direct completion): ~81 t/s prompt, ~38 t/s generation. First request after load is slow (cold buffers) then it settles. For reference, the previous lm1 (Qwen3.6-35B-A3B) did ~823 PP / 46 TG — the 120B trades throughput for depth. 38 tok/s on a 120B on an integrated GPU is the headline for me.
llama-server flags that matter here: -ngl -1, --flash-attn on, --cache-type-k q8_0 --cache-type-v q8_0 (KV quant is what makes 64k ctx fit alongside everything), --batch-size 2048 --ubatch-size 512, --no-mmap. Note: ubatch is not a lever for A3B MoE — llama.cpp's default 512 is already the peak, bigger just wastes VRAM.
Gotchas nobody tells you:
- gpt-oss reasons on every call via the harmony analysis channel, and
--reasoning-budget 0/<think>stripping does not suppress it — that's a Qwen-ism that doesn't apply. A trivial query still eats ~18s of reasoning. What does work: passchat_template_kwargs: {"reasoning_effort": "low"}per request — measured ~5× faster with equivalent output on mechanical tasks. Budget your client timeouts for reasoning+generation, not generation alone (learned that the hard way when a batch worker's 240s timeout started failing on big prompts). - On a 96/32 UMA split, the system side becomes the tight budget, not VRAM. I'm at ~5.6 GiB free of 32 with all three models up. Watch RAM, not just VRAM.
--parallel 1means callers queue serially — fine for batch, plan for it.
What it actually drives: a private OpenAI-compatible gateway that fronts all three as watched llama-server children (VRAM-largest-first, watchdog restarts). It powers a news-corroboration engine, a VSCode agent extension, and an always-on story generator — all pointing at :5000, nothing leaving the box.
Happy to answer anything — flags, the Vulkan path, MoE-on-bandwidth-bound-APU tradeoffs, the multi-model VRAM budgeting.