u/stereohype

tldr: the gap in the title is what the NPU tools buy on a real bug fix. And local now covers about 95% of my cloud calls, the hardest few percent still goes to GLM 5.3 or Opus. Receipts in the repo.

Qwen3.8 Flash-Next, a 125B MoE, runs my pi coding agent on a 70W tablet, and the NPU finally earns its power draw. Wired halogen's endpoints expecting a party trick, kept what paid.

v1.1 since launch: opt-in voice on the CPU, search moved to semble. The tools that stayed on the NPU:

• Decisions. Yes/no branching stops eating turns of the big model. A 0.8b answers in 120ms, 78% accurate.
• Dup scan. Catches copied and renamed files git never shows you. 45 pairs across 4 repos in 8.4s.
• Screening. Injection attempts get flagged before the agent acts. 0.7s, zero false alarms, fails open. 42% recall: a smoke detector, not a safe.
• Search. The agent lands on the right file first try, on any install. The NPU pipeline beat ripgrep 15/20 to 9/20; semble replaced it at 17/18 to 13/18 on the ground-truth A/B.
• Compaction. Big sessions summarize on a tiny sidecar, off the critical path. My 194k session: 50s, 97% cache hit.

The eight-run check was about taste and depth, not throughput: I pasted a real timeshift error to seven agents, the 125B at medium and max against 320B and 753B tiers at max. Eight runs, all healthy.

• Flash-Next, 2m55s: journalctl trace to a racing notify-send, pacman.log check, upstream PR found.
• glm-5.3-flashx, 2m41s: the most precise answer, spotting the snapshot mount unmounted under the script's last line. No PR.
• GLM 5.3 753B at max, 8m30s: the deepest forensics, source-level down to function names and the one-second race window. No PR.
• Flash-Next at max, 8m31s: level with the 753B, and still the only model with the PR.

I expected cloud to be faster. It never pulled away: 14 seconds, then one, GLM at max both times. The PR was not a tie.

All eight side by side. In opencode: flash called it in 1m8s with the wrong mechanism, the full 753B correct in 6m15s, PR missed. And when I had GLM 5.3 flashx rate both results, it picked the qwen answer too.

Reality: the GPU still does the thinking, ~7% iGPU cost only when they overlap. Decode: 64 tok/s best case, 45-50 sustained at 70W. Cap the box at 50W, give up a couple of tok/s, stay under 75C and quiet. First token in ~0.03s, 43x faster than the cloud call beside it, 7x with a second agent.

The official halogen launch is a 24-flag docker command. Mine is one, and uninstall undoes it. Fully local, 262k context. No Strix Halo? setup.sh --no-halogen runs pi against your own providers.

Anyone else putting their NPU to real use? I found nothing.

125B MoE (A6B) running faster than a 27B dense on the same iGPU, and producing better agent builds. Full receipts below, everything reproducible.

The numbers (all through an agent game-build workload, not llama-bench)

For scale: the same workload on Qwen3.8-27B runs 17.5-18.5 t/s decode. This is a 125B MoE doubling a 27B's decode on the same silicon.

The stack (each line is load-bearing)

1. Engine: Nathan's strix-halo-vulkan releases (validated v0.6.11 through v0.7.3) — built from source, not the portable payload. The payload's portable build leaves 6-24% on the table (measured: JSON-class decode 28.3 → 40.3 t/s on identical flags).
2. Driver: your system Mesa stable. It beat the bundled devel snapshot on agent-relevant classes (deep decode +9.5%, emission +13%).
3. Model: Unsloth UD-IQ4_XS (87.25GB, 3 shards — pin your revision, they re-upload under the same names).
4. Drafter: the native MTP head as a Q8_0 sidecar (~4GB), fixed n4. Don't use adaptive draft sizing here — it loses on MTP's chained drafts (measured both directions; it wins on DFlash2 block drafts, different mechanism).
5. KV: q8_0 both, context 131072. Needs ~91GB weights+sidecar+KV — fits 128GB Strix Halo. Check your own mem_info_vram_total + mem_info_gtt_total: both heaps count, the Vulkan allocator uses both.
6. The flags that matter more than any of the above: --reasoning-effort medium --reasoning-budget 2048. Without them, this model burns its entire output budget on reasoning and emits nothing — 8.5k chars of thinking, zero content, finish=length. With them: clean, terse, fast. This is not optional.

The part benchmarks can't see

I validate agent stacks by having a coding agent build a small canvas game from a fixed contract, then scoring the code, soaking the running game headlessly, and playtesting it. Two things this workload caught that raw throughput never would:

• The template is per-model infrastructure. Effort kwargs that work on one model's template are silently ignored here. The reasoning-budget flags above are the fix — they're server-native, not template-dependent.
• Every model has blind spots, and they're usually spec gaps. Flash-Next failed the same game subsystem (a floating obstacle pad) three builds running — hit registration, stickiness, bounce direction. One explicit contract sentence with a trace-before-you-finish check cured it: next build fully clean, in 12 minutes. The same cure fixed a different model's serve-rule bug earlier. 2-for-2.

And the closing control experiment: Flash-Next vs a 27B at low effort, same environment, same contract — identical scores, identical failure sets, identical line counts (1,412 each).

Flash-Next: 5 minutes, 40 t/s, no edits. The 27B: 39 minutes, 4.3× the reasoning, five of my edits — and it spontaneously wrote and ran its own smoke-test script mid-build. On explicit contracts, model size buys speed, not quality. Pick by token budget.

Reproduce

The workload harness is public: neon-ladder — contract, scorer, runtime gate, runner, one-comment recipe. One cell is ~25 minutes. Post your one-liner (quant / drafter / effort / wall / static / soak / playtest) and the variance rules are documented so your numbers and mine can argue productively.

Full methodology, the 27B side of the story, and twelve playtest-found failure classes that no static check ever caught: my earlier posts in this sub.

The short version of everything I learned: the contract is the tool, the playtest is the grade, and a model that benchmarks beautifully can still be unable to build a working game — which is why you measure the thing you actually do.

Dense 27B, meet DFlash2. On my Flow Z13 (Ryzen AI Max+ 395, Radeon 8060S, 128GB), Qwen3.8-27B hits 31 t/s burst at 80W and averages 26+ t/s across a full 90k-token generation, with every token paying for all 27B parameters. No MoE shortcuts this time.

Three things got it there: Inco AI's DFlash2 drafter, n-max 4, and a chat template that stops the model from drowning in its own reasoning. The last one surprised me most. The stock template's runaway thinking didn't just waste tokens, it collapsed draft acceptance at depth; fix the style and the speed decay disappears.

Last week's DeepSeek V4 Flash guide (https://www.reddit.com/r/LocalLLaMA/comments/1vlmh0b) covered the MoE side of this hardware. This post is the dense half.

Note: the writing is AI-assisted editing; the research, debugging, and every number are from my own runs on this machine.

Edit (Aug 20): the recommended config was revised after the burst table below was measured. Three findings landed on top of the initial numbers: n-max 4 beats 7 by 25% on long generations, the Sharp chat template fixes a runaway-reasoning failure mode and lifts sustained decode another 26%, and q8_0 drafter KV is a free +1.6% with bit-identical output. The launch command reflects all three; sustained expectations are now 26-29 t/s from 1k to 32k output tokens.

Edit (Aug 21): two more pieces landed. The 128k config is real now: 96k and 128k ctx with -ub 4096 are rock solid on this fork (a reported ub>1024 crash past 64k does not reproduce here), and a full 90k-token generation averaged 26.4 t/s with acceptance still climbing at the finish (0.77, mean accepted draft length 4.1). I also validated the whole stack inside a coding harness: pi + this server ran an 18-step tool-using task (read/write/edit/bash) plus a continuation turn with zero format errors and ~64% draft acceptance through tool traffic. The 90k numbers below replace the earlier 64k entry, which had stopped early at ~51k tokens.

The engine reports n_ctx_train 262144 and caps anything higher; the full spec config loads and runs healthy at -c 262144 (~55GB RAM), and since allocation costs nothing until filled there is no reason to set less if you have the memory.

Caveat: these are early benchmarks. The sustained matrix (both quants to 8k, Q5 to 90k), the Unsloth v2/v3 A/B, full-corpus PPL, and harness tool-use validation have landed; more workloads are still in progress.

TL;DR

• Daily driver: UD-Q5_K_XL + incoai DFlash2 Q4_K_M drafter (n-max 4) + Nathan's v0.6.5, f16 KV, drafter KV q8_0, ubatch 4096, Sharp chat template. ~30 t/s burst, 25-29 t/s sustained out to 90k tokens, 256k context available (the model's training cap).
• DFlash2 over the built-in MTP is +40% burst (30.2 vs 21.6 t/s at 70W) and +60% sustained (25.9 vs 16.3 t/s at 8k tokens). Lossless: output is bit-identical to unspec decode at temp 0, verified.
• Q5_K_XL decodes faster than Q4_K_XL with DFlash2 on short generations (31.4 vs 28.5 t/s at 80W burst; Q5's 55.9% vs 44.9% acceptance pays for the extra bandwidth). On long generations the ranking flips: at 8k tokens Q4 + Sharp beats Q5 + Sharp (28.8 vs 27.5 t/s). I run Q5 for the quality at ~1-2 t/s sustained cost; all-long-form-code workflows should take Q4 + Sharp.
• Pin your Unsloth revision: every number here is the pre-Aug-19 v2 file. Unsloth re-uploaded all UD quants as Dynamic v3 under the same filenames. v3 is better quality (PPL 13.84 vs 14.51) and faster on burst, but collapses at depth: 19.7 vs 27.5 t/s at 8k under the recommended config. The v2 download link is in Sources.
• n-max 4, not 7. At depth, deep drafts get rejected and waste verify bandwidth: n4 is +25% over n7 on long generations, costs ~1.5% on short ones. Nathan's v0.6.5 independently landed on the same number, and my re-sweep under Sharp (where higher acceptance could have shifted the optimum) still says 4.
• The Sharp template is the hidden lever. The stock template burned entire 4k and 8k budgets on reasoning with zero output on my agentic test prompt. With Sharp: the model answers, and sustained decode runs 25-29 t/s from 1k to 90k tokens, peaking at 32k (full-curve table below).
• Skip the FP4 builds: a full PPL tier worse than Q4_K_XL, slower than this stack anyway, and riding a one-off engine (gotcha #3). Tested --spec-draft-p-min 0.7 too: -3% here, an MTP-era trick that doesn't transfer to block drafting.

Quick start

1. Grab Nathan's v0.6.5: I benchmarked the identical payload when it was a dev tag (dev-20260819-0b0f35d); v0.6.5 is the same code, now gfx1151-validated. (v0.6.6/v0.6.7 landed after this bench; they fix DeepSeek V4 quantised-KV and Ling 3.0 tool parsing, and the Qwen path is unchanged, so these numbers carry over.)
2. Download the target: Qwen3.8-27B-UD-Q5_K_XL (20.6GB). Pin revision 408fcc1807ab (pre-Aug-19 v2, the file I benched): plain main now serves Dynamic v3, which trades sustained speed for quality (details in the note below).
3. Download the drafter: incoai/Qwen3.8-27B-DFlash2-GGUF, Q4_K_M (1.1GB). Caveat: the upstream DFlash2 PR is still in review, so if the drafter format changes before merge you may need to re-download.
4. Download the template: chat_template.jinja from the Sharp repo, save as sharp.jinja.
5. Launch:

llama-server \
  -m Qwen3.8-27B-UD-Q5_K_XL.gguf \
  -md Qwen3.8-27B-DFlash2-Q4_K_M.gguf \
  -ngl all -ngld all -fa on \
  -ctk f16 -ctv f16 -ctkd q8_0 -ctvd q8_0 \
  -c 262144 -np 1 \
  -b 4096 -ub 4096 \
  -t 16 -tb 32 \
  --spec-type draft-dflash --spec-draft-n-max 4 \
  --chat-template-file sharp.jinja \
  --jinja --host 127.0.0.1 --port 8080 --metrics

The -ctkd/-ctvd q8_0 flags quantize only the drafter's KV cache: +1.6% t/s, and I verified the output stays bit-identical. No boot params needed; the model is small enough that GTT defaults just work.

The bench matrix below was measured at -c 65536; every number carries over to bigger allocations unchanged, because a larger -c only reserves memory (decode at 8k depth measured identical at 65536, 98304 and 131072). The 256k cap in the command costs nothing until your sessions grow into it.

Results

Config-comparison table: 4-task streaming bench (python, math, JSON, long-prompt; short 100-200 token generations, warm runs), stock template, same engine build throughout.

Read those as burst rates. The number that matters for real work is sustained generation, measured with a fixed agentic coding prompt at n4 with drafter-KV q8_0, 70W:

Two lessons in that table. Burst belongs to Q5 (its distribution sits closer to the drafter's training, so acceptance climbs and pays for the extra weight bytes); at 8k the ranking flips to Q4, because acceptance converges between the quants and Q4's ~2.3GB fewer weight bytes per pass wins. Q5 stays my daily driver for the PPL and burst, Q4 + Sharp is the pick for long-form code, and the 90k figure comes from a single full-context run, so treat it as indicative.

Second, and bigger: with the Sharp template, decode gains speed with depth instead of decaying.

The "spec decode collapses at long context" story turned out to be half a template problem. Stock-template output drifts into sprawling reasoning that the drafter can't predict, so acceptance falls from ~56% to ~24% and t/s follows.

Sharp's terse style keeps acceptance climbing as code accumulates: 0.57 at 8k, 0.72 at 32k, 0.77 at 90k. The depth test: a 90k-token generation, context filled to ~90k, averaged 26.4 t/s end to end (peak stretch 30) with mean accepted draft length 4.1 at the finish. Decode hits slow patches in the high teens past ~60k, but the cumulative rate never collapses; what's left at depth is attention cost in patches, not draft mismatch.

Acceptance percentages are not comparable across methods (DFlash2 drafts blocks in parallel, MTP sequentially), so judge by t/s. Per-task burst spans 21 to 40 t/s depending on workload; that spread is content, not noise.

Prefill: what to expect at depth

Shallow (3k prompt): ~298-302 t/s at ubatch 4096, both spec methods identical. Ubatch 4096 is the knee: 8192 measured flat, 2048 leaves ~5% on the table.

Deep context, f16 KV, llama-bench:

Why the deep numbers hold up: Nathan's fork carries a flash-attention prefill series (dequant-once scratch, KV contiguize, coopmat1 tuning), and the gains grow with depth. I ported that series onto the ROCmFPX fork to A/B it properly: +59% at 32k with f16 KV (165 to 263 t/s). At 64k the unpatched engine didn't just slow down, it threw vk::DeviceLostError on every model I tried, while the ported one ran 199-219 t/s.

The port is here with full numbers, and the findings went upstream.

Why not FP4? The reddit hype, checked

The ROCmFP4 builds were everywhere last week with 30+ t/s headlines, so I ran the whole stack. What holds up: the FP4 engine's Vulkan path genuinely is fast, and it beat the MTP-era configs (23 vs 21.6 t/s). What doesn't: it's slower than Q5+DFlash2, it measures a full PPL tier worse than Q4_K_XL, and the engine is a dead end.

Same-engine, same-method short-corpus perplexity (90KB wikitext-2 slice, c8192, n512, 2 chunks, f16 KV):

The engine is also a one-off: the pinned build the benchmarks ran against was force-pushed out of the repo's history, and the prebuilt crashes at 64k f16 KV. Fun demo, not a daily driver.

Dynamic v3 note (updated Aug 20): Unsloth re-uploaded every UD quant with Dynamic v3 under the same filenames on Aug 19, so all numbers in this post are the v2 files. I A/B'd both Q5s under the recommended Sharp config, same session, 70W. v3 wins two of three: PPL 13.84 vs 14.51 (wikitext-2 full corpus, ~4x the error bars) and burst 30.6 vs 29.2 t/s at 73.7% acceptance.

Sustained is the one that keeps me on v2: at 8k tokens v3 collapses to 19.7 t/s vs 27.5 for v2, with acceptance decaying to 0.40 at depth. For agentic coding that's the whole game; for short generations and maximum quality, take v3 knowingly.

Reproducing my numbers means pinning the revision. The v2 files live at commit 408fcc1807ab (Aug 14, last before the re-upload): Q5 direct is huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/408fcc1807ab/Qwen3.8-27B-UD-Q5_K_XL.gguf, or hf download unsloth/Qwen3.8-27B-GGUF Qwen3.8-27B-UD-Q5_K_XL.gguf --revision 408fcc1807ab. My Q4_K_XL numbers are v2-era too; current main serves a different Q4.

Gotchas (learned the hard way)

1. The stock template can burn your entire budget on thinking. On my agentic test prompt it produced zero content tokens at both 4k and 8k max_tokens: 8192 reasoning tokens, no answer, six minutes of GPU time. That failure mode, not hardware, is the biggest speed killer this model ships with.

The Sharp template fixes it and lifted sustained decode 26% on top.

2. n-max is a depth knob. 4/7/8 are within ~1.5% on short generations, but at 8k tokens n4 beats n7 by 25% (19.8 vs 15.8 t/s). Deep drafts get rejected as acceptance decays, so the long tail just costs verify bandwidth. Nathan's v0.6.5 release notes independently measured the same crossover (width 4: 21.1 t/s vs width 7: 16.3 at 32k), and re-sweeping under Sharp confirmed 4: n5 is par on burst and -2% at 8k.

3. DSpark is broken for this model in llama.cpp right now. The community drafter gives 15-16% acceptance and no speedup, and it's not the drafter's fault: I verified the GGUF metadata and bisected the failure into llama.cpp's DSpark draft generation itself. Details in ggml-org/llama.cpp#25618; use MTP or DFlash2 until it lands.

4. p-min does not transfer. A commenter reported --spec-draft-p-min 0.7 lifting DS4+MTP acceptance dramatically; on DFlash2 here it cost 3%. Block drafting changes the economics, pruning just shrinks the block.

5. First bench run is always cold. First DFlash2 pass read 24 t/s; warm runs 27+. The drafter warms up across requests, so run the workload twice before believing a number.

6. Dense vs MoE is the real choice. Qwen3-Coder-30B-A3B (3B active) prefills at ~1900 t/s on this box vs ~300 here; that's arithmetic, not tuning, so huge-prompt short-answer workloads should use the A3B. For agentic coding where answer quality is the product, this stack is the strongest thing I've run locally.

Sources

• Engine: Nathanw1014/strix-halo-llamacpp v0.6.5 (DFlash2 validated on gfx1151; I benched the identical payload as dev-20260819-0b0f35d)
• DFlash2: https://inco.ai/blog/dflash2/
• Drafter: incoai/Qwen3.8-27B-DFlash2-GGUF
• Template: peculiar-ragdoll/Qwen-Sharp-Chat-Templates
• Target (v2, pinned): unsloth/Qwen3.8-27B-GGUF @ 408fcc1807ab UD-Q5_K_XL + UD-Q4_K_XL; current main = Dynamic v3, see the note above
• DSpark broken draft path: ggml-org/llama.cpp#25618
• My FA-prefill port + ROCmFPX findings: aic0d3r/ROCmFPX port-nathan-prefill, charlie12345/ROCmFPX#86
• Prior work on this machine: DeepSeek V4 Flash guide: https://www.reddit.com/r/LocalLLaMA/comments/1vlmh0b (full guide: https://www.reddit.com/r/LocalAiCore/comments/1vkq5kj)

Happy to answer questions. More Strix Halo benchmarks and guides at r/LocalAiCore.

I've spent a week benchmarking DSv4 Flash 0731 on a Flow Z13 (Ryzen AI MAX+ 395, Radeon 8060S / gfx1151, 128GB LPDDR5X). This is what actually works on this hardware and what doesn't.

Note: the writing is AI-assisted editing; the research, debugging, and every number are from my own runs on this machine.

Edit (Aug 13): since this went up, Nathan shipped the v0.6.3 beta with sparse-attention prefill, and I ran it on my Flow Z13 — the deeper your context, the bigger the win (+39% at 32k, +78% at 64k vs dense attention), and decode stays untouched. Long-context prefill no longer tanks like it used to — full numbers in the prefill section.

TL;DR

• You can run a ~300B MoE locally on a from-$2,920 AMD APU mini PC or laptop — roughly half the price of a DGX Spark — at 27+ t/s decode and ~285 t/s prefill. DSpark is the decode lever, Nathan's Vulkan fork is the prefill lever.
• Both levers come from the same place: MoE kernel tuning. On ROCm, with identical hardware and model, prefill drops ~30% and DSpark goes from +46% to +0%. Supported upgrades don't fix kernel tuning.
• Two config picks matter: q8_0 KV (doubles context to 131k, small quality tail, measured in gotcha #1) and the Q2K-Q8 drafter (frees ~4GB VRAM, no speed cost, gotcha #2). Everything else is table stakes.

Quick start

1. Download the model + drafter from unsloth/DeepSeek-V4-Flash-0731-GGUF: the 4-file UD-IQ3_XXS split (~98GB) and a DSpark drafter, Q2K-Q8 (~7GB) or bf16 (~11GB).
2. Patch the Q2K-Q8 drafter's header with the bf16 drafter's tokenizer if you use it (gotcha #2). Skip this step with bf16.
3. Add the boot params amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856 ttm.page_pool_size=32505856 to your kernel cmdline (how depends on GRUB vs systemd-boot). The model won't load without these.
4. Install Nathan's build — v0.6.1 for these exact numbers, or v0.6.3-beta1 (adds sparse-attention prefill, see the prefill section): Nathanw1014/strix-halo-llamacpp. Bundled RADV, no ROCm.
5. Run the command block below with your model and drafter paths.
6. Connect a client to http://127.0.0.1:8080 (Open WebUI, LM Studio, or any OpenAI-compatible frontend).

Setup

How I measured. All numbers are from one machine, and every comparison is same-session so thermals and clocks match. I used a streaming client bench with the same prompt, 3 trials per config, and I report ranges, not just bests, wherever the run-to-run spread is real (gotcha #7). Isolated prefill/decode numbers come from llama-bench, and DGX Spark numbers are cited from external sources at the bottom.

Hardware and config.

• Target: Unsloth UD-IQ3_XXS, ~98GB, 4-file split
• Draft: DSpark drafter, Q2K-Q8 (~7GB, header-patched, gotcha #2) or bf16 (~11GB)
• KV: q8_0
• Backend: Vulkan via Nathan's v0.6.1 build with bundled RADV, no ROCm. It's a pinned release you install and run as-is, all flags are in this script, nothing to tune day to day. The v0.6.3 beta adds sparse-attention prefill (see the prefill section); these numbers are the validated v0.6.1 line.
• OS: CachyOS, kernel 7.1.6, Mesa 26.1.6

llama-server \
  -m DeepSeek-V4-Flash-0731-UD-IQ3_XXS-00001-of-00004.gguf \
  -md DSV4-Flash-DSpark-draft-bf16.gguf \
  -ngl all -ngld all \
  -fa on \
  -ctk q8_0 -ctv q8_0 \
  -c 131072 -np 1 \
  -b 2048 -ub 2048 \
  --spec-type draft-dspark \
  --spec-draft-n-max 64 \
  --jinja --host 127.0.0.1 --port 8080

Boot params: amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856 ttm.page_pool_size=32505856 (without these the model won't load).

Power: z13ctl+ profile, CPU boost off with min freq pinned. Decode is bandwidth-bound, so the downclock costs nothing and keeps the tablet cool.

VRAM: 4GB BIOS carve-out, GTT covers the rest. Committed footprint is ~105GB (98GB model + 7GB Q8 draft), leaving ~23GB for KV and compute. The bf16 draft pushes that to ~109GB / ~19GB.

Results

Best client-side observation on 4096-token generation. Server-side timing for similar runs consistently shows 23-24 t/s. See gotcha #7 for the variance.

131k context runs on q8_0 KV, which doubles context but costs a little quality. Full measurement of that tradeoff is in gotcha #1.

Cross-platform comparison

Both are unified-memory APUs with similar LPDDR5X bandwidth (~256 vs ~273 GB/s). The Strix Vulkan rows are Nathan's fork, not upstream llama.cpp. Its hand-tuned MoE kernels (the GGML_VK_MMID_* flags) are most of why Vulkan leads here, not the API itself (gotcha #3).

The ROCm row is stock mainline llama.cpp.

Notes: ds4 is ROCm/HIP-only, no Vulkan backend. Its 122 t/s figure predates ROCm 7.14 and used the Q2 quant. Prefill is unaffected by speculation, and decode is bandwidth-bound, so context depth barely moves it.

Strix numbers use the header-patched Q8 drafter where noted (gotcha #2).

Decode is a dead heat. Plain decode clusters in the 12-20 t/s range everywhere, DSpark brings both platforms to ~27 t/s. The same-engine boost is 1.46x on Strix (27.13 / 18.55) vs 1.37x on DGX Spark (27.3 / 20.0). Speculation is the only lever on bandwidth-bound decode, and it behaves the same on both platforms.

Prefill is where platforms diverge. No engine runs optimally on both platforms, so the honest comparison is the best of each: llama.cpp Vulkan at 284.98 t/s on Strix vs the D2R CUDA fork at 960 t/s on DGX, a 3.37x gap. The fork's D2R pass is a clean 2.45x over its own CUDA baseline on identical hardware, which suggests most of the gap is closeable with better shaders.

I measured the Strix side with GPU perf counters (amdgpu gpu_metrics via amdgpu_top, 1 Hz during llama-bench pp2048, same v0.6.1 build and q8_0/ub2048 config; 274–285 t/s across sessions). During prefill the shader array is ~94% busy while DRAM traffic is only ~74 GB/s — about 30% of the ~256 GB/s bus. Strix prefill is shader-bound, not bandwidth-bound.

Effective compute works out to ~4.6 TFLOPS, under 10% of gfx1151's FP16 MMA peak. At ub2048 each routed expert sees exactly 48 tokens (2048 × 6 active experts ÷ 256 total = 48), so the M=48 tiles can't fill the tensor cores. The ~26 GB/s of writes is the dequant-to-fp16 staging pass — real traffic, but the bus absorbs it, so fusing it away wouldn't move prefill much.

What Strix would actually score with D2R-quality shaders remains a projection, so it stays parked in the footnote.

Since writing this, Nathan shipped the v0.6.3 sparse-prefill line (alpha1 → beta1, prefill numbers unchanged), and I validated it on this 128GB box: the raw/selected split is +26% over the single coopmat kernel, and vs dense attention prefill is +39% at 32k depth and +78% at 64k, because the split flattens the depth penalty (−8% vs −28% from 32k to 64k). That confirms the shader-bound diagnosis from the attention side too — a full 3x was on the table in the FA stage at depth, now realized. It's attention-stage work, so the MoE-GEMM lever (D2R-style shaders) in the footnote is still unclaimed, and decode is untouched.

Footnote (prefill math): MoE prefill doesn't scale with raw FP16 TFLOPS. The compute ratio is 4.2x but the measured gap is 3.37x, and expert GEMMs are tiny (M=48 at ub2048: 2048 tokens × 6 active experts ÷ 256 total), so tensor cores stay underutilized — on Strix now measured directly (94% shader busy at ~30% of DRAM peak, ~4.6 effective TFLOPS). If Vulkan's MoE shaders matched D2R quality, Strix would project to ~698 t/s (284.98 × 2.45), leaving a residual gap of ~1.4x. The 698 t/s figure remains a projection, not a measurement.

Gotchas (learned the hard way)

1. q8_0 KV: faster + more context, but it costs quality. The speed crossover first, measured same engine and prompt shape on single runs per config:

At short context q8_0 is ~6% slower because per-element dequant overhead dominates a small cache. At long context it's +18% faster because it halves bytes-per-step, and it doubles context (131k vs 65k). This is a crossover observation, not a precision delta, so treat the percentages as directional.

The real cost is quality. I ran llama-perplexity --kl-divergence (bf16 baseline vs q8_0) on ~50k tokens of prose with the same model and Nathan's build:

Mean PPL barely moves (1.0015x), but the tail is real: the 99.9th-percentile KLD is 64x the mean, with ±40% probability swings on rare tokens. The average stays clean while a thin slice of tokens goes somewhere else. My numbers look ~10x better than erazortt's, but that's corpus: Moby Dick (PPL 1.54) is far more predictable than wikitext (PPL ~5.8).

On harder text, expect degradation closer to his figures, so treat mine as a lower bound. Root cause: DS4F's MLA is built around fp8 KV natively, which llama.cpp doesn't expose, so it's f16 or q8_0 (worse dynamic range than the model expects). That's a tooling gap, not a choice.

For precision or agentic work, stay on f16 KV at 65k context. Methodology thread: https://www.reddit.com/r/LocalLLaMA/comments/1vduxth

2. The draft doesn't need to be bf16. Use the Q2K-Q8 drafter and keep ~4GB VRAM. The ~7GB quantized drafter crashed at first with invalid token = -1, but that was a broken conversion, not a precision problem. The file had its tokenizer metadata stripped and declared tokenizer.ggml.model = no_vocab, under which llama.cpp nulls every special-token id (including the draft mask token) to -1. Patching the header with the bf16 drafter's tokenizer fixes it.

Same-session, same q8_0 KV config, 4096-token runs: decode 25.3-27.8 t/s (best 27.78, mean ~26.3 vs bf16 ~24) and draft acceptance 0.657-0.740 (mean 0.687 vs bf16 0.606). Both differences sit inside the run-to-run noise band from gotcha #7, so count them as "no speed or acceptance cost", not wins. The drafter is a few billion params in a bandwidth-bound loop, so Q8 quantization costs nothing measurable.

The 4GB of freed VRAM is the real take.

3. Vulkan beats ROCm on gfx1151, but "supported ≠ tuned". Fair pushback in the comments that I'd judged ROCm on the ds4 engine and a pre-7.14 build. So I benched mainline llama.cpp + ROCm 7.14 properly: gfx1151 went officially supported on 2026-07-16 and loads natively now, no HSA_OVERRIDE or TheRock hacks. Same model, same session, q8_0 KV, -fa on, -ub 2048, 80W sustained:

ROCm 7.14 fixed support. It did not fix kernel tuning for DSv4's MoE: about 30% behind on prefill, 28% on decode, and the gap widens with context. From pp2048 to pp8192, ROCm drops 14.5% while the fork drops 6.9%, so the "ROCm wins long context" story doesn't hold on this APU.

DSpark acceptance is a dead heat (means 0.606 vs 0.613), so the drafter produces equally good tokens on both, but on ROCm the draft runs on the same slow MoE path and its cost cancels the ~4 accepted tokens per pass. Net: +0% on ROCm, +46% on Vulkan.

The mechanism is concrete. ROCm spills ~15% of work to the CPU, which on a unified-memory APU steals bandwidth from the GPU it's trying to help. That's the bursty load and the temperature spikes.

The fork's MoE kernels keep everything on-GPU and back to back. So this is stock-mainline-ROCm's generic gfx1151 kernels vs a hand-tuned MoE Vulkan path, not "ROCm vs Vulkan" as APIs. On a dense model, or any MoE where those flags don't fire, ROCm can absolutely win.

The broader ask: the 8060S has the bandwidth to be a real local-inference answer to the DGX Spark, and decode is already a dead heat. What's missing isn't hardware or a support checkbox, it's kernel tuning. Right now one community member's Vulkan fork beats AMD's official ROCm backend on AMD's flagship APU by ~30%, and turns DSpark from a +46% win into 0%.

If AMD wants Strix Halo taken seriously against CUDA, the MoE kernels in ROCm need the same attention the Vulkan path got from one person. Supported ≠ tuned.

4. Ubatch 2048 is the VRAM ceiling with DSpark. At 109GB committed (98 + 11GB bf16 draft), ubatch 4096 OOMs. The Q2K-Q8 draft (gotcha #2) moves that ceiling to ~105GB / ~11GB free, but since ubatch >2048 isn't a prefill win on this MoE, there's no reason to chase it. Measured same-session on the ngram config (f16 KV): pp2048 at ub2048 = 293.33 t/s and at ub8192 = 292.77 t/s, identical within noise.

Prefill saturates at ub2048 here. The ngram config does edge DSpark on prefill (~293 vs ~285), but that ~3% comes from f16 KV (which halves context to 65k), not from ub8192 or dropping the draft. It's a context-vs-prefill tradeoff, not a free speed win.

Decode drops to 18.12 t/s without DSpark, and ngram's cache from a 2.3K-token prompt is too sparse for general prose (3 speculation attempts over 4096 tokens). Ngram is a capacity lever, not a speed lever.

5. v0.6 regression: upgrade to v0.6.1. Nathan's v0.6 has a stride bug in TENSOR_ALLOW_RESHAPE that miscomputes strides for block-quantized tensors. Vulkan silently falls back 43 attention layers to CPU, showing 75% GPU and 50% CPU while running at half speed. v0.6.1 fixes it. Bug report: https://github.com/Nathanw1014/strix-halo-llamacpp/issues/2

6. GPU util drops to ~92% on v0.6.1 (99% on the 0.4 build, same config). The new MoE shaders (ROWLISTS, SMALLN, BM64, M128, F16B, FA_WAVE32) finish each kernel faster, so fixed dispatch overhead becomes a bigger fraction. Lower utilization, higher throughput (27.13 vs 20.88 t/s). Don't let the number fool you.

7. DSpark decode varies run-to-run on the same prompt. Best client-side observation: 27.13 t/s at 0.7209 acceptance. The same session's three trials spanned 20.96–27.13 t/s and 0.507–0.721 acceptance, the drafter warming up across requests — variance tracks GPU thermal/clock state and swings with output content. If you benchmark anything here, run the same prompt 5+ times and report the range, not a single number.

Sources

• Engine: Nathanw1014/strix-halo-llamacpp v0.6.1
• Sparse prefill (v0.6.3): v0.6.3-beta1 release (alpha1 → beta1, prefill numbers unchanged)
• Model: unsloth/DeepSeek-V4-Flash-0731-GGUF
• DGX Spark DSpark numbers: Entrpi/ds4-on-spark v0.5.6 (2026-08-08)
• Model specs: deepseek4 arch, 43 layers, 256 experts (6 active + 1 shared), 1M native context, MIT license. Technical report: https://arxiv.org/abs/2606.19348

More Strix Halo benchmarks and guides at r/LocalAiCore if you're interested in this hardware.

Following the Z13CTL+ and Z13GUI+ releases, I have released the third part of the stack: z13-tablet-kit.

https://github.com/aic0d3r/z13-tablet-kit

It adds posture-aware tablet support for the AMD Strix Halo Flow Z13 (GZ302) on KDE Plasma 6 / Wayland.

The kit automatically distinguishes between three modes:

- Desktop: dock or external display connected

- Laptop: keyboard folio attached

- Tablet: neither attached

It then adapts the system for each posture:

- Enables touchscreen and auto-rotate in tablet mode

- Uses 175% display scaling in tablet mode and 150% otherwise

- Enables the on-screen keyboard only in tablet mode

- Prevents the on-screen keyboard from appearing over the physical keyboard

- Restarts the keyboard after posture changes to prevent clipped layouts

- Provides two-finger scrolling in terminals, browsers, document viewers and other applications

- Optionally maps a stationary two-finger hold to a real right-click for context menus

- Creates the missing tablet-mode switch needed by Plasma’s “rotate only in tablet mode” option

- Disables the unreliable folio touchpad by default, with a guarded opt-in toggle

It uses Plasma’s native rotation support and does not require kernel patches or a root daemon. Only two udev rules require root access; the posture and gesture services run in the active user session.

The full stack consists of:

z13-tablet-kit

Posture detection, tablet mode and touch input

https://github.com/aic0d3r/z13-tablet-kit

Z13CTL+

Display coordination, persistent settings and hardware control

https://github.com/aic0d3r/z13ctl-plus

Z13GUI+

Optional touchscreen overlay with live posture status and tablet controls

https://github.com/aic0d3r/z13gui-plus

The kit currently targets Arch/CachyOS with KDE Plasma 6 on Wayland. z13ctl-plus is required, while z13gui-plus is optional.

Installation instructions and technical details are in the README. Testing, bug reports and contributions are welcome, especially from other GZ302 owners.

I have released Z13CTL+ and Z13GUI+, an expanded Linux control and telemetry stack for the Strix Halo-based 2025 ROG Flow Z13.

Before anything else, special thanks to dahui, who created the original z13ctl and z13gui projects. These releases build on that foundation and would not exist without the work already done there.

My goal was to add more automation, tuning and telemetry while preserving compatibility with the original commands, services, paths and API.

Z13CTL+
https://github.com/aic0d3r/z13ctl-plus

Z13GUI+
https://github.com/aic0d3r/z13gui-plus

Z13CTL+ is the daemon and CLI. It adds:

• Automatic switching between plugged-in and battery configurations
• Independent TDP, fan-curve and undervolt overrides
• AMD P-State controls for minimum CPU frequency, EPP and boost
• CPU, GPU, NPU, memory, VRAM, temperature, fan, battery and power telemetry
• KDE display posture, rotation, scale and refresh-rate coordination
• Recovery for lighting and firmware fan behavior
• CLI and API access for scripting and automation

Z13GUI+ is a GTK4 interface designed around the Z13 touchscreen. It includes:

• A live hardware and power overview
• Separate AC and battery profiles
• CPU policy controls
• 60 Hz and 180 Hz refresh-rate switching
• Battery information
• Fan presets and tuning controls
• Clearer feedback when changing tuning values

The tools are currently focused on the 2025 Flow Z13, but the telemetry side may also be interesting to other Strix Halo users, especially the CPU, GPU, NPU, unified-memory, VRAM, power, clock and temperature monitoring.

Packages are available through the AUR, along with DEB, RPM and portable release archives.

I would be interested in hearing what telemetry and power-management controls are exposed on other Strix Halo systems under Linux, and whether parts of this could eventually be generalized beyond the Z13.

Testing, bug reports and contributions are welcome.