I've been tuning llama.cpp on my Strix Halo box (Radeon 8060S, 128 GB) and the numbers are good enough to share. It's a branch called rdna35-perf on top of halo-box/strix-llama.cpp, a llama.cpp fork aimed at Strix Halo:

https://github.com/asf0/strix-llama.cpp/tree/rdna35-perf (10 commits, ROCm/HIP only)

TL;DR

• Prompt processing: 21–54% faster than Vulkan, and 8–174% faster than stock ROCm. The lead over ROCm grows with context, because stock ROCm flash attention falls off hard at depth.
• Generation: ties Vulkan on the dense 27B. On the MoE and the 2B it's 2–3% behind Vulkan at short context and within 1% from 64k up. 5–29% faster than stock ROCm.
• Quality: outputs stay closer to stock ROCm than Vulkan's do, and perplexity is unchanged.

Results

llama-bench, Q8_0 weights, q8_0 KV cache, flash attention on, tok/s. "Depth" is how many tokens are already in the KV cache when the test runs. Vulkan and ROCm are the same strix-llama.cpp master that rdna35-perf is built on.

Qwen3.6 35B-A3B (MoE, ub 4096)

Qwen3.8 27B (dense, ub 2048)

Qwen3.5 2B (dense, ub 2048)

f16 KV gives the same picture: PP 20–56% faster than Vulkan and up to +171% over stock ROCm at 128k, TG within −3% / +2% of Vulkan.

Quality

rdna35-perf changes kernels, so I checked it doesn't change outputs: KL divergence against stock ROCm on wikitext-2 (6 × 4096-token chunks), plus perplexity at 32k context.

rdna35-perf is closer to stock ROCm than Vulkan is, and perplexity is the same across all three builds.

What changed

• Flash attention kernel for RDNA3.5 (fattn-rdna35). This is the biggest win and the reason long-context PP holds up. GGML_FA_RDNA35=0 turns it off if you want to compare.
• Prefill GEMM tuning for the quantized matmuls.
• MoE: routing helper, fused reduction + residual, grouped mat-vec for speculative-decoding verify batches.
• Gated DeltaNet (the linear-attention layers in Qwen3.5/3.6/3.8): fused SiLU-gated output norm and fused decode.
• Decode: plain mat-vec for rows longer than 4096 (this closed the 27B decode gap to Vulkan), plus decode fusions for the Qwen3.6 MoE.
• Misc: HIP graphs off by default on RDNA3.5 (slower here), a faster penalties sampler, host-side prefill for Qwen3.5/3.6 with a draft model, and DFlash drafts read target features in place on the GPU.

I have only tried those 3 models so far.

Running Qwen3.8-Flash-Next locally on a BOSGAME M5 with:

• AMD Ryzen AI Max+ 395
• Radeon 8060S / Strix Halo
• 128 GB unified RAM
• Ubuntu 26.04
• Halogen 0.17.2
• 262K context configured
• Vision enabled

This is my Grafana monitoring dashboard during a real long-running workload, not a short synthetic benchmark.

In the screenshot:

• Decode: ~69 tok/s peak
• Prefill: ~1,155 tok/s at that moment, with peaks around 1,685 tok/s
• Context: ~84% full
• 82K tokens generated in the last hour
• GPU temperature around 62°C
• GPU power around 24 W at the captured moment
• No meaningful memory pressure despite the system showing ~121 GB physically occupied, because most of it is reclaimable model/page cache
• ~85 GB currently sitting in cache

One thing I really like about this setup is how usable Qwen3.8-Flash-Next remains with a large context and sustained agent workloads. Decode stays around the 60–70 tok/s range while prefill is still comfortably above 1k tok/s for much of the workload.

The dashboard is fed by Prometheus/Grafana and tracks Halogen throughput, KV/context usage, real vs reclaimable RAM, PSI memory pressure, GPU metrics, backend status and request activity.

Still testing it, but so far Strix Halo + unified memory is proving to be a very interesting platform for large local models.

LemonSeed Studio is an iPad editor/IDE with on-device inference on an AMD GPU in a Thunderbolt enclosure. It embeds the unmodified upstream Linux amdgpu + amdkfd driver (mac_linuxgpu) as a PCIDriverKit extension, and runs LemonSeed Engine (LSE) on the GPU. In the photos: iPad Pro + Sapphire Radeon AI PRO R9700, Qwen3.8-27B Q4 with a Q8 DFlash2 draft, 131k context.

LSE is the same engine on every platform: Linux, macOS and iPadOS. It records the model's forward pass as a graph, fuses ops and generates kernels for the GPU it's on. Where a kernel has several layouts, it measures each on the device and keeps the fastest. Speculative decoding (MTP and DFlash2) picks how many draft tokens to verify from measured acceptance and cost.

Decode, code prompt, Qwen3.8-27B Q4 (baseline / MTP=3 / DFlash2 tok/s):
- R9700 on iPadOS (LemonSeed Studio): 31.2 / 108.7 / 158.5
- R9700 on macOS: 32.2 / 111.9 / 159.4
- R9700 on Linux: 32.0 / 111.0 / 144.6
- Strix Halo (Radeon 8060S) on Linux: 14.0 / 47.8 / 64.4

Prefill at 4K tokens: 1,636 (iPadOS), 1,634 (macOS), 1,418 (Linux R9700), 517 (Strix Halo). It holds up at 32K: 1,395 / 1,401 / 1,226 / 462.

New in 0.5.8:
- DFlash2 draft trees on by default: one target pass verifies a tree of draft candidates when that's measured to be faster than a chain
- Strix Halo (gfx1151) prefill and decode work: fused gate/up GEMM, two query tiles per workgroup in prefill attention
- Fixed a startup GPU fault
- Linux archive bundles its own HSA runtime, so you only need the amdgpu driver with /dev/kfd access
- lse-server models / pull: grab Hugging Face models with their MTP or DFlash2 companions

OpenAI-compatible server, CLI, and a C library (libLSE).

Engine: https://github.com/Geramy/LSE
Studio: https://github.com/Geramy/LemonSeed-Studio

Happy to answer questions, especially about getting an eGPU working on iPadOS.

Recently tried running the Gufo engine (GitHub - gufo-org/gufo: Strix Halo inference engine. Qwen Flash Next Q4_K_XL: 1,700.52pp, 60.39tg single user, 162.98 tok/s 8 users; Qwen27B Q4_K_XL: 656.33pp, 70.56tg tok/s single user with DFlash2 · GitHub) on my Strix Halo device (GPD Win5 128G).

It's performance is close to the Halogen (much better than llama.cpp variants on the prefill performance). I'm switching to this as daily use.

Also created a short video as introduction on this: https://youtu.be/r-pFZzIMvgU

Sharing a ComfyUI node I've been working on, since video gen on Strix Halo is where a lot of us keep hitting the wall.

Credit first: Veda is a learned sparse-attention method for video diffusion models (ICML 2026). A small distilled predictor says in advance which tiles of the attention map actually carry the result; only the top ~10% get computed. The original project is by veda-sparse:

- Project page: https://veda-sparse.github.io/

- Original repo: https://github.com/veda-sparse/Veda-on-ComfyUI

- Predictor model file (275 MB): https://huggingface.co/Veda-Sparse/Minimax-H3-T2VA-Veda-8NFE-600Step-Preview — goes in ComfyUI/models/veda/ (opening a Veda template offers it in the missing-model dialog)

Upstream supports NVIDIA (SM80+) and Apple silicon, and lists ROCm as "no kernel, the model runs its own attention". So I ported it to ROCm and tuned it for Strix Halo. What that took:

- Backend layer: report the real gfx family (gfx1151) and unified memory on HIP, and admit ROCm devices to the Triton INT8 backend (kernels still self-test on the GPU before use, same as upstream)

- Triton's ROCm backend compiles the INT8 kernel for gfx1151: Q and K are block-quantized to int8, QK^T runs on the hardware int8 WMMA instructions (v_wmma_i32_16x16x16_iu8), P*V stays fp16 WMMA

- The launch config upstream shipped (4 warps / 3 stages) was tuned on an RTX 5070 and is wrong on this chip: measured on the real portrait grid, 8 warps / 2 stages is 2x on the attention kernel (35.3 -> 17.6 ms per call)

- A few more gfx1151-specific kernel tweaks, e.g. the quantize pass runs 23% faster with 2 warps, bit-identical output

- Accuracy verified on this hardware: kernel vs the INT8 reference is ~0.14% relative L2, below the quantization noise that's already there

Numbers on my machine, MiniMax H3, 11-second clip at 480x864:

- dense attention sampler: 691s

- sparse (Veda): 366s

- 1.89x on sampling, 2.15x on the full prompt (1016s -> 473s)

- only ~27% of the attention is computed, output looks the same to me

My ROCm port lives in rocm-ninodes (ComfyUI Manager, or comfy node install rocm-ninodes); the node is VedaSparseAttention, drop it on the MODEL wire last before the sampler:

- https://github.com/iGavroche/rocm-ninodes

If you try it on your Halo I'd be curious about numbers on other resolutions/lengths. Happy to answer questions about the sparse selection or the ROCm port.

tl;dr: Veda Sparse Attention on ROCm (optimized for Strix Halo)

tldr: the gap in the title is what the NPU tools buy on a real bug fix. And local now covers about 95% of my cloud calls, the hardest few percent still goes to GLM 5.3 or Opus. Receipts in the repo.

Qwen3.8 Flash-Next, a 125B MoE, runs my pi coding agent on a 70W tablet, and the NPU finally earns its power draw. Wired halogen's endpoints expecting a party trick, kept what paid.

v1.1 since launch: opt-in voice on the CPU, search moved to semble. The tools that stayed on the NPU:

• Decisions. Yes/no branching stops eating turns of the big model. A 0.8b answers in 120ms, 78% accurate.
• Dup scan. Catches copied and renamed files git never shows you. 45 pairs across 4 repos in 8.4s.
• Screening. Injection attempts get flagged before the agent acts. 0.7s, zero false alarms, fails open. 42% recall: a smoke detector, not a safe.
• Search. The agent lands on the right file first try, on any install. The NPU pipeline beat ripgrep 15/20 to 9/20; semble replaced it at 17/18 to 13/18 on the ground-truth A/B.
• Compaction. Big sessions summarize on a tiny sidecar, off the critical path. My 194k session: 50s, 97% cache hit.

The eight-run check was about taste and depth, not throughput: I pasted a real timeshift error to seven agents, the 125B at medium and max against 320B and 753B tiers at max. Eight runs, all healthy.

• Flash-Next, 2m55s: journalctl trace to a racing notify-send, pacman.log check, upstream PR found.
• glm-5.3-flashx, 2m41s: the most precise answer, spotting the snapshot mount unmounted under the script's last line. No PR.
• GLM 5.3 753B at max, 8m30s: the deepest forensics, source-level down to function names and the one-second race window. No PR.
• Flash-Next at max, 8m31s: level with the 753B, and still the only model with the PR.

I expected cloud to be faster. It never pulled away: 14 seconds, then one, GLM at max both times. The PR was not a tie.

All eight side by side. In opencode: flash called it in 1m8s with the wrong mechanism, the full 753B correct in 6m15s, PR missed. And when I had GLM 5.3 flashx rate both results, it picked the qwen answer too.

Reality: the GPU still does the thinking, ~7% iGPU cost only when they overlap. Decode: 64 tok/s best case, 45-50 sustained at 70W. Cap the box at 50W, give up a couple of tok/s, stay under 75C and quiet. First token in ~0.03s, 43x faster than the cloud call beside it, 7x with a second agent.

The official halogen launch is a 24-flag docker command. Mine is one, and uninstall undoes it. Fully local, 262k context. No Strix Halo? setup.sh --no-halogen runs pi against your own providers.

Anyone else putting their NPU to real use? I found nothing.

I've been running an always-on assistant at home on a couple of 128GB Strix Halo boxes for a few months, and struggled with getting solid cache hits. I tried two serving slots, with one for interactive and one for background, but didn't have a reliable way to route requests.

I ended up with a little proxy that sits in front of llama-server and deals with that. This then grew to a request scheduler, metrics dashboard and now added support for the Gufo engine (a newer engine aimed at Strix Halo: https://github.com/gufo-org/gufo ) too! It's currently running on both my boxes, one with a single-slot Gufo/Qwen3.8 Flash-Next and one with a four-slot Gufo/Qwen3.8 27b.

I've just tidied it up and made it public in case it's useful to anyone else: https://github.com/headbouyJB/slotpin

What it does:

• With llama-server: Pins each conversation to a slot, so the next turn lands where its cache already is.
• Sorts requests into lanes with regexes (my chat messages vs cron jobs and other background stuff), so the interactive lane is protected and background work waits its turn without being starved.
• With llama-server: Saves the interactive slot's KV to disk and restores it if something evicted it. On my box a returning conversation went from an 88 second cold turn to 0.7 seconds to restore, in a direct test of exactly that case.
• Writes one line of metrics per turn: time to first token, decode and prefill speed, cache hit, queue wait, memory, and optionally energy from a smart plug, so you get Wh per request. No prompt or reply text is recorded anywhere.
• Has a dashboard (screenshot below) with live slots, performance, cache, health and system views, plus a list of recent turns you can filter down to the slow ones or the cache misses.
• Works in front of Gufo as well as llama.cpp. Gufo does its own caching, so in that mode the proxy steps back and just does lanes, scheduling and metrics.
• Has an optional "fail fast" mode. I had an engine stall for 55 hours while still answering health checks and sending keep-alive pings, so my client never fell back to anything else. The proxy can now spot "no output and no token progress" and return a proper error instead of hanging. That one’s brand new... I’m only just turning it on myself!

Fair warning:

• It's alpha. Config keys and metric fields might still change.
• It's been run on exactly two machines, both mine, both Strix Halo on Linux. The test suite is decent and CI is green, but I'd expect rough edges on other setups.
• There's no authentication. It listens on localhost by default and the README says how to expose it sensibly.
• I'm not a professional developer. Most of the code was written with Claude, with me testing it on real hardware and my real serving environment.

If you just want to see the dashboard without setting anything up, there's a demo mode with made-up data:

git clone https://github.com/headbouyJB/slotpin && cd slotpin
uv run python tools/demo_dashboard.py

What I’m curious about is what everyone else puts between their client and their server. Are you scheduling or prioritising requests, routing different kinds of work to different slots or models, or just letting everything hit llama-server and living with it? That middle layer is where most of my pain was, and I can’t tell if I’ve solved a common problem or just my own.

I built LlamaStash to scratch a personal itch: I run local models through llama.cpp on AMD Strix Halo and got tired of writing the same llama-server wrapper script for the tenth time.

Ollama and LM Studio both wrap llama.cpp but hide too much (and cost real performance). Raw llama-server is fast but tedious. LlamaStash is the middle ground.

What it does:

• llamastash init — first-run wizard. Detects your hardware (CUDA / ROCm-HIP / Metal / Vulkan / CPU), installs llama-server, scans your existing HuggingFace / Ollama / LM Studio model caches, recommends a GGUF that fits your VRAM, downloads it, writes a tuned config, smoke-launches it.
• TUI + CLI + daemon + OpenAI-compatible proxy in one Rust binary. The proxy at 127.0.0.1:11435/v1 lets OpenCode, Cline, the OpenAI SDKs, and llm-cli work as-is. There's also an opt-in --ollama-compat mode that takes port 11434 and answers the byte-exact "Ollama is running" handshake.
• Multi-model concurrency with per-model port allocation, /health-probed state machine, intelligent context auto-fit (sidesteps llama.cpp's --fit collapse on Linux iGPUs).
• Agent-friendly CLI: every TUI capability has a CLI subcommand, --json is a stable agent contract, documented exit codes per failure class.
• In-TUI HuggingFace browser with search, sort, paginate, per-file hardware fit, download with cancel.

On performance — this is the part that matters for this sub.

LlamaStash spawns the unmodified upstream llama-server. So the wrapper should add zero overhead. I measured it. Across AMD APU (Ryzen AI Max+ 395), Apple Silicon, and NVIDIA, on four model sizes (small E2B Q4, mid 31B Q4, large 27B Q8, large MoE 35B-A3B Q8), every cell matches raw llama-server within ≤1%.

Cross-tool numbers on AMD APU (decode tok/s / TTFT ms on chat_turn):

LM Studio wins decode on small/mid/large_dense (their Vulkan path is well-tuned on gfx1151) but loses on the MoE and pays a 1-1.5s TTFT tax from its OpenAI shim. Ollama is consistently slower, and its RAG prefill is catastrophic (cold prefill every rep — 4 min on a 31B). Mac and NVIDIA tables are in the benchmarks page.

Methodology, variance gates, fairness rules, and per-cell JSONs are all checked in. The harness is reproducible: make bench-end-to-end. Tear it apart.

What it's not:

• Not an Ollama fork or replacement (though --ollama-compat exists for tools that auto-detect Ollama).
• Not a model hub.
• Not a llama.cpp fork. Same upstream binary.
• Not a hosted service. Loopback-only in 0.0.2. LAN + auth + TLS are on the roadmap.

Install:

curl -fsSL https://llamastash.dev/install.sh | sh   # macOS + Linux one-shot
irm https://llamastash.dev/install.ps1 | iex        # Windows 11 (PowerShell, no admin)
scoop bucket add llamastash https://github.com/llamastash/scoop-llamastash && scoop install llamastash
brew install llamastash/llamastash/llamastash       # Homebrew (macOS + Linuxbrew)
yay -S llamastash                                   # Arch Linux (AUR — source build)
yay -S llamastash-bin                               # Arch Linux (AUR — prebuilt binary)
yay -S llamastash-git                               # Arch Linux (AUR — main checkout)
cargo install llamastash                            # any Rust toolchain

Then llamastash init and you're up.

Platform: Linux (x86_64, aarch64), macOS (Intel, Apple Silicon), Windows 11 (x86_64). aarch64-pc-windows-msvc and Windows AMD GPU detection on the roadmap.

Honest tradeoffs: Single-author project. Bug reports especially welcome on hardware I don't own. The OpenAI-compat surface covers chat/completions, embeddings, rerank; Anthropic /v1/messages shim is coming.

Repo: https://github.com/llamastash/llamastash

Blog post with the full story: https://deepu.tech/introducing-llamastash

Benchmark methodology: https://deepu.tech/benchmarking-llamastash

Happy to answer questions in the thread.

I want to thank everyone for the kind words, trust, and encouragement in the development of this project. I am doing my best to make the halogen UX amazing for you. I also want to thank AMD's Jeremy Fowers. u/jfowers_amd has been nothing but supportive and sent me a development Strix Halo machine which has been taking a beating over the last couple weeks.

halogen-flash-server 0.16.0 is out.

It runs Qwen3.8-Flash-Next on the Strix Halo GPU, and this release puts the chip's other accelerator to work too.

The Ryzen AI NPU now serves small models beside the Flash model, on the same port, while the GPU keeps generating.

What's new

Small models on the NPU, with one variable. Set HALOGEN_NPU_MODELS and the server runs them on the NPU. Leave it out and nothing changes.

• decider-0.8b makes decisions. Give it a text, a question and 2 to 10 options, and it returns a probability for every option from one pass.
  • Use it as a guard, a router or a classifier.
• qwen3-embedding-0.6b answers `/v1/embeddings`, in the OpenAI request shape.
• qwen3-reranker-0.6b answers `/v1/rerank`, in the Cohere and Jina request shape.

Your own fine-tunes. Put a fine-tune of any of the three on the models volume and name its path. The server converts it for the NPU at the first start and serves it under the directory's name.

A smaller checkpoint, opt-in. qwen38-flash-next-ht43.hgn pins about 8 GiB less memory than the default checkpoint. Prefill is several percent slower, and decode with the default drafter is a few percent slower. The default checkpoint does not change. Select it with HALOGEN_CHECKPOINT=/models/qwen38-flash-next-ht43.hgn.

96 GB: On a 96 GB machine, run ht43 with HALOGEN_MAX_TOK=16384 and HALOGEN_CTX=131072. That leaves room for one conversation of 131k tokens or four of 32k.

Fixes. A long prefill can be cancelled now (#122, thanks u/semidark). The draft counters count every drafted token (#124, thanks u/wszgrcy).

How fast is the NPU?

• A decision with Qwen3.5-0.8B (the architecture `decider-0.8b` uses) takes 49 ms at 64 tokens, 84 ms at 256 and 183 ms at 1k. Eight decisions of 256 tokens in one pass take 41 ms each.
• Batched embeddings run at about 9,400 tokens a second through the server, eight texts of 512 tokens at a time. One text of 512 takes about 97 ms.
• The reranker scores 19 pairs a second, eight pairs of 512 tokens at a time.

The NPU and the GPU together

Something worth knowing even if you never run halogen. On Strix Halo, GPU work and NPU work at the same time can hang the whole machine while the GPU's fabric clock changes speed. Holding that clock at its top level stops it, at a cost of about a watt. The server holds it itself when it runs as root with /sys mounted. For rootless podman, a small systemd unit in the repo holds it at boot. If you run an NPU program next to a GPU program and have seen freezes, this may be why.

The Flash model runs somewhat slower while the NPU works, since the two share the memory and the power budget. With the NPU idle there is no cost.

Setup

The host needs the amdxdna driver, XRT with its NPU plugin, the NPU firmware, and the IOMMU on. If you added amd_iommu=off for GPU speed, it also turns the NPU off. iommu=ptkeeps it. The full setup, every request shape and the fine-tune rules are in https://github.com/peonist-ai/halogen-flash-server/blob/main/docs/NPU.md

podman pull ghcr.io/peonist-ai/halogen-flash-server:0.16.0

Repo: https://github.com/peonist-ai/halogen-flash-server

Weights: https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next

Discord: https://discord.gg/bcm6QknaV6

How to read this. FastFlowLM's numbers are from its own published benchmark pages. They were measured on a different chip (Ryzen AI 7 350) with older versions than its current one, in its default power mode. FastFlowLM lists no embedding model, so its Qwen3-0.6B row (the same backbone) stands in for it. Ours are medians from our own testing on Ryzen AI Max+ 395 machines, and where we measured two machines, the slower one is shown. This is not a same-machine test. If you run both on one machine, we would like to see your numbers.