Minisforum launches $3,599 N5 MAX NAS with Strix Halo and 128GB memory
videocardz.comu/RenatsMC — 1 day ago
r/StrixHalo
Qwen3.8-27B benchmarks on Strix Halo - Q8_0 MTP: 21 t/s, 3x. ROCmFP4 MTP: 26 t/s
TL;DR: On a Strix Halo laptop (Radeon 8060S, 128 GB unified), MTP takes Qwen3.8-27B Q8_0 from 7.1 to 21.9 tok/s. The julianmb/q38rocm FP4 fork gives 26.7 tok/s.
Setup: LlamaStash driving stock llama.cpp b10503 (5112b9738) for ROCm and Vulkan, and the q38rocm fork (server build 246) for ROCmFP4, all built from git today. Greedy, 256 max tokens, wall-clock tok/s from the API usage numbers, best of 2 reps, on AC power. Everything launched through my tool LlamaStash (v0.2.0) so the flags are reproducible.
Qwen3.8-27B decode tok/s (code prompt / prose prompt):
| Build and model | MTP off | MTP on | Draft acceptance |
|---|---|---|---|
| ROCm + Q8_0 (29 GB) | 7.1 | 21.9 (draft 5) | 73% |
| Vulkan + Q8_0 (29 GB) | 7.4 | 21.3 (draft 5) | 73% |
| ROCmFP4 FAST (14.6 GB) | 12.5 | 26.7 (draft 6) | 86% |
Draft length sweep, code prompt tok/s:
| draft n | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|---|---|---|---|---|---|---|---|
| Q8_0 | 15.4 | 18.4 | 19.1 | 21.9 | 21.2 | 21.8 | 20.5 |
| FP4 | - | 23.3 | 24.9 | 25.0 | 26.7 | 25.6 | - |
Observations:
- MTP really makes a difference 2.5x to 3.1x on Q8_0, 2.1x on ROCmFP4, at 73-95% draft acceptance. The model verifies every token, so quality does not change on stock llama.cpp.
- Draft length of 5 works best. The backend default is 3; 5 was best for Q8_0 and 6 for FP4.
- FP4 is a double win if you can live with the quant. Half the memory (29 to 14.6 GB), 1.8x faster raw decode without MTP, and 26.7 tok/s with MTP, which is 3.8x my stock Q8_0 baseline.
- Engine choice barely matters for decode at this size. ROCm, Vulkan and the rocWMMA build all land within about 3%. Vulkan does lose roughly half its prompt processing throughput though, so ROCm stays my default.
- Stuff that did not help: KV cache q8_0 (helps at draft 3, slightly worse at 5), flash attention off (minus 5%).
- One caveat on the ROCmFP4 fork: completion lengths drifted a little across draft settings (152-158 tokens for the same prompt) while stock llama.cpp was stable at 164, so the fork's speculative path is not strictly greedy-identical yet.
LlamsStash defaults were used for all runs. The only flags set were to enable disable MTP ( --spec-type draft-mtp), set draft length (--spec-draft-n-max N) and the --fit-ctx 16384 flag was used to ensure the GPU layers were fully utilized.
The below are the default flags that was applied. Some sent by LlamaStash and others are llamacpp defaults.
| Flag | Value | Set by |
|---|---|---|
--host |
127.0.0.1 |
LlamaStash (loopback policy) |
--port |
41100 |
LlamaStash (auto-assigned) |
-m |
<model>.gguf absolute path |
LlamaStash (resolves the name ref) |
--mmproj |
mmproj-F16.gguf |
LlamaStash (auto-paired sidecar; Q8_0 rows only, FP4 has none) |
--jinja |
on | LlamaStash (chat template handling) |
--spec-type |
draft-mtp |
LlamaStash (translated from --mtp on) |
--spec-draft-n-max |
5 (Q8_0) / 6 (FP4 best) / omitted on default rows |
LlamaStash (--mtp-draft-n); llama.cpp default 3 when omitted |
--fit-ctx |
16384 |
LlamaStash (ctx knob = auto, fit hint) |
--cache-type-k / -v |
f16 (q8_0 on kvq8 variant rows) |
llama.cpp default except for kvq8 runs |
--flash-attn |
on (off on the fa-off variant row only) |
llama.cpp default except for fa-off runs |
--n_ctx (final) |
262144 per slot (native 256K) |
llama.cpp fitter |
--n_parallel |
4, kv_unified true |
llama.cpp (auto) |
--n_gpu_layers |
all (-1) |
llama.cpp fitter (llamastash knob was auto) |
--cache-type-k/v (final) |
f16 / f16 |
llama.cpp default |
--n_threads |
16 (of 32 logical) |
llama.cpp default |
--n_batch / --n_ubatch |
server defaults, not overridden | llama.cpp |
| spec internals (fork only) | n_min=0, p_min=0.75, backend_sampling=1 |
llama.cpp (q38rocm fork defaults) |
| draft context (fork only) | all GPU layers, f16 KV |
llama.cpp (fork) |
u/deepu105 — 1 day ago
▲ 9 r/StrixHalo
# Qwen3.8-27B heretic-ara: Custom GGUF quants with MTP + Vision + ROCmFPX — ~42 t/s decode on Strix Halo
I've been running local LLMs on a Ryzen AI Max+ 395 (122 GB unified memory, gfx1151) and wanted to share the quant collection I put together for the best model I've found for this hardware.
Repo: cygnal/Qwen3.8-27B-heretic-ara-Q4_K_M-MTP-GGUF
What's in the box
Five GGUF files, all with MTP (Multi-Token Prediction) tensors preserved (866 tensors) for speculative decoding:
| File | Size | BPW | Format | Decode Speed (MTP) |
|---|---|---|---|---|
| ROCmFP4_FAST | 14 GB | 4.26 | ROCmFPX | ~42 t/s |
| Q4_K_M-MTP | 16 GB | 4.83 | Stock llama.cpp | ~27–28 t/s |
| Q6_K-MTP | 21 GB | 6.56 | Stock llama.cpp | ~17–23 t/s |
| ROCmFP6 | 21 GB | 6.50 | ROCmFPX | ~24 t/s |
| mmproj (BF16) | 931 MB | — | Vision encoder | Shared by all |
The model: heretic-ara
Base model is trohrbaugh/Qwen3.8-27B-heretic-ara — an Arbitrary-Rank Ablation (ARA) uncensored version of Qwen3.8-27B using heretic v1.2.0. Zero refusals on the standard censorship test suite, while preserving model quality (KL divergence 0.0535 from base). It handles creative writing, security topics, and controversial prompts without the usual Qwen guardrails.
Qwen3.8 uses the qwen3.5 architecture — a hybrid DeltaNet + attention model (48 linear + 16 full-attention layers). Only 16 layers grow KV cache, so context scaling is very cheap. The built-in vision encoder handles multimodal image understanding natively.
ROCmFPX: The speed story
The headline number is ROCmFP4_FAST at ~42 t/s average decode — that's 50–75% faster than stock Vulkan Q4_K_M-MTP (27–28 t/s). This uses ROCmFPX fork of llama.cpp which adds optimized quantization formats for RDNA 3.5.
ROCmFP4_FAST benchmarks (Vulkan0, MTP n-max 6, TurboQuant KV)
| Task | Decode (t/s) | Prefill (t/s) |
|---|---|---|
| Code gen (3 runs) | 34.5 / 41.6 / 49.5 | 42–62 |
| Math reasoning | 37.5 | 97.4 |
| JSON extraction | 44.7 | 137.1 |
| Technical writing | 43.8 | 59.6 |
| Average | ~42 |
Critical flags for Strix Halo
llama-server \
-m Qwen3.8-27B-heretic-ara-ROCmFP4-FAST.gguf \
-ngl 99 -dev Vulkan0 \
-c 32768 \
-ctk q8_0 -ctv turbo4 \
-fa on \
--spec-type draft-mtp \
--spec-draft-n-max 6
-dev Vulkan0— This is the make-or-break flag. Forces the RADV cooperative matrix path. Without it, you get ~18 t/s instead of ~42. If you take one thing from this post, it's this flag.-ctk q8_0 -ctv turbo4— TurboQuant KV cache. Speed-neutral but dramatically reduces KV memory. Enables 262K context in 20 GB vs 61 GB with f16 KV.--spec-draft-n-max 6— Optimal for ROCmFPX on Strix Halo. Stock Vulkan prefers 3; higher values hurt on stock.
Stock Vulkan (no fork needed)
If you don't want to maintain a fork, Q4_K_M-MTP on stock llama.cpp with --spec-type draft-mtp --spec-draft-n-max 3 gets you 27–28 t/s. Still a solid experience. Q6_K-MTP gives higher quality at ~17–23 t/s.
llama-server \
-m Qwen3.8-27B-heretic-ara-Q4_K_M-MTP.gguf \
-ngl 99 -ctk f16 -ctv f16 \
-c 32768 \
--spec-type draft-mtp \
--spec-draft-n-max 3
Vision
All quants work with the included mmproj for image understanding. Load it with --mmproj mmproj-Qwen3.8-27B-heretic-ara-BF16.gguf and --jinja. Vision decode runs at ~13 t/s — the vision encoder adds overhead, but text-only requests are completely unaffected by having mmproj loaded. Safe to keep it on permanently.
One gotcha: on ROCmFPX (b213), MTP + vision crashes due to non-consecutive token positions from image embeddings. Omit --spec-type draft-mtp when using mmproj on ROCmFPX. Stock llama.cpp handles vision + MTP fine.
Speed comparison summary
| Config | Avg Decode (MTP) | File Size | Notes |
|---|---|---|---|
| ROCmFPX FP4_FAST | ~42 t/s | 14 GB | Fastest. Requires ROCmFPX fork. |
| Stock Q4_K_M-MTP | ~27–28 t/s | 16 GB | Best all-rounder. No fork. |
| ROCmFPX FP6 | ~24 t/s | 21 GB | Higher quality ROCmFPX. |
| Stock Q6_K-MTP | ~17–23 t/s | 21 GB | Higher quality stock. |
How it was built
- Downloaded heretic-ara safetensors from HuggingFace
- Converted to BF16 GGUF with
convert_hf_to_gguf.py— preserving all 866 tensors (851 model + 15 MTP) - Generated mmproj with
convert_hf_to_gguf.py --mmproj --outtype bf16(334 vision tensors, CLIP arch) - Stock quants:
llama-quantizefor Q4_K_M and Q6_K - ROCmFPX quants: ROCmFPX
llama-quantize(b213/b2f5829) for FP4_FAST and FP6
All benchmarks run on AMD Ryzen AI Max+ 395 (122 GB unified, gfx1151), stock Vulkan on llama.cpp b10254, ROCmFPX on b213.
My daily driver setup
I run this as a local AI agent backend (hermes-agent) on the Strix Halo. The server runs as a systemd user service on port 8084 with mmproj permanently loaded, MTP enabled, and 131K context. For my workflow (multi-turn chat, tool calls, image understanding), the Q6_K-MTP on stock Vulkan hits the sweet spot of quality and speed. If I need pure text speed, the ROCmFP4_FAST is there at 42 t/s.
Happy to answer questions about the setup or Strix Halo-specific tweaks.
Edit: updated link to ROCmFPX
u/cygnal — 1 day ago
▲ 17 r/StrixHalo
Am I missing something? Qwen3.8 is very slow on my Strix Halo, while Qwen3.6 27b MTP Q4 can reach 20 tokens even on big contexts, but the 3.8 Q4 is 5 tokens.
reddit.comu/Teslaaforever — 2 days ago
▲ 16 r/StrixHalo
TheNoise - Image generation engine optimized for Strix Halo
Hello everyone,
I want to introduce you to TheNoise, an open source image generation engine made specifically to run well on Strix Halo.
Let me first say: this isn't better than ComfyUI, but definitely easier to install and use if you just want to get started with image generation on your Strix Halo without having to care about "workflows" and the likes.
Also, generation is fast - faster than a "default" ComfyUI installation.
It can be used standalone, from the command line or through a webui or through an OpenAI-compatible server like Lemonade, with which it is already integrated (full disclousure, I am also a Lemonade maintainer).
Supported models:
- Krea 2
- Z-Image / Z-Image-Turbo
- Anima (all variants, including the 2.9B version)
- Flux 2 Klein 4B / 9B
It also has some goodies like:
- Refiner-based upscaler (up to 2x) - this gives high quality upscaling in relatively short time
- Pixel-space upscalers (up to 4x) - these are your standard ESRGAN-based upscalers
- Film grain, sharpening
- LoRA support
Some numbers, at 1024x786 resolution generation on a Strix Halo are:
- Krea 2 Turbo: ~27 seconds
- Z-Image-Turbo (8 steps): ~10 seconds
- Anima (20 steps): ~22 seconds
- Anima Turbo (8 steps): ~5 seconds
- Flux Klein 9B (4 steps): ~9 seconds
- Flux Klein 4B (4 steps): ~4 seconds
You can get the software either directly in Lemonade or by download it from https://github.com/lemonade-sdk/thenoise and following the instructions. There are also precompiled bins that come with python and ROCm.
At the moment it has only been tested on Linux. Known to work on Strix Halo, Strix Point and Krackan Point.
u/mikkoph — 1 day ago
▲ 20 r/StrixHalo
Extra GPU?
I am wondering if I should buy an extra GPU for models like Qwen 3.8 27B, e.g. an R9700 or intel arc one. What are the pros and cons vs a second strix halo (I am on Bosgame M5) vs just running one solo?
Context: Currently I mostly use Qwen 3.5 122B Q4 Unsloth which isn’t bad at all but I still have to use, via cloud, DS4F or Luna a lot currently since those are stronger (won’t fail the task at hand, mostly C++) and are much much faster.
u/Reasonable_Goat — 3 days ago
▲ 8 r/StrixHalo
tg isnt everything
I’ve tested a number of ROCm-FPX models, including Qwen, DeepSeek, and many others. Some of them benchmark surprisingly well, reaching 30+ tok/s.
However, once I put them into real production workloads, I often find that they take significantly longer to complete the same task. A model may generate tokens quickly, but if it requires more reasoning steps, produces mistakes, or needs multiple attempts to reach the correct result, that raw token speed means very little.
So, to me, obsessing over quantization benchmarks and tok/s is often just a comfort drug—the numbers make you feel good, but what really matters is time-to-solution: how long it takes the model to actually finish the job correctly.
u/hycrice — 1 day ago
DeepSeek V4 Flash 0731 on Strix Halo: draft model, n_max sweep, and a launch line that actually helps
I know the front page is completely dominated by the Qwen3.8 27B. I know it's a fantastic model and everyone is rightfully discussing it, but I wanted to shift gears and talk about another model I’ve been experimenting with and got some solid results from: DeepSeek-V4-Flash-0731.
I just finished a week of running numbers on a Strix Halo box and figured I'd dump them here before I forget what I actually measured.
Hardware is a Strix Halo mini PC, Ubuntu, 128 GB unified memory.
Target is Unsloth's UD-IQ3_XXS with Q6 attention (~96 GB). At 264k context, no context quantization, the process sits at about 114 GB.
At 64k I see 200–220 tok/s prefill and 20–22 tok/s decode with no draft. That's the baseline I used for everything below: 20.48 tok/s.
I compared two DSpark drafters, Q8_0 (10.15 GB) and a Q2_K_S I quantized myself (6.45 GB), over n_max 2–7, 7 prompt types, 5 runs each.
Same sampling the whole way: temp 0.9, top_p 0.95, min_p 0.01. Thinking on (reasoning_effort=low for the sweep so it wouldn't take a week). ngram-mod was off during the sweep so draft_n is only the drafter.
Short version: n_max=3 is the peak. Average over both drafts is 28.5 tok/s, 1.39× the no-draft baseline. Q2 and Q8 are a wash. Every n_max, every category, they land inside 1–3% of each other. So the 6.45 GB file is the one I'd actually keep.
One caveat on that 28.5 average: the suite is stacked with tasks that speculative decoding loves (repeat, math, code), and the prompts only used up to ~32k of context. Don't treat it as what you'll see on a mixed chat/coding day with a fuller window. Realistically I'd budget 22–28 tok/s.
Best n_max per category (avg vs 20.48 tok/s):
- code — n_max 3 — 28.99 t/s — 1.42×
- json — n_max 3 — 28.59 — 1.40×
- math — n_max 3 — 31.22 — 1.52×
- dialog — n_max 3 — 25.48 — 1.24×
- translate — n_max 2 — 25.09 — 1.22×
- prose — n_max 2 — 23.38 — 1.14×
- repeat — n_max 5 — 40.17 — 1.96×
n_max 5–7 only pays off on the high-acceptance stuff (repeat, some math). On prose / translate / dialog the extra draft tokens just get rejected and you go backwards. Overall table if you want it:
n_max 2: 27.12 t/s, 1.32×, accept 0.69 n_max 3: 28.50, 1.39×, 0.60 n_max 4: 27.73, 1.35×, 0.52 n_max 5–7: ~26.4–26.5, 1.29×, accept ~0.45
I run this day to day at 128k with ngram-mod stacked on the drafter and thinking set to max. That's not what the sweep used (sweep was a 64k window, ~32k of it actually filled, draft-only, thinking=low), so don't treat the 28.5 as a promise for this exact line. The n_max=3 result is what I kept; 22–28 tok/s is what I'd quote for mixed use.
llama-server \
-m DeepSeek-V4-Flash-0731-UD-IQ3_XXS-q6kattn.gguf \
-a DeepSeek-V4-Flash-0731-UD-IQ3_XXS \
--no-ui -ngl 999 -c 131072 --jinja -fa 1 --port 9989 \
--no-mmap --no-warmup -np 1 \
--temp 0.9 --top-p 0.95 --min-p 0.01 --host 0.0.0.0 \
-ngld 999 -fit off \
-md DeepSeek-V4-Flash-0731-DSpark-Drafter-Q2_K_S-dflash.gguf \
--spec-type ngram-mod,draft-dspark --spec-draft-n-max 3 \
-ub 1024 --cache-ram 2048 \
--chat-template-kwargs '{"reasoning_effort":"max"}'
Fork llama.cpp is strix-halo-llamacpp (FA + MoE-prefill fixes, bundled Mesa, Vulkan/HIP).
GGUF I actually load: DeepSeek-V4-Flash-0731-StrixHalo-Verified-GGUF.
Drafter: Lynxpda/DeepSeek-V4-Flash-0731-DSpark-Drafter-Q2_K_S-GGUF.
Charts from the sweep (baseline drawn at 20.48):
If you're on Strix Halo and already sitting on Flash 0731, --spec-draft-n-max 3 and the Q2 drafter is what I'd start with. Happy to answer questions about the sweep if something looks off.
upd: corrected errors.
u/Responsible_Pain3278 — 3 days ago
Qwen3.8 Q4_k_m 1M context Strix Halo / 3080 -> 45 tps
Been playing qwen3.8 for a couple of days, on a AMD 128GB strix machine, with oculink eGPU (3080ti) 12GB.
Running Q6, 262K context
on just strix halo without MTP 10tps, with MTP with n=4 24 tps, with FastMTP offloaded to 3080 with n=4 about 28 tps
Q4_k_m 262K context
Splitting layers 5GB weights & FastMPT on 3080 and rest on iGPU~ 53 tpsQ4_k_m, 1M context, k-q8, v-q4
Splitting layers 2GB weights & FastMPT on 3080 and rest on iGPU~ 45 tps
Still isn’t as good as a 5090 but AMD still show 60-70GB free so I can still load qwen3 embedding and a qwen3 Reranker to go with everything
Now need to see how good this model really is compared to qwen3.6-35b that I have running on 4x3090 and runs at 450 tps….
u/TrifleHopeful5418 — 3 days ago
llama.cpp PR#26856: Faster Prefill, Better Quality for ROCm on RDNA3+
tl;dr: This is about this llama.cpp PR here: https://github.com/ggml-org/llama.cpp/pull/26856
Over the weekend I decided to dig into why, when specifying the BF16 type for KV cache on my Strix Halo box, that token generation performance tanked by ~20% over using F16.
Qwen3.6-35B-A3B-Q8_0 @ 16K CTX, no MTP
| KV type | tg64 (t/s) |
|---|---|
| F16 | 43.09 |
| BF16 | 34.90 |
| Q8_0 | 40.83 |
| F32 | 29.54 |
What was worse was that the output quality didn't seem to improve either. Perplexity scores for F16 vs BF16 were the same, and both are about 5% above the F32 baseline. I dug into the architectural specs for RDNA 3, 3.5, and 4, and it seems that all 7xxx, 9xxx series GPUs, and all Strix Point/Strix Halo iGPUs have native support for BF16 dot product matrix math. The scalar ALU portions seems to lack complete native BF16 support, but that's not really needed for Flash Attention.
So, with the assistance of DeepSeek V4 Flash 0731, I started to dig into the situation and see if it was possible to make BF16 KV caches go faster. It didn't take long to learn that llama.cpp doesn't really implement native BF16 support beyond simply accepting the data type, and then converting everything to F16 anyway. It was this BF16->F16 conversion that was killing the generation performance on RDNA GPUs and the Strix Halo.
So I worked to implement native BF16 support for KV Flash Attention, and it turned out it was possible to do so. In fact, through bypassing the old BF16->FP16 path I was able to see achieve between a 10-15% pre-fill speedup on the Strix Halo. The gain is smaller on the 7900XTX and R9700 GPUs (more like 5-10%), but it was real.
Then I set about tuning the matrix kernels, and was able to achieve token generation parity with the original F16 path, which is the fastest token generation path available on the Strix Halo.
Now, since BF16 has a much larger dynamic range than F16 I then thought to compare the Perplexity results when using the native BF16 kernel path, and was surprised to find that, at depth, the BF16 KV cache path almost exactly matches the baseline F32 KV cache. Incidentally, the F32 KV cache path also uses F16 math under the hood if Flash Attention is enabled, so I had to disable Flash Attention to get at the true F32 Perplexity baseline figures.
Precision at depth (PPL @ 32k, Qwen3.5-4B-Q8_0, full wikitext-2 test, 9 chunks)
| KV cache / FA | PPL @ 32k |
|---|---|
| F32 baseline (FA off) | 8.6368 |
| BF16 (this change) | 8.6403 (+0.04%) |
| F16 | 9.1400 (+5.8%) |
What does that mean? It means that for long context work that the models are less likely to go astray and/or suffer from context rot.
Note: For Strix Halo boxes that until such time that the upstream ROCm libraries get fixed, that you must disable mmap, as well as set HIP_LAUNCH_BLOCKING=1 to avoid running into a pair of rather nasty bugs that will heavily corrupt your KV cache. See my post here for more details: https://www.reddit.com/r/StrixHalo/s/Qbh3RDVsXg
What About Vulkan?
Now you may ask: "What about Vulkan?". The answer is that the Vulkan path in llama.cpp already implements this BF16 behavior properly, so this PR just brings ROCm up to Perplexity parity with Vulkan. The part from this PR that can't be done on Vulkan though is the RDNA3+ specific dot-product optimizations, so this PR makes ROCm pre-fill significantly faster than Vulkan.
Here's Vulkan vs ROCm results with Qwen3.6-35B at 32K context depth:
F16 K/V Performance
| model | size | params | backend | ngl | fa | test | t/s |
|---|---|---|---|---|---|---|---|
| qwen35moe 35B.A3B Q8_0 | 35.19 GiB | 35.51 B | ROCm | -1 | 1 | pp1024 @ d32768 | 618.31 ± 4.83 |
| qwen35moe 35B.A3B Q8_0 | 35.19 GiB | 35.51 B | ROCm | -1 | 1 | tg256 @ d32768 | 42.42 ± 0.29 |
| qwen35moe 35B.A3B Q8_0 | 35.19 GiB | 35.51 B | Vulkan | -1 | 1 | pp1024 @ d32768 | 717.93 ± 3.69 |
| qwen35moe 35B.A3B Q8_0 | 35.19 GiB | 35.51 B | Vulkan | -1 | 1 | tg256 @ d32768 | 46.86 ± 0.01 |
BF16 K/V Performance
| model | size | params | backend | ngl | type_k | type_v | fa | test | t/s |
|---|---|---|---|---|---|---|---|---|---|
| qwen35moe 35B.A3B Q8_0 | 35.19 GiB | 35.51 B | ROCm | -1 | bf16 | bf16 | 1 | pp1024 @ d32768 | 684.02 ± 8.36 |
| qwen35moe 35B.A3B Q8_0 | 35.19 GiB | 35.51 B | ROCm | -1 | bf16 | bf16 | 1 | tg256 @ d32768 | 42.51 ± 0.06 |
| qwen35moe 35B.A3B Q8_0 | 35.19 GiB | 35.51 B | Vulkan | -1 | bf16 | bf16 | 1 | pp1024 @ d32768 | 484.00 ± 3.83 |
| qwen35moe 35B.A3B Q8_0 | 35.19 GiB | 35.51 B | Vulkan | -1 | bf16 | bf16 | 1 | tg256 @ d32768 | 47.59 ± 0.01 |
Interesting Results here:
- For F16 Vulkan handily beats ROCm for both pre-fill and decode, but keep in mind that this is coming at the cost of long context quality (for both).
- For BF16 ROCm handily beats Vulkan by 40% for pre-fill, but Vulkan beats ROCm by ~11% for generation. If using the Strix Halo for agentic coding, then you're clearly going to want ROCm here for speed and quality at long contexts.
Edit: 11th Aug - Faster ROCm generation
Maybe I'll make it my next job to see if I can make ROCm close that generation performance gap with Vulkan. I have a feeling that it must be possible.
I have completed the ROCm generation speed-up work to as far as I can take it. At the end of the day the Vulkan dispatcher is just more efficient than ROCm's, and I'm now up against that fundamental library difference. Still, I've managed to more than halve the generation performance gap from ROCm to Vulkan, while even boosting ROCm's prefill advantage. The source-code branch for this follow work is here: https://github.com/stew675/llama.cpp/tree/make-rocm-gen-faster
It's unlikely that the follow-up branch would ever be accepted into upstream llama.cpp, so I'll try to make daily upstream rebases to keep that work current for people who are interested.
Edit: 12th Aug - RDNA4 boosts
I've created a consolidated branch that includes all prior work plus a host of RDNA4 speed boosts and put it into this branch here: https://github.com/stew675/llama.cpp/tree/rdna-boosts
This branch boosts ROCm prefill speeds on RDNA4 GPUs by a further 5-25% over the prefill speed boosts mentioned above. This means a total of +10-40% prefill speed ups on RDNA4, and +5-15% prefill speedups on RDNA3/3.5, Sadly the RDNA3/3.5 architectures simply don't have the CU hardware to support the additional prefill speeds that RDNA4 can achieve.
The rdna-boosts branch also includes the 5-10% ROCm generation speed boosts, which applies to all RDNA3+ GPUs. It also fixes the async-race memory corruption issues on Strix Halo that is still present in upstream llama.cpp. Note that mmap loading is still broken on Strix Halo at the moment.
u/Look_0ver_There — 3 days ago
▲ 26 r/StrixHalo
Z13 Clusters pretty well with dual USB-4
Little bit of thermal throttling compared to other Strix boxes like Framework, but I do like that I could always take this setup on the go or repurpose these into individual dev machines if needed.
u/Jawnnypoo — 3 days ago
Qwen3.8-27B Q8 MTP benchmarks on Strix Halo — MTP is actually making it slower. Are others seeing the same?
I've been testing Qwen3.8-27B Q8 on a Ryzen AI Max+ 395 / Strix Halo system through Lemonade + llama.cpp, specifically to see whether the new MTP speculative decoding support actually improves generation speed.
I kept the prompt and output length identical between runs:
- 96 input tokens
- 1024 output tokens
- Temperature 0
- Same Qwen3.8-27B model/quant
- Flash Attention enabled
--no-mmap- Only backend/MTP settings changed
These are the results so far:
| Backend | MTP setting | Generation speed | TTFT |
|---|---|---|---|
| Vulkan | Off | 9.159 tok/s | 0.758 s |
| Vulkan | n-max=1 |
6.579 tok/s | 0.950 s |
| Vulkan | n-max=3 |
7.122 tok/s | 0.764 s |
| ROCm | Off | 6.534 tok/s | 0.715 s |
| ROCm | n-max=3 |
4.689 tok/s | 0.688 s |
So on my machine:
- Vulkan + MTP n=3 is about 22% slower than Vulkan without MTP.
- Vulkan + MTP n=1 is about 28% slower.
- ROCm itself is about 29% slower than Vulkan without MTP.
- ROCm + MTP n=3 drops another ~28% versus ROCm without MTP.
- Overall, Vulkan without MTP is almost 2x the generation throughput of ROCm + MTP in this test.
For MTP I'm loading llama.cpp with:
--spec-type draft-mtp --spec-draft-n-max 3
(and also tested n-max=1 on Vulkan).
For ROCm, I'm using Lemonade's current stable ROCm backend. Lemonade reports the llama.cpp backend as b10397; the bundled ROCm/TheRock stack appears to be ROCm 7.13.x. I haven't tested ROCm 7.14 yet.
It's surprising to see that with MTP there's a pretty substantial regression on both Vulkan and ROCm.
I'd be interested to compare with other Strix Halo owners:
- Are you seeing MTP actually improve Qwen3.8 throughput?
- What
--spec-draft-n-maxvalue works best for you? - Are you using Vulkan or ROCm?
- Which ROCm version / llama.cpp build?
- Does ROCm 7.14 materially improve Strix Halo performance versus 7.13?
- What Qwen3.8-27B quant are you using?
- If you're getting a significant MTP speedup, what kind of draft acceptance rate are you seeing?
I'm mainly trying to figure out whether these numbers are normal for the current llama.cpp MTP implementation on Strix Halo, or whether something is wrong with my setup.
At least with my current stack, Vulkan with MTP disabled is very clearly the fastest configuration I've tested.
u/SecuredStealth — 3 days ago
I post-trained Qwen3.6-35B-A3B into my daily-driver local coding/agent model QwiVer3.6-35B-A3B GGUF
Edit: (Rewriting this because my first post sounded way too much like a model card lol.)
I've been working on a post-trained version of Qwen3.6-35B-A3B that I mainly use for coding and agent workflows locally. I called it QwiVer3.6-35B-A3B.
Model is here:
https://huggingface.co/BlackRiverAI/QwiVer3.6-35B-A3B-GGUF
This isn't just Qwen with a different name slapped on the GGUF.
I trained a LoRA against the BF16 base, selected the Phase 10.2 adapter, merged it back into the frozen BF16 model, then converted/quantized the merged model to GGUF and checked that the Qwen3.6 architecture + MTP stuff survived the process correctly.
The final training run was:
1,531 examples 3,999,924 training tokens 383 optimizer steps
Base was unsloth/Qwen3.6-35B-A3B.
Architecture is still the original Qwen3.6-35B-A3B setup:
~35B total parameters ~3B active 40 main layers 256 experts 8 routed + 1 shared expert active 262,144 native context vision native MTP
What I was trying to change was mostly behavior rather than architecture.
I wanted it to be more useful when I'm actually sitting in front of a coding agent for hours: following an existing codebase, debugging instead of immediately rewriting everything, using tools properly, staying on task across longer jobs, and generally being less annoying to work with.
That's also why I've been calling it my daily driver rather than trying to claim I invented a new base model.
On my own workloads I prefer it pretty clearly over stock Qwen3.6-35B-A3B. That's subjective, obviously, which is also why I'm running proper benchmarks now instead of asking anyone here to take my word for it.
First ones running are AIME 2026 and GPQA Diamond against the official Qwen3.6 published scores. I'm using the public Q8 GGUF through llama.cpp, saving the raw per-question results/config, and I'll publish those when the runs are finished.
The little smoke test before starting the full run was 2/2 AIME and 2/2 GPQA, but that's four questions so it means absolutely nothing beyond confirming that the harness works 😅
GGUFs currently uploaded:
Q2 ~12.57 GB Q3 ~17.23 GB Q4 ~22.85 GB Q8 ~39.10 GB
Q4 is probably the sensible download for most people. Q8 is what I'm using for the benchmark run.
There's also the BF16 vision projector and the chat template in the repo.
I'm running it on a Ryzen AI Max+ PRO 395 / Radeon 8060S with 128 GB unified memory, Windows, llama.cpp Vulkan.
For normal use I also run the native MTP head, although I've disabled speculative decoding for the quality benchmark so it isn't another variable in the results.
If anyone actually tries the model, I'm especially interested in coding/agent failures compared with stock Qwen3.6. That's more useful to me than "feels smarter" either way.
And yes, benchmarks are coming. I got the message.
u/RIP26770 — 4 days ago
If you would have told me half a year ago that a local model running in my office would be able to one-shot a Super Mario clone, I would have called you nuts. Qwen3.8-27B is a different beast.
Running the Q8 GGUF on my Framework Desktop is not fast, but it's extremely smart for overnight batches and background jobs. Can't wait to play around with MTP and other quants.
Have any of you found ways to improve speed while keeping accuracy?
u/MikeNonect — 6 days ago
▲ 80 r/StrixHalo
Qwen 3.8 27B running at up to 36tps on the Halo
Opus 4.6 level intelligence at home for les than $3k
Got Qwen 3.8 27B running at up to 36 tps on the AMD Strix Halo
⚡ ROCmFP4 block quant (13.5 GB)
⚡ MTP Speculative Decoding (2.9× speedup)
⚡ Full 262K context in ~33 GB RAM
u/MrWidmoreHK — 5 days ago
Qwen 3.8 27B on Strix Halo - some Results
Weights dropped today, Unsloth GGUFs followed right after. Here are numbers from a 128GB Strix Halo box. Vulkan/RADV, flash attention on, batch 2048/512, depths 0 and 32k, two runs per cell, drift check at the end was under 0.8 percent.
It loads on existing llama.cpp builds. Architecture string is the same as Qwen 3.6 (qwen35), 65 layers, 4 KV heads at dim 256, 262k native context. No update needed.
Weight quants, f16 KV cache:
| Weights | Prefill d0 (t/s) | Gen d0 (t/s) | Prefill 32k (t/s) | Gen 32k (t/s) |
|---|---|---|---|---|
| UD-Q5_K_XL (20 GB) | 347.0 | 10.5 | 252.4 | 9.4 |
| Q8_0 (29 GB) | 340.3 | 7.5 | 249.5 | 6.9 |
Compared to Qwen 3.6 27B in the same Q5 quant on the same build: within 2.5 percent everywhere. So switching costs nothing in throughput.
The Q8 row is there because someone asked for it under my last post. Q8 generates 29 percent slower than Q5 at depth 0 and 27 percent at 32k. That matches the file sizes: 29 vs 20 GB is a factor of 1.44, the decode gap is 1.41. Dense model, reads all weights every token, so generation scales with file size.
KV cache quantization, measured at 32k (at depth 0 all three KV types are identical to within half a percent):
| Weights | KV cache | Prefill 32k (t/s) | Gen 32k (t/s) |
|---|---|---|---|
| UD-Q5_K_XL | f16 | 252.4 | 9.4 |
| UD-Q5_K_XL | q8_0 | 250.9 | 9.9 |
| UD-Q5_K_XL | q4_0 | 255.2 | 10.0 |
| Q8_0 | f16 | 249.5 | 6.9 |
| Q8_0 | q8_0 | 247.0 | 7.1 |
| Q8_0 | q4_0 | 243.9 | 7.2 |
Same pattern as on 3.6: prefill within 2 percent, generation gains 4 to 6 percent with quantized cache. Matches what I measured across nine models recently.
Both GGUFs have the MTP tensors, I haven't tested speculative decoding yet. Caveats: community fork build (10283, strix-halo-vulkan branch), throughput only. No idea yet whether 3.8 is actually better than 3.6 at anything.
If you have numbers on other hardware, especially the Q8 vs Q5 gap on something with more bandwidth, post them with your build commit.
u/javaeeeee — 5 days ago
▲ 0 r/StrixHalo
I'm surprised no-one seems to be talking about medusa halo mini
It seems like most of the discussion with medusa is about medusa point and medusa halo but there's not much about medusa halo mini.
It's rumoured to have 14 cores(4 zen 6 + 8 zen 6c + 2 zen 6 LP), 24CU rDNA5 with 10MB of L2 cache and a 128 bit lpddr5X, along with using the standard FP10 socket, same as medusa point is rumored to use.
This solves the main problem with strix point and to some extent strix halo. Strix point was trying to be a great CPU and IGPU at once and didn't do well at either. In the Asus zephyrus G16, you already have a high end Nvidia graphics card so don't need a big iGPU and in something like the zenbook S16, it had more CPU than needed and was a huge 232mm2 die, both problems lead to a lot of laptops using kraken point like the Ryzen AI 350.
Strix halo was supposed to be a great gaming APU, 40CU graphics and 256 bit lpddr5X but the die size was huge and too AI focused, only available in 2 laptops at first and now only in around 5 or 6 total, with most being extremely expensive.
Now medusa point with the full 22 cores that has been rumoured seems to be filling the efficient and powerful CPU slot, to work great with a high end dGPU like a 5080.
That leaves medusa halo mini to do what strix point and strix halo struggled to do, be a great APU for gaming without a dGPU, while being more available and affordable than strix halo.
I don't know how everyone else is feeling but this is tech I'm most excited for in 2027, I'm crossing my fingers that we'll finally have an APU that's readily available unlike strix halo and pantherlake, not insanely expensive and has a powerful IGPU for gaming.
Finally doing some scratch math with navi 44 (Rx 9060 XT) and the Ryzen 9700X, I predict that we will get around a 135mm2 gpu die and a 80-95mm2 CPU die, both on 3nm.
Now of course my numbers could be off but should be accurate enough to show that it could be affordable if AMD wants it to be, but we'll just have to wait and see.
Anyway I'd just thought I'd put this out there to see how everyone else is feeling about medusa halo mini as it doesn't seem to be being discussed anywhere.
u/Quiet_Honeydew_6760 — 5 days ago
▲ 0 r/StrixHalo
Absurdly high tps when running Agentic tasks in OpenCode with Qwen 3.8 27B, can you explain why it shows > 100 tps in certain regions of the execution?
12.21.695.447 I slot print_timing: id 0 | task 14785 | total time = 15239.13 ms / 340 tokens
112.21.695.449 I slot print_timing: id 0 | task 14785 | graphs reused = 14490
112.21.695.460 I slot print_timing: id 0 | task 14785 | draft acceptance = 0.56019 ( 121 accepted / 216 generated), mean len = 3.24
112.21.696.421 I slot release: id 0 | task 14785 | stop processing: n_tokens = 10835, truncated = 0
112.21.811.710 I slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, f_sim_best = 0.363 (> 0.100 thold), f_keep = 1.000
112.21.812.914 I slot launch_slot_: id 0 | task 14842 | processing task, is_child = 0
112.28.848.199 I slot print_timing: id 0 | task 14842 | prompt processing, n_tokens = 1024, progress = 0.40, t = 3.45 s / 297.16 tokens per second
112.35.945.624 I slot print_timing: id 0 | task 14842 | prompt processing, n_tokens = 2048, progress = 0.43, t = 10.60 s / 193.17 tokens per second
112.42.998.468 I slot print_timing: id 0 | task 14842 | prompt processing, n_tokens = 3072, progress = 0.47, t = 17.62 s / 174.31 tokens per second
112.50.641.489 I slot print_timing: id 0 | task 14842 | prompt processing, n_tokens = 4096, progress = 0.50, t = 24.76 s / 165.45 tokens per second
112.58.601.250 I slot print_timing: id 0 | task 14842 | prompt processing, n_tokens = 5120, progress = 0.53, t = 32.80 s / 156.08 tokens per second
113.06.213.403 I slot print_timing: id 0 | task 14842 | prompt processing, n_tokens = 6144, progress = 0.57, t = 40.60 s / 151.32 tokens per second
113.13.499.971 I slot print_timing: id 0 | task 14842 | prompt processing, n_tokens = 7168, progress = 0.60, t = 48.26 s / 148.53 tokens per second
113.18.864.030 I slot print_timing: id 0 | task 14842 | prompt processing, n_tokens = 8192, progress = 0.64, t = 54.45 s / 150.45 tokens per second
113.24.080.317 I slot print_timing: id 0 | task 14842 | prompt processing, n_tokens = 9216, progress = 0.67, t = 59.67 s / 154.45 tokens per second
113.29.365.192 I slot print_timing: id 0 | task 14842 | prompt processing, n_tokens = 10240, progress = 0.71, t = 64.92 s / 157.74 tokens per second
113.34.682.023 I slot print_timing: id 0 | task 14842 | prompt processing, n_tokens = 11264, progress = 0.74, t = 70.22 s / 160.41 tokens per second
113.40.060.176 I slot print_timing: id 0 | task 14842 | prompt processing, n_tokens = 12288, progress = 0.77, t = 75.55 s / 162.64 tokens per second
113.45.570.016 I slot print_timing: id 0 | task 14842 | prompt processing, n_tokens = 13312, progress = 0.81, t = 81.01 s / 164.32 tokens per second
113.51.128.917 I slot print_timing: id 0 | task 14842 | prompt processing, n_tokens = 14336, progress = 0.84, t = 86.52 s / 165.70 tokens per second
113.56.745.798 I slot print_timing: id 0 | task 14842 | prompt processing, n_tokens = 15360, progress = 0.88, t = 92.10 s / 166.77 tokens per second
114.02.663.952 I slot print_timing: id 0 | task 14842 | prompt processing, n_tokens = 16384, progress = 0.91, t = 97.77 s / 167.57 tokens per second
114.08.946.924 I slot print_timing: id 0 | task 14842 | prompt processing, n_tokens = 17408, progress = 0.95, t = 103.97 s / 167.43 tokens per second
114.15.120.433 I slot print_timing: id 0 | task 14842 | prompt processing, n_tokens = 18432, progress = 0.98, t = 110.29 s / 167.12 tokens per second
114.15.917.316 I slot print_timing: id 0 | task 14842 | prompt processing, n_tokens = 18504, progress = 0.98, t = 113.47 s / 163.08 tokens per second
114.18.918.214 I slot print_timing: id 0 | task 14842 | prompt processing, n_tokens = 19016, progress = 1.00, t = 114.85 s / 165.57 tokens per second
114.22.651.796 I slot print_timing: id 0 | task 14842 | prompt eval time = 117455.14 ms / 19020 tokens ( 6.18 ms per token, 161.93 tokens per second)
114.22.651.803 I slot print_timing: id 0 | task 14842 | eval time = 3383.27 ms / 87 tokens ( 39.34 ms per token, 25.42 tokens per second)
114.22.651.805 I slot print_timing: id 0 | task 14842 | total time = 120838.41 ms / 19107 tokens
114.22.651.807 I slot print_timing: id 0 | task 14842 | graphs reused = 14507
114.22.651.815 I slot print_timing: id 0 | task 14842 | draft acceptance = 0.94444 ( 68 accepted / 72 generated), mean len = 4.78
114.22.653.541 I slot release: id 0 | task 14842 | stop processing: n_tokens = 29941, truncated = 0
114.22.765.963 I slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, f_sim_best = 0.979 (> 0.100 thold), f_keep = 1.000
114.22.766.823 I slot launch_slot_: id 0 | task 14882 | processing task, is_child = 0
114.33.459.459 I slot print_timing: id 0 | task 14882 | n_gen = 103, tg = 16.58 t/s, tg_3s = 16.74 t/s
114.36.465.999 I slot print_timing: id 0 | task 14882 | n_gen = 164, tg = 17.79 t/s, tg_3s = 20.29 t/s
114.39.467.120 I slot print_timing: id 0 | task 14882 | n_gen = 209, tg = 17.10 t/s, tg_3s = 14.99 t/s
114.42.467.394 I slot print_timing: id 0 | task 14882 | n_gen = 256, tg = 16.82 t/s, tg_3s = 15.67 t/s
114.45.467.725 I slot print_timing: id 0 | task 14882 | n_gen = 295, tg = 16.19 t/s, tg_3s = 13.00 t/s
114.48.648.157 I slot print_timing: id 0 | task 14882 | n_gen = 344, tg = 16.07 t/s, tg_3s = 15.41 t/s
114.51.659.619 I slot print_timing: id 0 | task 14882 | n_gen = 389, tg = 15.93 t/s, tg_3s = 14.94 t/s
114.54.660.324 I slot print_timing: id 0 | task 14882 | n_gen = 425, tg = 15.50 t/s, tg_3s = 12.00 t/s
114.57.669.807 I slot print_timing: id 0 | task 14882 | n_gen = 488, tg = 16.04 t/s, tg_3s = 20.93 t/s
115.00.682.805 I slot print_timing: id 0 | task 14882 | n_gen = 541, tg = 16.18 t/s, tg_3s = 17.59 t/s
115.03.872.347 I slot print_timing: id 0 | task 14882 | n_gen = 590, tg = 16.11 t/s, tg_3s = 15.36 t/s
115.06.889.953 I slot print_timing: id 0 | task 14882 | n_gen = 643, tg = 16.22 t/s, tg_3s = 17.56 t/s
115.09.905.609 I slot print_timing: id 0 | task 14882 | n_gen = 698, tg = 16.36 t/s, tg_3s = 18.24 t/s
115.12.932.183 I slot print_timing: id 0 | task 14882 | n_gen = 756, tg = 16.55 t/s, tg_3s = 19.16 t/s
115.15.957.787 I slot print_timing: id 0 | task 14882 | n_gen = 830, tg = 17.04 t/s, tg_3s = 24.46 t/s
115.19.007.104 I slot print_timing: id 0 | task 14882 | n_gen = 895, tg = 17.29 t/s, tg_3s = 21.32 t/s
115.22.028.343 I slot print_timing: id 0 | task 14882 | n_gen = 949, tg = 17.32 t/s, tg_3s = 17.87 t/s
115.25.061.394 I slot print_timing: id 0 | task 14882 | n_gen = 1014, tg = 17.54 t/s, tg_3s = 21.43 t/s
115.28.076.420 I slot print_timing: id 0 | task 14882 | n_gen = 1065, tg = 17.51 t/s, tg_3s = 16.92 t/s
115.31.119.722 I slot print_timing: id 0 | task 14882 | n_gen = 1127, tg = 17.65 t/s, tg_3s = 20.37 t/s
115.34.171.556 I slot print_timing: id 0 | task 14882 | n_gen = 1189, tg = 17.77 t/s, tg_3s = 20.32 t/s
u/AllSignalNoNoise — 4 days ago
Qwen 3.8 27B at 30 tok/s in decode, running on a Strix Halo with 64 GB of unified memory!
30 tok/s on Qwen 3.8 27B with a Strix Halo and 64 GB of unified memory.
And this was achieved using only MTP with Draft 7, running entirely through GGUF. No DS Flash, no DS Park, no other acceleration tricks.
The model is running with 64K context, and even beyond 64K context, with Q8 for the KV cache.
All of this with just a little over 200 GB/s of memory bandwidth, which is honestly very little for this kind of workload.
It took some work to get there, but 30 tok/s on a 27B model with this setup is pretty damn impressive.
u/Pitiful_Fennel8767 — 6 days ago
I did something crazy: connected 2x R9700 to Framework Desktop
I used the pcie 4x and one of the M.2 slots with respective raisers and external PSU.
Idea was to check: how far can I bring this platform, the 128GB vram is great but iGPU is kinda slow and bigger models struggle on it.
Also DSV4 flash sounds really nice but runs quite poorly on Strix Halo alone.
Here are some models I ran, as my workflows are mainly 'low concurrency' the tests were aimed at measuring single stream inference, not concurrent inference.
Please let me know if you would do anything different / try any other interesting model!
*Edit(i) The UD Q8 results of DS4 Flash (Q8_K_XL) are also computed now and its very close to Q4: pf 236 / tg 19.5
u/Sharp-Translator6401 — 6 days ago