I've been running local LLMs on a Ryzen AI Max+ 395 (122 GB unified memory, gfx1151) and wanted to share the quant collection I put together for the best model I've found for this hardware.

Repo: cygnal/Qwen3.8-27B-heretic-ara-Q4_K_M-MTP-GGUF

What's in the box

Five GGUF files, all with MTP (Multi-Token Prediction) tensors preserved (866 tensors) for speculative decoding:

File	Size	BPW	Format	Decode Speed (MTP)
ROCmFP4_FAST	14 GB	4.26	ROCmFPX	~42 t/s
Q4_K_M-MTP	16 GB	4.83	Stock llama.cpp	~27–28 t/s
Q6_K-MTP	21 GB	6.56	Stock llama.cpp	~17–23 t/s
ROCmFP6	21 GB	6.50	ROCmFPX	~24 t/s
mmproj (BF16)	931 MB	—	Vision encoder	Shared by all

The model: heretic-ara

Base model is trohrbaugh/Qwen3.8-27B-heretic-ara — an Arbitrary-Rank Ablation (ARA) uncensored version of Qwen3.8-27B using heretic v1.2.0. Zero refusals on the standard censorship test suite, while preserving model quality (KL divergence 0.0535 from base). It handles creative writing, security topics, and controversial prompts without the usual Qwen guardrails.

Qwen3.8 uses the qwen3.5 architecture — a hybrid DeltaNet + attention model (48 linear + 16 full-attention layers). Only 16 layers grow KV cache, so context scaling is very cheap. The built-in vision encoder handles multimodal image understanding natively.

ROCmFPX: The speed story

The headline number is ROCmFP4_FAST at ~42 t/s average decode — that's 50–75% faster than stock Vulkan Q4_K_M-MTP (27–28 t/s). This uses ROCmFPX fork of llama.cpp which adds optimized quantization formats for RDNA 3.5.

ROCmFP4_FAST benchmarks (Vulkan0, MTP n-max 6, TurboQuant KV)

Task	Decode (t/s)	Prefill (t/s)
Code gen (3 runs)	34.5 / 41.6 / 49.5	42–62
Math reasoning	37.5	97.4
JSON extraction	44.7	137.1
Technical writing	43.8	59.6
Average	~42

Critical flags for Strix Halo

llama-server \
  -m Qwen3.8-27B-heretic-ara-ROCmFP4-FAST.gguf \
  -ngl 99 -dev Vulkan0 \
  -c 32768 \
  -ctk q8_0 -ctv turbo4 \
  -fa on \
  --spec-type draft-mtp \
  --spec-draft-n-max 6

• -dev Vulkan0 — This is the make-or-break flag. Forces the RADV cooperative matrix path. Without it, you get ~18 t/s instead of ~42. If you take one thing from this post, it's this flag.
• -ctk q8_0 -ctv turbo4 — TurboQuant KV cache. Speed-neutral but dramatically reduces KV memory. Enables 262K context in 20 GB vs 61 GB with f16 KV.
• --spec-draft-n-max 6 — Optimal for ROCmFPX on Strix Halo. Stock Vulkan prefers 3; higher values hurt on stock.

Stock Vulkan (no fork needed)

If you don't want to maintain a fork, Q4_K_M-MTP on stock llama.cpp with --spec-type draft-mtp --spec-draft-n-max 3 gets you 27–28 t/s. Still a solid experience. Q6_K-MTP gives higher quality at ~17–23 t/s.

llama-server \
  -m Qwen3.8-27B-heretic-ara-Q4_K_M-MTP.gguf \
  -ngl 99 -ctk f16 -ctv f16 \
  -c 32768 \
  --spec-type draft-mtp \
  --spec-draft-n-max 3

Vision

All quants work with the included mmproj for image understanding. Load it with --mmproj mmproj-Qwen3.8-27B-heretic-ara-BF16.gguf and --jinja. Vision decode runs at ~13 t/s — the vision encoder adds overhead, but text-only requests are completely unaffected by having mmproj loaded. Safe to keep it on permanently.

One gotcha: on ROCmFPX (b213), MTP + vision crashes due to non-consecutive token positions from image embeddings. Omit --spec-type draft-mtp when using mmproj on ROCmFPX. Stock llama.cpp handles vision + MTP fine.

Speed comparison summary

Config	Avg Decode (MTP)	Avg Prefill	File Size	Notes
ROCmFPX FP4_FAST	~42 t/s	~77 t/s	14 GB	Fastest. Requires ROCmFPX fork.
Stock Q4_K_M-MTP	~27–28 t/s	~170 t/s	16 GB	Best all-rounder. No fork.
ROCmFPX FP6	~24 t/s	~33 t/s	21 GB	Higher quality ROCmFPX.
Stock Q6_K-MTP	~17–23 t/s	~65 t/s	21 GB	Highest quality stock.

Code quality — EvalPlus HumanEval+ (164 problems, pass@1, temp=0)

Quant	HumanEval	HumanEval+	BPW
Q6_K-MTP	88.4%	81.7%	6.56
Q4_K_M-MTP	82.3%	75.6%	4.83
ROCmFP4_FAST	75.6%	71.3%	4.26
ROCmFP6	73.8%	67.7%	6.50

Stock llama.cpp quants (Q6_K, Q4_K_M) outperform ROCmFPX quants at similar BPW — Q6_K-MTP (6.56 BPW) scores 14% higher on HumanEval+ than ROCmFP6 (6.50 BPW). The ROCmFPX quantization format trades code quality for decode speed. If code quality matters for your use case, Q6_K on stock Vulkan is worth the speed trade-off.

How it was built

1. Downloaded heretic-ara safetensors from HuggingFace
2. Converted to BF16 GGUF with convert_hf_to_gguf.py — preserving all 866 tensors (851 model + 15 MTP)
3. Generated mmproj with convert_hf_to_gguf.py --mmproj --outtype bf16 (334 vision tensors, CLIP arch)
4. Stock quants: llama-quantize for Q4_K_M and Q6_K
5. ROCmFPX quants: ROCmFPX llama-quantize (b213/b2f5829) for FP4_FAST and FP6

All benchmarks run on AMD Ryzen AI Max+ 395 (122 GB unified, gfx1151), stock Vulkan on llama.cpp b10254, ROCmFPX on b213.

My daily driver setup

I run this as a local AI agent backend (hermes-agent) on the Strix Halo. The server runs as a systemd user service on port 8084 with mmproj permanently loaded, MTP enabled, and 131K context. For my workflow (multi-turn chat, tool calls, image understanding), the Q6_K-MTP on stock Vulkan hits the sweet spot of quality and speed. If I need pure text speed, the ROCmFP4_FAST is there at 42 t/s.

Happy to answer questions about the setup or Strix Halo-specific tweaks.

Edit: updated link to ROCmFPX
Edit2: Added coding benchmarks and updated speed summary with prefill stats as well.