---

Edit: Important note @ 12:01am US EST 16th Aug 2026 - If you had downloaded a copy of my diffs before this time then please update them as soon as possible. A number of issues with MTP correctness were discovered both in the upstream mainline branch, as well as my own code, and a further decode correctness issue was uncovered for certain model sizes. These should all be fixed now. If anyone finds any further issues, just drop me a note and we'll try to get it sorted ASAP.

---

Qwen3.8-27B-Q8_0 on AMD Radeon AI Pro R9700

This post is a followup to my earlier post here: https://www.reddit.com/r/ROCm/s/87da5FHpPO

My goal was to make ROCm the clear and easy choice for which backend to run.

I've tried to label the graphs a little more clearly this time around after feedback from the other post.

Since that post I've managed to eke out a few more small gains, and decided to test and share my results against the new Qwen3.8-27B model

The purpose of this post is to compare and present graphs for the following:

• Comparison of Qwen3.8-27B-Q8_0 on a single R9700 with different backends

• Comparison of Qwen3.8-27B-Q8_0 on two R9700's with different backends

• Comparison of Qwen3.8-27B-Q8_0 on three R9700's with different backends

• Scaling Comparison of Qwen3.8-27B-Q8_0 on 1 vs 2 vs 3 R9700s with Vulkan

• Scaling Comparison of Qwen3.8-27B-Q8_0 on 1 vs 2 vs 3 R9700s with Standard ROCm

• Scaling Comparison of Qwen3.8-27B-Q8_0 on 1 vs 2 vs 3 R9700s with Tweaked ROCm

Environment

Test Hardware

CPU: AMD 9950X3D2

Motherboard: Asus ProArt X870E Creator Wifi

GPU: 3 x AMD AI Pro R9700 - All set to run in PCIe5x4 mode

Memory: 96GB DDR5 @ 6000MHz CL30

Two of the GPUs are in the PCIe5x16 slots, but set to PCIe5x4 mode.

The third card is installed via an M.2->PCIe5 riser, which also grants PCIe5x4 speeds

All PCIe lane bandwidths are measured and verified

Libraries Used

• ROCm executables are linked against AMD ROCm 7.14.0 libraries from here: https://rocm.docs.amd.com/en/latest/install/rocm.html
• Vulkan executables are linked against the default system RADV libraries installed by Linux Fedora 44

Tweaked ROCm source code

The source code branch for the tweaked ROCm changes is found here: https://github.com/stew675/llama.cpp/tree/rdna-boosts

Building ROCm tweaked llama.cpp from source

I'm going to trust that you already know how to use git to pull the source code and change to the source directory

My Cmake ROCm build setup flags are these. Note that you must set ROCM_714 to installation directory of your ROCm 7.14 SDK

# You MUST set this path correctly
ROCM_714=/path/to/your/rocm-7.14/installation

export ROCM_PATH="$ROCM_714"

HIPCXX="$ROCM_714/lib/llvm/bin/clang" HIP_PATH="$ROCM_714" cmake -S . -B build \
  -DGGML_RPC=1 \
  -DGGML_HIP=ON \
  -DGGML_NATIVE=1 \
  -DGGML_HIP_RCCL=1 \
  -DHIP_PLATFORM=amd \
  -DGGML_HIP_GRAPHS=ON \
  -DGPU_TARGETS="gfx1200,gfx1201" \
  -DCMAKE_C_COMPILER="$ROCM_714/lib/llvm/bin/clang" \
  -DAMDGPU_TARGETS="gfx1200,gfx1201" \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER="$ROCM_714/lib/llvm/bin/clang++" \
  -DCMAKE_INSTALL_RPATH="\$ORIGIN:$ROCM_714/lib" \
  -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
  -DCMAKE_HIP_FLAGS="-mllvm --amdgpu-unroll-threshold-local=600"

# Then build
cmake --build build --config Release -j 16 -- VERBOSE=1

llama-server setup

Aside from the differences for specific backend setup, ie. Vulkan vs ROCm vs Tweaked ROCm, the following configuration is common for all backends used:

Common Server Command: ./build/bin/llama-server --model /llm/models/Qwen3.8/27B/Q8_0/Qwen3.8-27B-Q8_0.gguf --alias Qwen3.8-27B-Q8_0 --prio 2 --fit false --top-k 20 --port 8033 --threads 8 --parallel 1 --top-p 0.8 --min-p 0.001 --verbosity 3 --host 0.0.0.0 --cpu-strict 1 --cpu-range 0-7 --predict 98304 --threads-http 4 --load-mode mlock --cache-ram 16384 --ctx-size 70000 --flash-attn auto --temperature 0.7 --batch-size 1024 --ubatch-size 1024 --n-gpu-layers all --no-kv-unified --cache-type-k bf16 --cache-type-v bf16 --ctx-checkpoints 64 --cache-idle-slots --reasoning-budget 65536 --reasoning-preserve --checkpoint-min-step 4096 --repeat-penalty 1.0 --presence-penalty 1.5

Benchmark command

Test Command: uvx llama-benchy --base-url http://localhost:8033/v1 --tg 240 --pp 2520 --model Qwen3.8-27B-Q8_0 --tokenizer Qwen/Qwen3.8-27B --depth 0 4096 8192 16384 32768 65536

KEY TAKEWAYS

• ROCm Tweaked outperforms both Vulkan and standard ROCm build in every scenario
• For Tweaked ROCm, the gain from 1 to 2 GPUs is roughly 60%
• For Tweaked ROCm, the gain from 2 to 3 GPUs is roughly 10%
• Unless you need the extra VRAM for context, two cards is best
• Vulkan gets worse with more than two cards
• Stock ROCm is just slow once context grows

RAW RESULTS

Qwen3.8-27B-Q8_0, KV=BF16, 1 x Radeon AI Pro R9700

Vulkan

model	test	t/s	ttfr (ms)
Qwen3.8-27B-Q8_0	pp2520	946.78 ± 6.80	2663.43 ± 19.13
Qwen3.8-27B-Q8_0	tg240	19.60 ± 0.01
Qwen3.8-27B-Q8_0	pp2520 @ d4096	986.24 ± 7.02	6710.27 ± 48.76
Qwen3.8-27B-Q8_0	tg240 @ d4096	19.44 ± 0.01
Qwen3.8-27B-Q8_0	pp2520 @ d8192	969.12 ± 2.88	11055.59 ± 33.38
Qwen3.8-27B-Q8_0	tg240 @ d8192	19.27 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d16384	920.24 ± 2.80	20543.26 ± 63.60
Qwen3.8-27B-Q8_0	tg240 @ d16384	19.08 ± 0.01
Qwen3.8-27B-Q8_0	pp2520 @ d32768	820.18 ± 2.67	43026.47 ± 140.70
Qwen3.8-27B-Q8_0	tg240 @ d32768	18.45 ± 0.01
Qwen3.8-27B-Q8_0	pp2520 @ d65536	666.92 ± 2.06	102047.27 ± 316.31
Qwen3.8-27B-Q8_0	tg240 @ d65536	17.34 ± 0.00

ROCm Standard

model	test	t/s	ttfr (ms)
Qwen3.8-27B-Q8_0	pp2520	1079.74 ± 8.61	2335.32 ± 18.63
Qwen3.8-27B-Q8_0	tg240	18.93 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d4096	1072.23 ± 7.80	6172.38 ± 45.39
Qwen3.8-27B-Q8_0	tg240 @ d4096	18.27 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d8192	1009.76 ± 2.89	10608.92 ± 29.33
Qwen3.8-27B-Q8_0	tg240 @ d8192	17.68 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d16384	896.40 ± 2.39	21090.34 ± 56.25
Qwen3.8-27B-Q8_0	tg240 @ d16384	16.39 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d32768	721.45 ± 1.72	48913.28 ± 118.16
Qwen3.8-27B-Q8_0	tg240 @ d32768	14.46 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d65536	520.91 ± 1.19	130652.51 ± 298.47
Qwen3.8-27B-Q8_0	tg240 @ d65536	11.70 ± 0.00

ROCm Tweaked

model	test	t/s	ttfr (ms)
Qwen3.8-27B-Q8_0	pp2520	1108.34 ± 12.55	2274.71 ± 25.31
Qwen3.8-27B-Q8_0	tg240	20.10 ± 0.02
Qwen3.8-27B-Q8_0	pp2520 @ d4096	1157.21 ± 8.47	5719.09 ± 42.31
Qwen3.8-27B-Q8_0	tg240 @ d4096	19.94 ± 0.03
Qwen3.8-27B-Q8_0	pp2520 @ d8192	1141.54 ± 3.73	9383.75 ± 31.09
Qwen3.8-27B-Q8_0	tg240 @ d8192	19.79 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d16384	1087.36 ± 3.68	17386.11 ± 58.35
Qwen3.8-27B-Q8_0	tg240 @ d16384	19.46 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d32768	969.84 ± 2.96	36385.90 ± 110.97
Qwen3.8-27B-Q8_0	tg240 @ d32768	18.84 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d65536	792.60 ± 2.77	85866.79 ± 299.56
Qwen3.8-27B-Q8_0	tg240 @ d65536	17.69 ± 0.00

Qwen3.8-27B-Q8_0, KV=BF16, 2 x Radeon AI Pro R9700

Vulkan split-mode=layer

model	test	t/s	ttfr (ms)
Qwen3.8-27B-Q8_0	pp2520	941.77 ± 6.36	2677.34 ± 18.08
Qwen3.8-27B-Q8_0	tg240	18.05 ± 0.02
Qwen3.8-27B-Q8_0	pp2520 @ d4096	1173.67 ± 11.23	5638.30 ± 54.39
Qwen3.8-27B-Q8_0	tg240 @ d4096	17.89 ± 0.01
Qwen3.8-27B-Q8_0	pp2520 @ d8192	1309.91 ± 4.56	8178.90 ± 28.44
Qwen3.8-27B-Q8_0	tg240 @ d8192	17.79 ± 0.01
Qwen3.8-27B-Q8_0	pp2520 @ d16384	1388.40 ± 3.96	13616.82 ± 38.79
Qwen3.8-27B-Q8_0	tg240 @ d16384	17.52 ± 0.03
Qwen3.8-27B-Q8_0	pp2520 @ d32768	1340.08 ± 2.63	26333.53 ± 51.22
Qwen3.8-27B-Q8_0	tg240 @ d32768	16.99 ± 0.01
Qwen3.8-27B-Q8_0	pp2520 @ d65536	1153.97 ± 6.27	58978.89 ± 321.04
Qwen3.8-27B-Q8_0	tg240 @ d65536	16.05 ± 0.01

ROCm Standard, split-mode=tensor

model	test	t/s	ttfr (ms)
Qwen3.8-27B-Q8_0	pp2520	1416.38 ± 23.48	1780.06 ± 28.80
Qwen3.8-27B-Q8_0	tg240	29.01 ± 0.05
Qwen3.8-27B-Q8_0	pp2520 @ d4096	1541.34 ± 23.71	4294.45 ± 66.06
Qwen3.8-27B-Q8_0	tg240 @ d4096	28.21 ± 0.07
Qwen3.8-27B-Q8_0	pp2520 @ d8192	1528.94 ± 9.33	7007.80 ± 42.43
Qwen3.8-27B-Q8_0	tg240 @ d8192	27.46 ± 0.04
Qwen3.8-27B-Q8_0	pp2520 @ d16384	1425.14 ± 7.16	13265.79 ± 66.99
Qwen3.8-27B-Q8_0	tg240 @ d16384	26.17 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d32768	1208.55 ± 4.46	29200.20 ± 107.67
Qwen3.8-27B-Q8_0	tg240 @ d32768	23.71 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d65536	909.59 ± 4.58	74823.96 ± 376.66
Qwen3.8-27B-Q8_0	tg240 @ d65536	19.86 ± 0.00

ROCm Tweaked, split-mode=tensor

model	test	t/s	ttfr (ms)
Qwen3.8-27B-Q8_0	pp2520	1457.92 ± 21.43	1729.87 ± 25.42
Qwen3.8-27B-Q8_0	tg240	31.24 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d4096	1641.60 ± 21.37	4031.51 ± 52.77
Qwen3.8-27B-Q8_0	tg240 @ d4096	30.79 ± 0.07
Qwen3.8-27B-Q8_0	pp2520 @ d8192	1674.79 ± 7.66	6397.07 ± 29.24
Qwen3.8-27B-Q8_0	tg240 @ d8192	30.52 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d16384	1655.24 ± 10.02	11421.77 ± 68.86
Qwen3.8-27B-Q8_0	tg240 @ d16384	30.11 ± 0.02
Qwen3.8-27B-Q8_0	pp2520 @ d32768	1541.65 ± 9.55	22890.95 ± 141.86
Qwen3.8-27B-Q8_0	tg240 @ d32768	29.36 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d65536	1303.84 ± 9.27	52200.88 ± 371.47
Qwen3.8-27B-Q8_0	tg240 @ d65536	27.95 ± 0.01

Qwen3.8-27B-Q8_0, KV=BF16, 3 x Radeon AI Pro R9700

Vulkan split-mode=layer

model	test	t/s	ttfr (ms)
Qwen3.8-27B-Q8_0	pp2520	936.08 ± 7.40	2693.75 ± 21.30
Qwen3.8-27B-Q8_0	tg240	16.26 ± 0.02
Qwen3.8-27B-Q8_0	pp2520 @ d4096	1111.70 ± 10.05	5953.08 ± 53.82
Qwen3.8-27B-Q8_0	tg240 @ d4096	16.13 ± 0.05
Qwen3.8-27B-Q8_0	pp2520 @ d8192	1162.28 ± 5.23	9217.87 ± 42.34
Qwen3.8-27B-Q8_0	tg240 @ d8192	16.13 ± 0.21
Qwen3.8-27B-Q8_0	pp2520 @ d16384	1161.92 ± 2.42	16270.07 ± 33.92
Qwen3.8-27B-Q8_0	tg240 @ d16384	15.13 ± 0.22
Qwen3.8-27B-Q8_0	pp2520 @ d32768	1071.11 ± 2.82	32947.18 ± 86.33
Qwen3.8-27B-Q8_0	tg240 @ d32768	14.62 ± 0.02
Qwen3.8-27B-Q8_0	pp2520 @ d65536	888.53 ± 4.50	76597.24 ± 388.46
Qwen3.8-27B-Q8_0	tg240 @ d65536	13.94 ± 0.02

ROCm Standard, split-mode=tensor

model	test	t/s	ttfr (ms)
Qwen3.8-27B-Q8_0	pp2520	1469.22 ± 28.82	1717.62 ± 33.00
Qwen3.8-27B-Q8_0	tg240	34.20 ± 0.09
Qwen3.8-27B-Q8_0	pp2520 @ d4096	1653.39 ± 28.81	4003.70 ± 69.76
Qwen3.8-27B-Q8_0	tg240 @ d4096	32.24 ± 0.06
Qwen3.8-27B-Q8_0	pp2520 @ d8192	1628.31 ± 22.93	6580.95 ± 93.27
Qwen3.8-27B-Q8_0	tg240 @ d8192	30.48 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d16384	1522.01 ± 14.99	12422.69 ± 122.34
Qwen3.8-27B-Q8_0	tg240 @ d16384	28.86 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d32768	1266.96 ± 7.91	27855.14 ± 174.19
Qwen3.8-27B-Q8_0	tg240 @ d32768	25.96 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d65536	917.55 ± 7.92	74178.18 ± 639.89
Qwen3.8-27B-Q8_0	tg240 @ d65536	21.49 ± 0.01

ROCm Tweaked, split-mode=tensor

model	test	t/s	ttfr (ms)
Qwen3.8-27B-Q8_0	pp2520	1494.97 ± 3.91	1686.30 ± 4.75
Qwen3.8-27B-Q8_0	tg240	37.30 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d4096	1745.76 ± 31.97	3792.19 ± 69.15
Qwen3.8-27B-Q8_0	tg240 @ d4096	35.71 ± 0.10
Qwen3.8-27B-Q8_0	pp2520 @ d8192	1787.03 ± 11.47	5995.43 ± 38.47
Qwen3.8-27B-Q8_0	tg240 @ d8192	34.35 ± 0.08
Qwen3.8-27B-Q8_0	pp2520 @ d16384	1780.61 ± 15.12	10617.95 ± 90.45
Qwen3.8-27B-Q8_0	tg240 @ d16384	33.89 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d32768	1660.52 ± 8.73	21252.42 ± 111.99
Qwen3.8-27B-Q8_0	tg240 @ d32768	32.84 ± 0.02
Qwen3.8-27B-Q8_0	pp2520 @ d65536	1374.56 ± 15.54	49518.77 ± 560.02
Qwen3.8-27B-Q8_0	tg240 @ d65536	31.09 ± 0.03

---------------------------------
Edit: Important note @ 12:01am US EST 16th Aug 2026 - If you had downloaded a copy of my diffs before this time then please update them as soon as possible. A number of issues with MTP correctness were discovered both in the upstream mainline branch, as well as my own code, and a further decode correctness issue was uncovered for certain model sizes. These should all be addressed now. If anyone finds any further issues, just drop me a note and we'll try to get it sorted ASAP.

---------------------------------

With a lot of new model releases lately and Qwen3.8-27B right around the corner, I wanted to share a set of ROCm/RDNA focused speed up I've been working on. These largely exist as a set of diffs that can be applied cleanly on top of upstream llama.cpp, and my goal was to get ROCm up to parity with Vulkan, or as close as possible.

I have a 7900XTX, Strix Halo, and a machine with 3 x R9700's GPUs in it, so I've been testing my work against all three architectures. I want to share a bunch of graphs for the R9700's which shows the relative speedups achieved.

My git repo branch is here: https://github.com/stew675/llama.cpp/tree/rdna-boosts

I tested Qwen3.6-27B, Muse Glimmer 30B, and Gemma4-31B. For single cards I used the Q6_K quants of these models (except for Glimmer where Q8_0 did fit well), and for 2 or 3 cards I used the Q8_0 model weights.

One element of my changes is I targetted native BF16 support for the KV cache. I know that it came as a surprise to me that standard ROCm support in llama.cpp just maps BF16 to F16 for the KV cache, and this mapping both slows things down, AND gives zero precision benefit. There was literally no good reason to choose BF16 for KV cache, ever, with ROCm. The curious thing though is that for Vulkan, it DOES implement BF16 KV cache correctly.

As such, the graphed runs below are all with a BF16 KV cache, which is hardware supported, at least for the important vector dot product operations, on anything RDNA3+. Scalar BF16 support isn't properly there, but that doesn't matter here.

My test system runs Fedora44. My build sequence for standard Fedora ROCm is:

HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" cmake -S . -B build \
  -DGGML_RPC=1 \
  -DGGML_HIP=ON \
  -DGGML_NATIVE=1 \
  -DGGML_HIP_RCCL=1 \
  -DHIP_PLATFORM=amd \
  -DGGML_HIP_GRAPHS=ON \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_HIP_ROCWMMA_FATTN=OFF \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DGPU_TARGETS="gfx1200;gfx1201" \
  -DCMAKE_INSTALL_RPATH="\$ORIGIN" \
  -DAMDGPU_TARGETS="gfx1200;gfx1201" \
  -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
  -DCMAKE_HIP_FLAGS="-mllvm --amdgpu-unroll-threshold-local=600"

cmake --build build --config Release -j 16 -- VERBOSE=1

and if you have ROCm 7.14 installed, it's this:

ROCM_714=/opt/rocm-7.14-gfx1201

[ -d "$ROCM_714" ] || { echo "error: $ROCM_714 not found" >&2; exit 1; }


export ROCM_PATH="$ROCM_714"

HIPCXX="$ROCM_714/lib/llvm/bin/clang" HIP_PATH="$ROCM_714" cmake -S . -B build \
 -DGGML_RPC=1 \
 -DGGML_HIP=ON \
 -DGGML_NATIVE=1 \
 -DGGML_HIP_RCCL=1 \
 -DHIP_PLATFORM=amd \
 -DGGML_HIP_GRAPHS=ON \
 -DGPU_TARGETS="gfx1200,gfx1201" \
 -DCMAKE_C_COMPILER="$ROCM_714/lib/llvm/bin/clang" \
 -DAMDGPU_TARGETS="gfx1200,gfx1201" \
 -DCMAKE_BUILD_TYPE=Release \
 -DCMAKE_CXX_COMPILER="$ROCM_714/lib/llvm/bin/clang++" \
 -DCMAKE_INSTALL_RPATH="\$ORIGIN:$ROCM_714/lib" \
 -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
 -DCMAKE_HIP_FLAGS="-mllvm --amdgpu-unroll-threshold-local=600"

cmake --build build --config Release -j 16 -- VERBOSE=1

All tests were done using llama-benchy against a live llama-server instance. I chose a generation size of 240 as that seems to align fairly closely with the typical tool-call generation size, and I chose a prefill size of 2520 to be representative of typical agentic coding input sizes.

As such, my test command invocation looks like so:

uvx llama-benchy --base-url http://localhost:8033/v1 --tg 240 --pp 2520 --model Muse-Glimmer-30B-Q8_0  --tokenizer meta-models/Muse-Glimmer-30B --depth 0 4096 8192 16384 32768 65536

I have a PR against upstream llama.cpp for some of this work, but it seems to be getting ignored, likely because it doing changes against the CUDA kernels, and so they (probably rightfully) don't trust some random yahoo like myself to be messing with that stuff. Still, I don't want to let that get in the way of sharing these tweaks with the ROCm community.

I hope people find this stuff to be useful. The speeds do apply to all 7xxx series cards, and the Strix Point/Halo architectures, but some speedups were only viable on RDNA4, so your results may be mixed depending on what you're using.

tl;dr: This is about this llama.cpp PR here: https://github.com/ggml-org/llama.cpp/pull/26856

Over the weekend I decided to dig into why, when specifying the BF16 type for KV cache on my Strix Halo box, that token generation performance tanked by ~20% over using F16.

Qwen3.6-35B-A3B-Q8_0 @ 16K CTX, no MTP

KV type	tg64 (t/s)
F16	43.09
BF16	34.90
Q8_0	40.83
F32	29.54

What was worse was that the output quality didn't seem to improve either. Perplexity scores for F16 vs BF16 were the same, and both are about 5% above the F32 baseline. I dug into the architectural specs for RDNA 3, 3.5, and 4, and it seems that all 7xxx, 9xxx series GPUs, and all Strix Point/Strix Halo iGPUs have native support for BF16 dot product matrix math. The scalar ALU portions seems to lack complete native BF16 support, but that's not really needed for Flash Attention.

So, with the assistance of DeepSeek V4 Flash 0731, I started to dig into the situation and see if it was possible to make BF16 KV caches go faster. It didn't take long to learn that llama.cpp doesn't really implement native BF16 support beyond simply accepting the data type, and then converting everything to F16 anyway. It was this BF16->F16 conversion that was killing the generation performance on RDNA GPUs and the Strix Halo.

So I worked to implement native BF16 support for KV Flash Attention, and it turned out it was possible to do so. In fact, through bypassing the old BF16->FP16 path I was able to see achieve between a 10-15% pre-fill speedup on the Strix Halo. The gain is smaller on the 7900XTX and R9700 GPUs (more like 5-10%), but it was real.

Then I set about tuning the matrix kernels, and was able to achieve token generation parity with the original F16 path, which is the fastest token generation path available on the Strix Halo.

Now, since BF16 has a much larger dynamic range than F16 I then thought to compare the Perplexity results when using the native BF16 kernel path, and was surprised to find that, at depth, the BF16 KV cache path almost exactly matches the baseline F32 KV cache. Incidentally, the F32 KV cache path also uses F16 math under the hood if Flash Attention is enabled, so I had to disable Flash Attention to get at the true F32 Perplexity baseline figures.

Precision at depth (PPL @ 32k, Qwen3.5-4B-Q8_0, full wikitext-2 test, 9 chunks)

KV cache / FA	PPL @ 32k
F32 baseline (FA off)	8.6368
BF16 (this change)	8.6403 (+0.04%)
F16	9.1400 (+5.8%)

What does that mean? It means that for long context work that the models are less likely to go astray and/or suffer from context rot.

Note: For Strix Halo boxes that until such time that the upstream ROCm libraries get fixed, that you must disable mmap, as well as set HIP_LAUNCH_BLOCKING=1 to avoid running into a pair of rather nasty bugs that will heavily corrupt your KV cache. See my post here for more details: https://www.reddit.com/r/StrixHalo/s/Qbh3RDVsXg

What About Vulkan?

Now you may ask: "What about Vulkan?". The answer is that the Vulkan path in llama.cpp already implements this BF16 behavior properly, so this PR just brings ROCm up to Perplexity parity with Vulkan. The part from this PR that can't be done on Vulkan though is the RDNA3+ specific dot-product optimizations, so this PR makes ROCm pre-fill significantly faster than Vulkan.

Here's Vulkan vs ROCm results with Qwen3.6-35B at 32K context depth:

F16 K/V Performance

model	size	params	backend	ngl	fa	test	t/s
qwen35moe 35B.A3B Q8_0	35.19 GiB	35.51 B	ROCm	-1	1	pp1024 @ d32768	618.31 ± 4.83
qwen35moe 35B.A3B Q8_0	35.19 GiB	35.51 B	ROCm	-1	1	tg256 @ d32768	42.42 ± 0.29
qwen35moe 35B.A3B Q8_0	35.19 GiB	35.51 B	Vulkan	-1	1	pp1024 @ d32768	717.93 ± 3.69
qwen35moe 35B.A3B Q8_0	35.19 GiB	35.51 B	Vulkan	-1	1	tg256 @ d32768	46.86 ± 0.01

BF16 K/V Performance

model	size	params	backend	ngl	type_k	type_v	fa	test	t/s
qwen35moe 35B.A3B Q8_0	35.19 GiB	35.51 B	ROCm	-1	bf16	bf16	1	pp1024 @ d32768	684.02 ± 8.36
qwen35moe 35B.A3B Q8_0	35.19 GiB	35.51 B	ROCm	-1	bf16	bf16	1	tg256 @ d32768	42.51 ± 0.06
qwen35moe 35B.A3B Q8_0	35.19 GiB	35.51 B	Vulkan	-1	bf16	bf16	1	pp1024 @ d32768	484.00 ± 3.83
qwen35moe 35B.A3B Q8_0	35.19 GiB	35.51 B	Vulkan	-1	bf16	bf16	1	tg256 @ d32768	47.59 ± 0.01

Interesting Results here:

• For F16 Vulkan handily beats ROCm for both pre-fill and decode, but keep in mind that this is coming at the cost of long context quality (for both).
• For BF16 ROCm handily beats Vulkan by 40% for pre-fill, but Vulkan beats ROCm by ~11% for generation. If using the Strix Halo for agentic coding, then you're clearly going to want ROCm here for speed and quality at long contexts.

Edit: 11th Aug - Faster ROCm generation

Maybe I'll make it my next job to see if I can make ROCm close that generation performance gap with Vulkan. I have a feeling that it must be possible.

I have completed the ROCm generation speed-up work to as far as I can take it. At the end of the day the Vulkan dispatcher is just more efficient than ROCm's, and I'm now up against that fundamental library difference. Still, I've managed to more than halve the generation performance gap from ROCm to Vulkan, while even boosting ROCm's prefill advantage. The source-code branch for this follow work is here: https://github.com/stew675/llama.cpp/tree/make-rocm-gen-faster

It's unlikely that the follow-up branch would ever be accepted into upstream llama.cpp, so I'll try to make daily upstream rebases to keep that work current for people who are interested.

Edit: 12th Aug - RDNA4 boosts

I've created a consolidated branch that includes all prior work plus a host of RDNA4 speed boosts and put it into this branch here: https://github.com/stew675/llama.cpp/tree/rdna-boosts

This branch boosts ROCm prefill speeds on RDNA4 GPUs by a further 5-25% over the prefill speed boosts mentioned above. This means a total of +10-40% prefill speed ups on RDNA4, and +5-15% prefill speedups on RDNA3/3.5, Sadly the RDNA3/3.5 architectures simply don't have the CU hardware to support the additional prefill speeds that RDNA4 can achieve.

The rdna-boosts branch also includes the 5-10% ROCm generation speed boosts, which applies to all RDNA3+ GPUs. It also fixes the async-race memory corruption issues on Strix Halo that is still present in upstream llama.cpp. Note that mmap loading is still broken on Strix Halo at the moment.

https://huggingface.co/amd/Instella-MoE-16B-A3B-Think

I was browsing HuggingFace and came across this model apparently uploaded a day ago, and thought to share it here. I've not tried it out yet, but it's good to see AMD joining the open source model game.

Let's start a discussion about what can be done to make local models more reliable.

I've been using Qwen3.6-27B a lot lately, and have noticed the same thing that many others talk about here, which is the tool-call failures and looping that really gets in the way of being able to use what is otherwise a pretty intelligent mode as a fully local stand-in for some of the frontier models.

I found that I have to watch it like a hawk constantly, otherwise it'll start to loop after a while, or just randomly stop with a tool-call hallucination.

I managed to work around this issue by creating an extension to the Pi coding agent, that monitors the JSON stream for looping, as well as checking to see if the user's goal has been reached, and if it hasn't and the main model has stopped, then it'll automatically inject some prompts encouraging the model to continue.

That was my attempt to solve the issue and it seems to work pretty well. I can now reasonably safely walk away from it when it's working on a large task when I turn my extension on, and I've seen it kick in and keep the model going when it gets off track.

I'm sure I'm not alone, and in this world of everyone writing their own coding agents, I'm wanting to know what others are using to get around these quirks of local models, or are you all just using paid models, or watching the monitor constantly, when you want to be sure that a job gets done?

Edit: A number of users have pointed out that this chat template has fixed common looping/tool-call failure issues: https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates

I wanted something that I could easily configure to manage a set of sensible defaults, that supports multiple llama-server binaries, with per-model over-rides, and command line over-rides.

The utility is here: https://github.com/stew675/start-llama

I know that llama-server has its own model loading configuration available via the API end-point, but I just wanted something that I could start from the command line easily in one step.

I don't know if anyone else may find this useful or not, but I'll share it here anyway in case someone does.