​

I built a native vLLM + ROCm 7.15 runtime for AMD RX 6000 Series on Windows 11 — now with a one-click installer.

AMD lists RX 6750 XT / 6700 XT / 6600 XT as "Runtime only" on Windows with HIP SDK excluded. I built rocBLAS binaries for gfx1031 via ROCm/TheRock to close that gap.

This runs native Windows HIP and ROCm directly, no WSL2 wrapper, no compiler needed.

Tested on AMD Radeon RX 6750 XT 12GB (gfx1031) — Windows 11 Native — August 2026

# Verification — Real terminal logs

1. Environment:

torch 2.12.0+rocm7.15.0 | cuda_avail True | dev AMD Radeon RX 6750 XT

vLLM 0.19.1 | plugin vllm_windows_rocm activated | TRITON_ATTN | enforce_eager=True

2. rocBLAS Benchmark — 26 TFLOPS FP16:

Device ID 0 : AMD Radeon RX 6750 XT gfx1031 with 12.9 GB memory

rocBLAS version: 5.7.0.67811f1ee52

transA,transB,M,N,K,alpha,lda,beta,ldb,ldc,cold_iters,hot_iters,rocblas-Gflops,us

N,N,4096,4096,4096,1,4096,0,4096,4096, 2, 10, 25977.3, 5290.73

=> 25.97 TFLOPS in 5.29ms

3. vLLM Inference (Qwen3.5-4B, web chat UI):

Output: 59-62 tok/s | Init ~1s | hidden thinking (spinner) then answer

Model auto-detected via /v1/models

Full logs in benchmarks/ and screenshots in assets/ on GitHub.

# How it works

1. TheRock builds clr (HIP) and rocBLAS with Tensile kernels for gfx1031

2. HSA_OVERRIDE_GFX_VERSION=10.3.0 forces HIP to recognize RX 6750 XT

3. PyTorch 2.12.0+rocm7.15 links against TheRock runtime => torch.cuda.is_available() True

4. vLLM plugin vllm_windows_rocm bypasses vllm._C dependency and registers WinRocmAwqGemvKernel with TRITON_ATTN

5. vLLM engine loads with enforce_eager=True and runs native

# Quick Start — One click

Prerequisites: Windows 11 23H2+, RX 6600-6750 XT (gfx1030/gfx1031/gfx1032), Adrenalin 24.x+, ~8GB free disk

1. Download the repo (or git clone) and run INSTALL.bat as Administrator

2. The installer downloads the release archives automatically (multi-part join included, no manual steps) and sets up C:\TheRock\ + Python 3.11 + venv

3. Run CHAT.bat — the browser opens the chat UI. Expected: 59-62 tok/s

VERIFY.bat runs a 3-in-1 check: GPU detection + rocBLAS benchmark + vLLM smoke test.

# What's new in v2.0

- One-click installer: INSTALL.bat auto-downloads and joins split archives (no browser upload/download hassle)

- Chat web UI (OpenAI-compatible API): hidden thinking with spinner, then the answer streams — like NVIDIA ChatRTX

- Fat-binary gemv kernels for all RDNA2 (gfx1030/1031/1032) — no per-card rebuild

- ROCm 7.15 native, vLLM 0.19.1, torch 2.12.0+rocm7.15

- Faster: 59-62 tok/s output (was 54.2)

- Repo cleaned: INSTALL.bat + CHAT.bat + VERIFY.bat + MANIFEST.json + docs, archives on GitHub Releases

# Known Issues — Please note this may have bugs

This is an experimental reference implementation. It works on my RX 6750 XT but could present bugs on other hardware.

- Other RDNA2 cards (6600/6600 XT/6700 XT) not yet tested — may need 10.3.0 vs 10.3.1 or rocBLAS rebuild for gfx1030

- enforce_eager=True required — torch.compile disabled on RDNA2 Windows

- FP8 / AWQ not tested yet, multi-GPU not tested

- If you test it on your RDNA2 card, please open an Issue with GPU model and logs — contributions welcome

# GitHub Repo:

https://github.com/sebastianmechno-sys/vllm-rocm-windows-rdna2

Includes INSTALL.bat, CHAT.bat, VERIFY.bat, scripts/serve.py (OpenAI-compatible API), chat.html, assets with screenshots, docs/BUILD_ROCBLAS.md, MANIFEST.json with release checksums

Let me know if you test it on other RDNA2 cards.

Built on ROCm/TheRock, PyTorch ROCm, vLLM. Not affiliated with AMD. License Apache 2.0

Specs:
AMD 9070 XT 16GB
32GB DDR4 3200
i7-12700k
All models loaded from m.2.

I'm using A14B models from CivitAI(HIGH NOISE, and LOW NOISE). The actual sampling steps for 4 steps is less than 2 minutes for 81 frames at 720p, while the overrall gen takes 8-10 minutes, almost all of it spent on VAE decode. I tried using KJ node to FORCE a GPU load for VAE, in case any silent fallback to CPU or system RAM was happening, but I got no meaningful boost from that. Watching task manager I am getting sawtooth style compute pattern during VAE decode, but I can confirm decoding is happening on GPU.

Is this a known pain point for AMD/ROCm?

Using this Comfy distro: https://github.com/patientx-cfz/comfyui-rocm

**TL;DR:** Running a 35B A3B model with CPU/GPU split on a 12GB card means prefill/TTFT at large context (70K+) is what actually makes or breaks usability — not decode. Vulkan was fine at 16-32K context but became painful past that. Got llama.cpp running fully native on ROCm for gfx1031 (RX 6700XT) instead — flash attention on, quantized KV cache, no `HSA_OVERRIDE_GFX_VERSION` hacks, no core dumps, prefill peaking around 580 tok/s even with 20K+ tokens already cached. The fix ended up being a one-line change in `ggml/src/ggml-cuda/fattn.cu` forcing the MMA kernel path for a specific head_dim instead of the default tile kernel. Once this was fixed at the ROCm/HIPBLAS level, every other inference engine I tested (Ollama, Unsloth, LM Studio, SGLang, vLLM) also started working natively — this wasn't a llama.cpp-only fix, it was a root-level fix for the whole ROCm stack on this GPU.

I'm not a developer. No CS/programming background — I'm a creative director by trade (photography/cinematography) who's been self-teaching local LLM inference for a few months. So this write-up is going to read more like "how I diagnosed this empirically" than "here's the technical theory of why it works." If anyone with deeper kernel/ROCm knowledge can fill in the *why*, I'd genuinely love to learn it.

---

## The problem

gfx1031 sits in an awkward spot: it's RDNA2, but not gfx1030 (the RX 6800/6900 series, which has the most complete official ROCm support and by far the most community documentation). Most guides, fixes, and reference configs you'll find online target gfx1030. Applying them as-is to a 6700XT either silently falls back to worse behavior, or crashes outright.

The common workaround people recommend is `HSA_OVERRIDE_GFX_VERSION=10.3.0` (telling the runtime to treat your gfx1031 card as gfx1030). I tried this. Here's what happened:

- Flash attention wouldn't enable

- Some ops (noticed it specifically with top_k) silently fell back to CPU, tanking decode speed

- Random core dumps mid-inference, not tied to any consistent trigger

I made the call to require **native gfx1031 recognition, no override, no spoofing the runtime**. That decision alone cost me a lot of trial and error, but I think it was the right one — see below why.

## Why I bothered (this isn't about decode speed)

I want to be specific about this because I think it's the part most people miss. If you're running a model that fits entirely in VRAM, ROCm being suboptimal (or partial CPU fallback on certain ops) doesn't hurt you that much — most of the compute is already on GPU anyway.

My case is different: **35B A3B with CPU/GPU split** (`--n-cpu-moe 22`, dense/attention layers on GPU, part of the MoE experts on CPU). In this setup, prefill isn't pure GPU compute — a portion of it depends on the CPU-offloaded expert path too. When the backend isn't efficient there, and context climbs into the 70-80K+ range, TTFT doesn't degrade linearly — it gets genuinely painful. At 16-32K context, Vulkan's prefill (roughly 70-100 tok/s in my case) was still tolerable. Past that, it wasn't.

The other piece of motivation: RX 6700XT's boost clock sits dramatically above an RTX 3060 12GB (2581MHz vs 1777MHz, roughly +45%), while VRAM bandwidth is only modestly higher (~384GB/s vs ~360GB/s, about +7%). That distinction matters here: decode is largely bandwidth-bound (reading/writing KV cache and weights per token), so that 7% bandwidth gap barely moves the needle on decode speed. Prefill, on the other hand, leans more on raw compute throughput — which is exactly where the 45% clock gap should show up. I'd read threads of people running 35B A3B with CPU/GPU split on a 3060 hitting peak prefill around ~300 tok/s. Given the clock gap, I'd expect the 6700XT to clear that by a wide margin, not just edge past it. Getting nowhere close to that on Vulkan was the signal that I was leaving real performance on the table — a software bottleneck, not a hardware ceiling — and that gap is what pushed me to actually fix this instead of settling.

After the fix, here's an actual log excerpt from a real session (task with context already at ~18-23K tokens, prompt processing in progress):

```

prompt processing, n_tokens = 2048, progress = 0.55, t = 3.52s / 581.81 tokens per second

prompt processing, n_tokens = 3072, progress = 0.58, t = 5.60s / 548.24 tokens per second

prompt processing, n_tokens = 4096, progress = 0.60, t = 7.71s / 531.24 tokens per second

prompt processing, n_tokens = 5120, progress = 0.63, t = 10.07s / 508.61 tokens per second

prompt processing, n_tokens = 6144, progress = 0.66, t = 12.24s / 502.11 tokens per second

```

Peak prefill of ~580 tok/s, staying above 500 tok/s while cached context was already north of 20K tokens. That's the number that mattered to me — not decode.

## Environment

- OS: Ubuntu Desktop 26.04 LTS

- GPU: AMD RX 6700XT 12GB (gfx1031, RDNA2)

- CPU: Intel i5-11400F

- RAM: 16GB DDR4 3200MT/s

- ROCm: custom build from TheRock binaries (official gfx1031 support isn't guaranteed across all standard ROCm releases)

- Inference engine: llama.cpp (build-rocm), Ollama, Unsloth, LM Studio, SGLang, vLLM

**A specific version note, since ROCm/TheRock builds vary a lot:** don't install just any TheRock ROCm build and expect this to match. The exact version I validated this on:

```

$ hipcc --version

HIP version: 7.14.60850-0000000

AMD clang version 23.0.0git (https://github.com/ROCm/llvm-project.git 46fcb339fb61119b337f973c7ca9e710a319fdd0+PATCHED:440716f8b87be9d8e20ed910e10e5b6d14d57cf6)

Target: x86_64-unknown-linux-gnu

Thread model: posix

InstalledDir: /opt/rocm/core-7.14/lib/llvm/bin

```

If you're on a different HIP/ROCm build, expect some variance — flag this version when comparing notes so we're not chasing different behavior across different builds.

**llama.cpp build/commit used:**

```

$ llama-server --version

version: 10307 (fc3f10b38)

built with GNU 15.2.0 for Linux x86_64

```

Given how much commit drift there is upstream (see caveats below), this matters if you're trying to reproduce the fix exactly — the `fattn.cu` logic may have shifted on a different commit.

**Confirmation ROCm actually sees this as gfx1031 (not spoofed via override):**

```

$ rocminfo | grep -i gfx

Name: gfx1031

Name: amdgcn-amd-amdhsa--gfx1031

Name: amdgcn-amd-amdhsa--gfx10-3-generic

```

**Exact card / board partner** (in case behavior varies by AIB):

```

$ lspci -vv | grep -A 2 "VGA"

03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M/6850M XT] (rev c1)

Subsystem: Sapphire Technology Limited Radeon RX 6700 XT GAMING OC 12G [Sapphire PULSE]

```

**I'll drop screenshots in the comments** as proof this is actually running native ROCm (not Vulkan silently, not CPU fallback) — GPU load/clock from a monitoring tool like `btop`/`lact`, prefill and decode averages from actual server logs, and junction temperature under sustained load, similar to what I used to validate stability throughout this whole process.

You can empirically look at my data.

1. ROCm natively
2. Total token when i use in just that runtime.
3. Temperatur edge, junction, and VRAM when prefill bound and decode bound.
4. Power usage both (prefill, decode).
5. BTOP verified some layers in cpu.

## The debugging process (the messy, real version)

Build kept failing. Tried the documented head_dim config for RDNA2 — failed. Tried random power-of-2 values (1024, 2048, 4096) as a guess — failed on all of them.

At that point I stopped touching the keyboard, made coffee, and just watched the failing terminal output for a while. Then it occurred to me to check `btop` instead of guessing blind. I ran the model through **Vulkan** (which was stable on this GPU, just slow) as a known-good baseline, watched which kernel showed up in btop tied to the running process, and correlated that behavior back to the head_dim condition in `fattn.cu`.

That's where the number `512` came from — not from any documentation I could find (I genuinely don't remember a source for it), but from matching observed hardware behavior to the kernel dispatch logic in source.

## The fix

In `ggml/src/ggml-cuda/fattn.cu`:

```cpp

// Force MMA kernel for head_dim 512 on AMD to avoid tile kernel shared memory limit

if (amd_mfma_available(cc) && Q->ne[0] == 512) {

return BEST_FATTN_KERNEL_MMA_F16;

}

return BEST_FATTN_KERNEL_TILE;

```

The default tile kernel path was hitting a shared memory limit on this hardware under certain conditions. Forcing the MMA (matrix-core) kernel path at this specific head_dim avoided that entirely.

**Why this seems to matter for the whole chain:** flash attention needs to be stable first before KV cache quantization is viable, and KV cache quantization is what makes long context (I run up to 131072 ctx) actually usable on 12GB VRAM. So this one fix unblocked flash-attn → which unblocked KV quant → which unblocked long context. If flash-attn doesn't work, none of the rest follows.

## Result

- Prefill peaking around ~580 tok/s (measured with 20K+ tokens already cached, not a cold-start number), staying in the 500-580 tok/s range across that phase — this is the number that actually matters for my use case, since TTFT at large context is what was hurting on Vulkan

- Native ROCm/HIPBLAS/ROCBLAS, no override, across every inference engine I tested — not just llama.cpp

- Flash attention stable, KV cache running q8_0, ctx up to 131072 confirmed stable

- Decode holds steady around ~22-23 tok/s on a 35B A3B model (IQ4_NL), flat even past 40K+ tokens in a session — decode wasn't the bottleneck I was chasing, but it stayed consistent throughout

- Ollama specifically had been running on Vulkan before this because ROCm wasn't cooperating — now runs native ROCm too

---

## Full launch config

For anyone trying to reproduce or compare against a similar setup, here's the exact `llama-server` command I run:

```bash

BASE="$HOME/Documents/Model LLM/Ornith-1.0-35B"

MODEL="$BASE/Ornith-1.0-35B-UD-IQ4_NL.gguf"

PORT=8082

~/Projects/llama2/build-rocm-test/bin/llama-server \

--model "$MODEL" \

--host 0.0.0.0 \

--port "$PORT" \

--n-gpu-layers 99 \

--threads 4 \

--threads-batch 4 \

--n-cpu-moe 22 \

--ctx-size 131072 \

--batch-size 1024 \

--ubatch-size 1024 \

--keep 20480 \

--cache-type-k q8_0 \

--cache-type-v q8_0 \

--swa-checkpoints 24 \

--checkpoint-min-step 2048 \

--embd-normalize 0 \

--no-kv-unified \

--kv-offload \

--jinja \

--reasoning-preserve \

--flash-attn on \

--parallel 1 \

--cache-ram 8192 \

--cache-idle-slots \

--temp 0.6 \

--top-k 20 \

--top-p 0.95 \

--min-p 0.1 \

--repeat-penalty 1.1 \

--repeat-last-n 512 \

--alias udinllama \

--log-verbosity 4

```

A few notes on the less obvious choices, in case they save someone else time:

- **`--n-cpu-moe 22`** — set based on how much fits in 12GB VRAM after everything else is accounted for, not an arbitrary number. Check your own VRAM headroom before copying this value directly.

- **`--threads 4` / `--threads-batch 4`** — this isn't a thermal-only choice. On my i5-11400F, the CPU-offloaded MoE compute is bandwidth-bound against dual-channel DDR4-3200, not core-count-bound. Above 4 threads I saw prefill get *worse*, not better — contention on memory bandwidth outweighs the parallelism gain. I tried 3 threads too (slightly faster on paper) but junction temps became unpredictable (spiking to 95°C occasionally vs a controlled 83°C peak at 4 threads), so I settled on 4 as the stable point, not just the fastest one on a spec sheet.

- **`--batch-size` / `--ubatch-size 1024`** — pushed to 1536/2048 in testing; didn't move prefill meaningfully but added ~10°C. Not worth it for this hardware.

- **`--cache-type-k q8_0` / `--cache-type-v q8_0`** — paired with `--kv-offload` to fit long context in 12GB VRAM with less quality loss than default f16 KV cache would cost in size.

- **`--no-kv-unified`** — only relevant because I run `--parallel 1` (single user, single active chat). If you're serving multiple concurrent sessions, you'll want KV unified on.

- **`--cache-ram 8192` / `--cache-idle-slots`** — this is a fallback safety net against OOM during inference, not something I expect to actually get fully utilized in normal use.

## Caveats / your mileage may vary (please read before trying this)

I want to be upfront about scope here, because I don't think this is a universal copy-paste fix:

1. **This was validated on Ubuntu 26.04 + TheRock's gfx1031 ROCm build specifically.** Other distros or ROCm sources may behave differently.
2. **llama.cpp commit drift is real.** I'm not confident this exact patch applies cleanly to every commit/fork out there. Some engines (Unsloth, in my case) auto-detect the llama.cpp commit and ROCm config already present and adapt; if that detection doesn't find a compatible match, it may fall back to Vulkan on its own.
3. **head_dim 512 is conditional, not universal.** It's tied to `amd_mfma_available(cc)` — meaning it only applies where the matrix-core path is relevant and where the tile kernel is actually hitting the shared memory limit this addresses. I wouldn't recommend hardcoding this blindly without checking whether your setup hits the same failure mode first.
4. **Hardware variance beyond just "same GPU model" matters.** Even identical GPU models can differ in memory bandwidth, clock offsets, and power limits depending on board partner/cooler design. My CPU/RAM (i5-11400F, 16GB DDR4 3200) also shapes some of the numbers above (especially decode speed, which on this setup is RAM-bandwidth-bound on the CPU-offloaded MoE portion).

**What I think *is* generalizable here isn't the number 512 — it's the method:** if you're on an AMD GPU architecture that's under-documented, using a known-stable backend (Vulkan, in my case) as a behavioral baseline, watching kernel activity in `btop` while running real workloads, and correlating that to the kernel dispatch logic in source, is a workable way to find your specific fix even without deep kernel-level theory knowledge.

Genuinely don't know why fixing this one file also fixed SGLang and vLLM, which don't share any code with llama.cpp/ggml. I was troubleshooting other ROCm-level things around the same time and didn't track every step carefully — so there's likely something else that got resolved in parallel that I can't identify. If anyone has insight into what else might explain this, I'd like to know

Happy to share exact configs (llama-server flags, ROCm build steps, benchmark logs) if anyone wants to try reproducing this on their own gfx1031 or adjacent-tier card. If anyone here actually understands the shared-memory-limit mechanics well enough to explain *why* this works, I'd genuinely appreciate the explanation — I found this by observation, not by theory.

Trellis.2 allows you to create 3D models from images and export their meshes: great for game world and character model prototyping. I couldn't find an existing fork that worked for my setup so I've been working on adding gfx103x + Windows support to the relevant Python packages: CuMesh, o_voxel (part of Trellis.2 repo), FlexGEMM. Of course, hardware support on RDNA2 is very limiting, but each of the packages passed a basic accuracy test and work well enough to produce results in pic above which took 245 seconds to complete. Not fast, but it's something.

I also ran into some issues on rocm-7.12 and toch-2.10 that were solved by upgrading to rocm-7.14 and torch-2.13. I did not need to patch torch-2.13 to get this working. Here are the pre-requisite wheels that I used, and they should work for almost any RDNA2 (gfx103x) user to run ComfyUI:

https://rocm.nightlies.amd.com/v2-staging/gfx103X-all/

rocm-7.14.0a20260612.tar.gz

rocm_sdk_core-7.14.0a20260612-py3-none-win_amd64.whl

rocm_sdk_devel-7.14.0a20260612-py3-none-win_amd64.whl

rocm_sdk_libraries_gfx103x_all-7.14.0a20260612-py3-none-win_amd64.whl

torch-2.13.0a0+rocm7.14.0a20260612-cp312-cp312-win_amd64.whl

torchaudio-2.11.0a0+rocm7.14.0a20260612-cp312-cp312-win_amd64.whl

torchvision-0.28.0a0+rocm7.14.0a20260612-cp312-cp312-win_amd64.whl

You'll also need my additional wheels (CuMesh fork, o_voxel fork, FlexGEMM fork) and nvdiffrast-hip to run Trellis.2. I'll provide those repos when I've finished testing my clean build + install scripts.

---

Edit: Important note @ 12:01am US EST 16th Aug 2026 - If you had downloaded a copy of my diffs before this time then please update them as soon as possible. A number of issues with MTP correctness were discovered both in the upstream mainline branch, as well as my own code, and a further decode correctness issue was uncovered for certain model sizes. These should all be fixed now. If anyone finds any further issues, just drop me a note and we'll try to get it sorted ASAP.

---

Qwen3.8-27B-Q8_0 on AMD Radeon AI Pro R9700

This post is a followup to my earlier post here: https://www.reddit.com/r/ROCm/s/87da5FHpPO

My goal was to make ROCm the clear and easy choice for which backend to run.

I've tried to label the graphs a little more clearly this time around after feedback from the other post.

Since that post I've managed to eke out a few more small gains, and decided to test and share my results against the new Qwen3.8-27B model

The purpose of this post is to compare and present graphs for the following:

• Comparison of Qwen3.8-27B-Q8_0 on a single R9700 with different backends

• Comparison of Qwen3.8-27B-Q8_0 on two R9700's with different backends

• Comparison of Qwen3.8-27B-Q8_0 on three R9700's with different backends

• Scaling Comparison of Qwen3.8-27B-Q8_0 on 1 vs 2 vs 3 R9700s with Vulkan

• Scaling Comparison of Qwen3.8-27B-Q8_0 on 1 vs 2 vs 3 R9700s with Standard ROCm

• Scaling Comparison of Qwen3.8-27B-Q8_0 on 1 vs 2 vs 3 R9700s with Tweaked ROCm

Environment

Test Hardware

CPU: AMD 9950X3D2

Motherboard: Asus ProArt X870E Creator Wifi

GPU: 3 x AMD AI Pro R9700 - All set to run in PCIe5x4 mode

Memory: 96GB DDR5 @ 6000MHz CL30

Two of the GPUs are in the PCIe5x16 slots, but set to PCIe5x4 mode.

The third card is installed via an M.2->PCIe5 riser, which also grants PCIe5x4 speeds

All PCIe lane bandwidths are measured and verified

Libraries Used

• ROCm executables are linked against AMD ROCm 7.14.0 libraries from here: https://rocm.docs.amd.com/en/latest/install/rocm.html
• Vulkan executables are linked against the default system RADV libraries installed by Linux Fedora 44

Tweaked ROCm source code

The source code branch for the tweaked ROCm changes is found here: https://github.com/stew675/llama.cpp/tree/rdna-boosts

Building ROCm tweaked llama.cpp from source

I'm going to trust that you already know how to use git to pull the source code and change to the source directory

My Cmake ROCm build setup flags are these. Note that you must set ROCM_714 to installation directory of your ROCm 7.14 SDK

# You MUST set this path correctly
ROCM_714=/path/to/your/rocm-7.14/installation

export ROCM_PATH="$ROCM_714"

HIPCXX="$ROCM_714/lib/llvm/bin/clang" HIP_PATH="$ROCM_714" cmake -S . -B build \
  -DGGML_RPC=1 \
  -DGGML_HIP=ON \
  -DGGML_NATIVE=1 \
  -DGGML_HIP_RCCL=1 \
  -DHIP_PLATFORM=amd \
  -DGGML_HIP_GRAPHS=ON \
  -DGPU_TARGETS="gfx1200,gfx1201" \
  -DCMAKE_C_COMPILER="$ROCM_714/lib/llvm/bin/clang" \
  -DAMDGPU_TARGETS="gfx1200,gfx1201" \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CXX_COMPILER="$ROCM_714/lib/llvm/bin/clang++" \
  -DCMAKE_INSTALL_RPATH="\$ORIGIN:$ROCM_714/lib" \
  -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
  -DCMAKE_HIP_FLAGS="-mllvm --amdgpu-unroll-threshold-local=600"

# Then build
cmake --build build --config Release -j 16 -- VERBOSE=1

llama-server setup

Aside from the differences for specific backend setup, ie. Vulkan vs ROCm vs Tweaked ROCm, the following configuration is common for all backends used:

Common Server Command: ./build/bin/llama-server --model /llm/models/Qwen3.8/27B/Q8_0/Qwen3.8-27B-Q8_0.gguf --alias Qwen3.8-27B-Q8_0 --prio 2 --fit false --top-k 20 --port 8033 --threads 8 --parallel 1 --top-p 0.8 --min-p 0.001 --verbosity 3 --host 0.0.0.0 --cpu-strict 1 --cpu-range 0-7 --predict 98304 --threads-http 4 --load-mode mlock --cache-ram 16384 --ctx-size 70000 --flash-attn auto --temperature 0.7 --batch-size 1024 --ubatch-size 1024 --n-gpu-layers all --no-kv-unified --cache-type-k bf16 --cache-type-v bf16 --ctx-checkpoints 64 --cache-idle-slots --reasoning-budget 65536 --reasoning-preserve --checkpoint-min-step 4096 --repeat-penalty 1.0 --presence-penalty 1.5

Benchmark command

Test Command: uvx llama-benchy --base-url http://localhost:8033/v1 --tg 240 --pp 2520 --model Qwen3.8-27B-Q8_0 --tokenizer Qwen/Qwen3.8-27B --depth 0 4096 8192 16384 32768 65536

KEY TAKEWAYS

• ROCm Tweaked outperforms both Vulkan and standard ROCm build in every scenario
• For Tweaked ROCm, the gain from 1 to 2 GPUs is roughly 60%
• For Tweaked ROCm, the gain from 2 to 3 GPUs is roughly 10%
• Unless you need the extra VRAM for context, two cards is best
• Vulkan gets worse with more than two cards
• Stock ROCm is just slow once context grows

RAW RESULTS

Qwen3.8-27B-Q8_0, KV=BF16, 1 x Radeon AI Pro R9700

Vulkan

model	test	t/s	ttfr (ms)
Qwen3.8-27B-Q8_0	pp2520	946.78 ± 6.80	2663.43 ± 19.13
Qwen3.8-27B-Q8_0	tg240	19.60 ± 0.01
Qwen3.8-27B-Q8_0	pp2520 @ d4096	986.24 ± 7.02	6710.27 ± 48.76
Qwen3.8-27B-Q8_0	tg240 @ d4096	19.44 ± 0.01
Qwen3.8-27B-Q8_0	pp2520 @ d8192	969.12 ± 2.88	11055.59 ± 33.38
Qwen3.8-27B-Q8_0	tg240 @ d8192	19.27 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d16384	920.24 ± 2.80	20543.26 ± 63.60
Qwen3.8-27B-Q8_0	tg240 @ d16384	19.08 ± 0.01
Qwen3.8-27B-Q8_0	pp2520 @ d32768	820.18 ± 2.67	43026.47 ± 140.70
Qwen3.8-27B-Q8_0	tg240 @ d32768	18.45 ± 0.01
Qwen3.8-27B-Q8_0	pp2520 @ d65536	666.92 ± 2.06	102047.27 ± 316.31
Qwen3.8-27B-Q8_0	tg240 @ d65536	17.34 ± 0.00

ROCm Standard

model	test	t/s	ttfr (ms)
Qwen3.8-27B-Q8_0	pp2520	1079.74 ± 8.61	2335.32 ± 18.63
Qwen3.8-27B-Q8_0	tg240	18.93 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d4096	1072.23 ± 7.80	6172.38 ± 45.39
Qwen3.8-27B-Q8_0	tg240 @ d4096	18.27 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d8192	1009.76 ± 2.89	10608.92 ± 29.33
Qwen3.8-27B-Q8_0	tg240 @ d8192	17.68 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d16384	896.40 ± 2.39	21090.34 ± 56.25
Qwen3.8-27B-Q8_0	tg240 @ d16384	16.39 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d32768	721.45 ± 1.72	48913.28 ± 118.16
Qwen3.8-27B-Q8_0	tg240 @ d32768	14.46 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d65536	520.91 ± 1.19	130652.51 ± 298.47
Qwen3.8-27B-Q8_0	tg240 @ d65536	11.70 ± 0.00

ROCm Tweaked

model	test	t/s	ttfr (ms)
Qwen3.8-27B-Q8_0	pp2520	1108.34 ± 12.55	2274.71 ± 25.31
Qwen3.8-27B-Q8_0	tg240	20.10 ± 0.02
Qwen3.8-27B-Q8_0	pp2520 @ d4096	1157.21 ± 8.47	5719.09 ± 42.31
Qwen3.8-27B-Q8_0	tg240 @ d4096	19.94 ± 0.03
Qwen3.8-27B-Q8_0	pp2520 @ d8192	1141.54 ± 3.73	9383.75 ± 31.09
Qwen3.8-27B-Q8_0	tg240 @ d8192	19.79 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d16384	1087.36 ± 3.68	17386.11 ± 58.35
Qwen3.8-27B-Q8_0	tg240 @ d16384	19.46 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d32768	969.84 ± 2.96	36385.90 ± 110.97
Qwen3.8-27B-Q8_0	tg240 @ d32768	18.84 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d65536	792.60 ± 2.77	85866.79 ± 299.56
Qwen3.8-27B-Q8_0	tg240 @ d65536	17.69 ± 0.00

Qwen3.8-27B-Q8_0, KV=BF16, 2 x Radeon AI Pro R9700

Vulkan split-mode=layer

model	test	t/s	ttfr (ms)
Qwen3.8-27B-Q8_0	pp2520	941.77 ± 6.36	2677.34 ± 18.08
Qwen3.8-27B-Q8_0	tg240	18.05 ± 0.02
Qwen3.8-27B-Q8_0	pp2520 @ d4096	1173.67 ± 11.23	5638.30 ± 54.39
Qwen3.8-27B-Q8_0	tg240 @ d4096	17.89 ± 0.01
Qwen3.8-27B-Q8_0	pp2520 @ d8192	1309.91 ± 4.56	8178.90 ± 28.44
Qwen3.8-27B-Q8_0	tg240 @ d8192	17.79 ± 0.01
Qwen3.8-27B-Q8_0	pp2520 @ d16384	1388.40 ± 3.96	13616.82 ± 38.79
Qwen3.8-27B-Q8_0	tg240 @ d16384	17.52 ± 0.03
Qwen3.8-27B-Q8_0	pp2520 @ d32768	1340.08 ± 2.63	26333.53 ± 51.22
Qwen3.8-27B-Q8_0	tg240 @ d32768	16.99 ± 0.01
Qwen3.8-27B-Q8_0	pp2520 @ d65536	1153.97 ± 6.27	58978.89 ± 321.04
Qwen3.8-27B-Q8_0	tg240 @ d65536	16.05 ± 0.01

ROCm Standard, split-mode=tensor

model	test	t/s	ttfr (ms)
Qwen3.8-27B-Q8_0	pp2520	1416.38 ± 23.48	1780.06 ± 28.80
Qwen3.8-27B-Q8_0	tg240	29.01 ± 0.05
Qwen3.8-27B-Q8_0	pp2520 @ d4096	1541.34 ± 23.71	4294.45 ± 66.06
Qwen3.8-27B-Q8_0	tg240 @ d4096	28.21 ± 0.07
Qwen3.8-27B-Q8_0	pp2520 @ d8192	1528.94 ± 9.33	7007.80 ± 42.43
Qwen3.8-27B-Q8_0	tg240 @ d8192	27.46 ± 0.04
Qwen3.8-27B-Q8_0	pp2520 @ d16384	1425.14 ± 7.16	13265.79 ± 66.99
Qwen3.8-27B-Q8_0	tg240 @ d16384	26.17 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d32768	1208.55 ± 4.46	29200.20 ± 107.67
Qwen3.8-27B-Q8_0	tg240 @ d32768	23.71 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d65536	909.59 ± 4.58	74823.96 ± 376.66
Qwen3.8-27B-Q8_0	tg240 @ d65536	19.86 ± 0.00

ROCm Tweaked, split-mode=tensor

model	test	t/s	ttfr (ms)
Qwen3.8-27B-Q8_0	pp2520	1457.92 ± 21.43	1729.87 ± 25.42
Qwen3.8-27B-Q8_0	tg240	31.24 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d4096	1641.60 ± 21.37	4031.51 ± 52.77
Qwen3.8-27B-Q8_0	tg240 @ d4096	30.79 ± 0.07
Qwen3.8-27B-Q8_0	pp2520 @ d8192	1674.79 ± 7.66	6397.07 ± 29.24
Qwen3.8-27B-Q8_0	tg240 @ d8192	30.52 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d16384	1655.24 ± 10.02	11421.77 ± 68.86
Qwen3.8-27B-Q8_0	tg240 @ d16384	30.11 ± 0.02
Qwen3.8-27B-Q8_0	pp2520 @ d32768	1541.65 ± 9.55	22890.95 ± 141.86
Qwen3.8-27B-Q8_0	tg240 @ d32768	29.36 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d65536	1303.84 ± 9.27	52200.88 ± 371.47
Qwen3.8-27B-Q8_0	tg240 @ d65536	27.95 ± 0.01

Qwen3.8-27B-Q8_0, KV=BF16, 3 x Radeon AI Pro R9700

Vulkan split-mode=layer

model	test	t/s	ttfr (ms)
Qwen3.8-27B-Q8_0	pp2520	936.08 ± 7.40	2693.75 ± 21.30
Qwen3.8-27B-Q8_0	tg240	16.26 ± 0.02
Qwen3.8-27B-Q8_0	pp2520 @ d4096	1111.70 ± 10.05	5953.08 ± 53.82
Qwen3.8-27B-Q8_0	tg240 @ d4096	16.13 ± 0.05
Qwen3.8-27B-Q8_0	pp2520 @ d8192	1162.28 ± 5.23	9217.87 ± 42.34
Qwen3.8-27B-Q8_0	tg240 @ d8192	16.13 ± 0.21
Qwen3.8-27B-Q8_0	pp2520 @ d16384	1161.92 ± 2.42	16270.07 ± 33.92
Qwen3.8-27B-Q8_0	tg240 @ d16384	15.13 ± 0.22
Qwen3.8-27B-Q8_0	pp2520 @ d32768	1071.11 ± 2.82	32947.18 ± 86.33
Qwen3.8-27B-Q8_0	tg240 @ d32768	14.62 ± 0.02
Qwen3.8-27B-Q8_0	pp2520 @ d65536	888.53 ± 4.50	76597.24 ± 388.46
Qwen3.8-27B-Q8_0	tg240 @ d65536	13.94 ± 0.02

ROCm Standard, split-mode=tensor

model	test	t/s	ttfr (ms)
Qwen3.8-27B-Q8_0	pp2520	1469.22 ± 28.82	1717.62 ± 33.00
Qwen3.8-27B-Q8_0	tg240	34.20 ± 0.09
Qwen3.8-27B-Q8_0	pp2520 @ d4096	1653.39 ± 28.81	4003.70 ± 69.76
Qwen3.8-27B-Q8_0	tg240 @ d4096	32.24 ± 0.06
Qwen3.8-27B-Q8_0	pp2520 @ d8192	1628.31 ± 22.93	6580.95 ± 93.27
Qwen3.8-27B-Q8_0	tg240 @ d8192	30.48 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d16384	1522.01 ± 14.99	12422.69 ± 122.34
Qwen3.8-27B-Q8_0	tg240 @ d16384	28.86 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d32768	1266.96 ± 7.91	27855.14 ± 174.19
Qwen3.8-27B-Q8_0	tg240 @ d32768	25.96 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d65536	917.55 ± 7.92	74178.18 ± 639.89
Qwen3.8-27B-Q8_0	tg240 @ d65536	21.49 ± 0.01

ROCm Tweaked, split-mode=tensor

model	test	t/s	ttfr (ms)
Qwen3.8-27B-Q8_0	pp2520	1494.97 ± 3.91	1686.30 ± 4.75
Qwen3.8-27B-Q8_0	tg240	37.30 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d4096	1745.76 ± 31.97	3792.19 ± 69.15
Qwen3.8-27B-Q8_0	tg240 @ d4096	35.71 ± 0.10
Qwen3.8-27B-Q8_0	pp2520 @ d8192	1787.03 ± 11.47	5995.43 ± 38.47
Qwen3.8-27B-Q8_0	tg240 @ d8192	34.35 ± 0.08
Qwen3.8-27B-Q8_0	pp2520 @ d16384	1780.61 ± 15.12	10617.95 ± 90.45
Qwen3.8-27B-Q8_0	tg240 @ d16384	33.89 ± 0.00
Qwen3.8-27B-Q8_0	pp2520 @ d32768	1660.52 ± 8.73	21252.42 ± 111.99
Qwen3.8-27B-Q8_0	tg240 @ d32768	32.84 ± 0.02
Qwen3.8-27B-Q8_0	pp2520 @ d65536	1374.56 ± 15.54	49518.77 ± 560.02
Qwen3.8-27B-Q8_0	tg240 @ d65536	31.09 ± 0.03

Hi,

I checked this blog post which mentions ~50 tokens/s for Qwen3.8 27B on single R9700.

I checked huggingface but no specific instructions for running it on RCom.

https://www.amd.com/en/blogs/2026/run-qwen-3-8-27b-on-amd-ryzen-ai-max-and-radeon-graphics-cards-day-0.html

Is there any specific documentation for running it using llama.cpp or vllm ? All the craze is only for Nvidia GPUs :⁠-⁠(

---------------------------------
Edit: Important note @ 12:01am US EST 16th Aug 2026 - If you had downloaded a copy of my diffs before this time then please update them as soon as possible. A number of issues with MTP correctness were discovered both in the upstream mainline branch, as well as my own code, and a further decode correctness issue was uncovered for certain model sizes. These should all be addressed now. If anyone finds any further issues, just drop me a note and we'll try to get it sorted ASAP.

---------------------------------

With a lot of new model releases lately and Qwen3.8-27B right around the corner, I wanted to share a set of ROCm/RDNA focused speed up I've been working on. These largely exist as a set of diffs that can be applied cleanly on top of upstream llama.cpp, and my goal was to get ROCm up to parity with Vulkan, or as close as possible.

I have a 7900XTX, Strix Halo, and a machine with 3 x R9700's GPUs in it, so I've been testing my work against all three architectures. I want to share a bunch of graphs for the R9700's which shows the relative speedups achieved.

My git repo branch is here: https://github.com/stew675/llama.cpp/tree/rdna-boosts

I tested Qwen3.6-27B, Muse Glimmer 30B, and Gemma4-31B. For single cards I used the Q6_K quants of these models (except for Glimmer where Q8_0 did fit well), and for 2 or 3 cards I used the Q8_0 model weights.

One element of my changes is I targetted native BF16 support for the KV cache. I know that it came as a surprise to me that standard ROCm support in llama.cpp just maps BF16 to F16 for the KV cache, and this mapping both slows things down, AND gives zero precision benefit. There was literally no good reason to choose BF16 for KV cache, ever, with ROCm. The curious thing though is that for Vulkan, it DOES implement BF16 KV cache correctly.

As such, the graphed runs below are all with a BF16 KV cache, which is hardware supported, at least for the important vector dot product operations, on anything RDNA3+. Scalar BF16 support isn't properly there, but that doesn't matter here.

My test system runs Fedora44. My build sequence for standard Fedora ROCm is:

HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" cmake -S . -B build \
  -DGGML_RPC=1 \
  -DGGML_HIP=ON \
  -DGGML_NATIVE=1 \
  -DGGML_HIP_RCCL=1 \
  -DHIP_PLATFORM=amd \
  -DGGML_HIP_GRAPHS=ON \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_HIP_ROCWMMA_FATTN=OFF \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DGPU_TARGETS="gfx1200;gfx1201" \
  -DCMAKE_INSTALL_RPATH="\$ORIGIN" \
  -DAMDGPU_TARGETS="gfx1200;gfx1201" \
  -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
  -DCMAKE_HIP_FLAGS="-mllvm --amdgpu-unroll-threshold-local=600"

cmake --build build --config Release -j 16 -- VERBOSE=1

and if you have ROCm 7.14 installed, it's this:

ROCM_714=/opt/rocm-7.14-gfx1201

[ -d "$ROCM_714" ] || { echo "error: $ROCM_714 not found" >&2; exit 1; }


export ROCM_PATH="$ROCM_714"

HIPCXX="$ROCM_714/lib/llvm/bin/clang" HIP_PATH="$ROCM_714" cmake -S . -B build \
 -DGGML_RPC=1 \
 -DGGML_HIP=ON \
 -DGGML_NATIVE=1 \
 -DGGML_HIP_RCCL=1 \
 -DHIP_PLATFORM=amd \
 -DGGML_HIP_GRAPHS=ON \
 -DGPU_TARGETS="gfx1200,gfx1201" \
 -DCMAKE_C_COMPILER="$ROCM_714/lib/llvm/bin/clang" \
 -DAMDGPU_TARGETS="gfx1200,gfx1201" \
 -DCMAKE_BUILD_TYPE=Release \
 -DCMAKE_CXX_COMPILER="$ROCM_714/lib/llvm/bin/clang++" \
 -DCMAKE_INSTALL_RPATH="\$ORIGIN:$ROCM_714/lib" \
 -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
 -DCMAKE_HIP_FLAGS="-mllvm --amdgpu-unroll-threshold-local=600"

cmake --build build --config Release -j 16 -- VERBOSE=1

All tests were done using llama-benchy against a live llama-server instance. I chose a generation size of 240 as that seems to align fairly closely with the typical tool-call generation size, and I chose a prefill size of 2520 to be representative of typical agentic coding input sizes.

As such, my test command invocation looks like so:

uvx llama-benchy --base-url http://localhost:8033/v1 --tg 240 --pp 2520 --model Muse-Glimmer-30B-Q8_0  --tokenizer meta-models/Muse-Glimmer-30B --depth 0 4096 8192 16384 32768 65536

I have a PR against upstream llama.cpp for some of this work, but it seems to be getting ignored, likely because it doing changes against the CUDA kernels, and so they (probably rightfully) don't trust some random yahoo like myself to be messing with that stuff. Still, I don't want to let that get in the way of sharing these tweaks with the ROCm community.

I hope people find this stuff to be useful. The speeds do apply to all 7xxx series cards, and the Strix Point/Halo architectures, but some speedups were only viable on RDNA4, so your results may be mixed depending on what you're using.