**TL;DR:** Running a 35B A3B model with CPU/GPU split on a 12GB card means prefill/TTFT at large context (70K+) is what actually makes or breaks usability — not decode. Vulkan was fine at 16-32K context but became painful past that. Got llama.cpp running fully native on ROCm for gfx1031 (RX 6700XT) instead — flash attention on, quantized KV cache, no `HSA_OVERRIDE_GFX_VERSION` hacks, no core dumps, prefill peaking around 580 tok/s even with 20K+ tokens already cached. The fix ended up being a one-line change in `ggml/src/ggml-cuda/fattn.cu` forcing the MMA kernel path for a specific head_dim instead of the default tile kernel. Once this was fixed at the ROCm/HIPBLAS level, every other inference engine I tested (Ollama, Unsloth, LM Studio, SGLang, vLLM) also started working natively — this wasn't a llama.cpp-only fix, it was a root-level fix for the whole ROCm stack on this GPU.

I'm not a developer. No CS/programming background — I'm a creative director by trade (photography/cinematography) who's been self-teaching local LLM inference for a few months. So this write-up is going to read more like "how I diagnosed this empirically" than "here's the technical theory of why it works." If anyone with deeper kernel/ROCm knowledge can fill in the *why*, I'd genuinely love to learn it.

---

## The problem

gfx1031 sits in an awkward spot: it's RDNA2, but not gfx1030 (the RX 6800/6900 series, which has the most complete official ROCm support and by far the most community documentation). Most guides, fixes, and reference configs you'll find online target gfx1030. Applying them as-is to a 6700XT either silently falls back to worse behavior, or crashes outright.

The common workaround people recommend is `HSA_OVERRIDE_GFX_VERSION=10.3.0` (telling the runtime to treat your gfx1031 card as gfx1030). I tried this. Here's what happened:

- Flash attention wouldn't enable

- Some ops (noticed it specifically with top_k) silently fell back to CPU, tanking decode speed

- Random core dumps mid-inference, not tied to any consistent trigger

I made the call to require **native gfx1031 recognition, no override, no spoofing the runtime**. That decision alone cost me a lot of trial and error, but I think it was the right one — see below why.

## Why I bothered (this isn't about decode speed)

I want to be specific about this because I think it's the part most people miss. If you're running a model that fits entirely in VRAM, ROCm being suboptimal (or partial CPU fallback on certain ops) doesn't hurt you that much — most of the compute is already on GPU anyway.

My case is different: **35B A3B with CPU/GPU split** (`--n-cpu-moe 22`, dense/attention layers on GPU, part of the MoE experts on CPU). In this setup, prefill isn't pure GPU compute — a portion of it depends on the CPU-offloaded expert path too. When the backend isn't efficient there, and context climbs into the 70-80K+ range, TTFT doesn't degrade linearly — it gets genuinely painful. At 16-32K context, Vulkan's prefill (roughly 70-100 tok/s in my case) was still tolerable. Past that, it wasn't.

The other piece of motivation: RX 6700XT's boost clock sits dramatically above an RTX 3060 12GB (2581MHz vs 1777MHz, roughly +45%), while VRAM bandwidth is only modestly higher (~384GB/s vs ~360GB/s, about +7%). That distinction matters here: decode is largely bandwidth-bound (reading/writing KV cache and weights per token), so that 7% bandwidth gap barely moves the needle on decode speed. Prefill, on the other hand, leans more on raw compute throughput — which is exactly where the 45% clock gap should show up. I'd read threads of people running 35B A3B with CPU/GPU split on a 3060 hitting peak prefill around ~300 tok/s. Given the clock gap, I'd expect the 6700XT to clear that by a wide margin, not just edge past it. Getting nowhere close to that on Vulkan was the signal that I was leaving real performance on the table — a software bottleneck, not a hardware ceiling — and that gap is what pushed me to actually fix this instead of settling.

After the fix, here's an actual log excerpt from a real session (task with context already at ~18-23K tokens, prompt processing in progress):

```

prompt processing, n_tokens = 2048, progress = 0.55, t = 3.52s / 581.81 tokens per second

prompt processing, n_tokens = 3072, progress = 0.58, t = 5.60s / 548.24 tokens per second

prompt processing, n_tokens = 4096, progress = 0.60, t = 7.71s / 531.24 tokens per second

prompt processing, n_tokens = 5120, progress = 0.63, t = 10.07s / 508.61 tokens per second

prompt processing, n_tokens = 6144, progress = 0.66, t = 12.24s / 502.11 tokens per second

```

Peak prefill of ~580 tok/s, staying above 500 tok/s while cached context was already north of 20K tokens. That's the number that mattered to me — not decode.

## Environment

- OS: Ubuntu Desktop 26.04 LTS

- GPU: AMD RX 6700XT 12GB (gfx1031, RDNA2)

- CPU: Intel i5-11400F

- RAM: 16GB DDR4 3200MT/s

- ROCm: custom build from TheRock binaries (official gfx1031 support isn't guaranteed across all standard ROCm releases)

- Inference engine: llama.cpp (build-rocm), Ollama, Unsloth, LM Studio, SGLang, vLLM

**A specific version note, since ROCm/TheRock builds vary a lot:** don't install just any TheRock ROCm build and expect this to match. The exact version I validated this on:

```

$ hipcc --version

HIP version: 7.14.60850-0000000

AMD clang version 23.0.0git (https://github.com/ROCm/llvm-project.git 46fcb339fb61119b337f973c7ca9e710a319fdd0+PATCHED:440716f8b87be9d8e20ed910e10e5b6d14d57cf6)

Target: x86_64-unknown-linux-gnu

Thread model: posix

InstalledDir: /opt/rocm/core-7.14/lib/llvm/bin

```

If you're on a different HIP/ROCm build, expect some variance — flag this version when comparing notes so we're not chasing different behavior across different builds.

**llama.cpp build/commit used:**

```

$ llama-server --version

version: 10307 (fc3f10b38)

built with GNU 15.2.0 for Linux x86_64

```

Given how much commit drift there is upstream (see caveats below), this matters if you're trying to reproduce the fix exactly — the `fattn.cu` logic may have shifted on a different commit.

**Confirmation ROCm actually sees this as gfx1031 (not spoofed via override):**

```

$ rocminfo | grep -i gfx

Name: gfx1031

Name: amdgcn-amd-amdhsa--gfx1031

Name: amdgcn-amd-amdhsa--gfx10-3-generic

```

**Exact card / board partner** (in case behavior varies by AIB):

```

$ lspci -vv | grep -A 2 "VGA"

03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M/6850M XT] (rev c1)

Subsystem: Sapphire Technology Limited Radeon RX 6700 XT GAMING OC 12G [Sapphire PULSE]

```

**I'll drop screenshots in the comments** as proof this is actually running native ROCm (not Vulkan silently, not CPU fallback) — GPU load/clock from a monitoring tool like `btop`/`lact`, prefill and decode averages from actual server logs, and junction temperature under sustained load, similar to what I used to validate stability throughout this whole process.

You can empirically look at my data.

1. ROCm natively
2. Total token when i use in just that runtime.
3. Temperatur edge, junction, and VRAM when prefill bound and decode bound.
4. Power usage both (prefill, decode).
5. BTOP verified some layers in cpu.

## The debugging process (the messy, real version)

Build kept failing. Tried the documented head_dim config for RDNA2 — failed. Tried random power-of-2 values (1024, 2048, 4096) as a guess — failed on all of them.

At that point I stopped touching the keyboard, made coffee, and just watched the failing terminal output for a while. Then it occurred to me to check `btop` instead of guessing blind. I ran the model through **Vulkan** (which was stable on this GPU, just slow) as a known-good baseline, watched which kernel showed up in btop tied to the running process, and correlated that behavior back to the head_dim condition in `fattn.cu`.

That's where the number `512` came from — not from any documentation I could find (I genuinely don't remember a source for it), but from matching observed hardware behavior to the kernel dispatch logic in source.

## The fix

In `ggml/src/ggml-cuda/fattn.cu`:

```cpp

// Force MMA kernel for head_dim 512 on AMD to avoid tile kernel shared memory limit

if (amd_mfma_available(cc) && Q->ne[0] == 512) {

return BEST_FATTN_KERNEL_MMA_F16;

}

return BEST_FATTN_KERNEL_TILE;

```

The default tile kernel path was hitting a shared memory limit on this hardware under certain conditions. Forcing the MMA (matrix-core) kernel path at this specific head_dim avoided that entirely.

**Why this seems to matter for the whole chain:** flash attention needs to be stable first before KV cache quantization is viable, and KV cache quantization is what makes long context (I run up to 131072 ctx) actually usable on 12GB VRAM. So this one fix unblocked flash-attn → which unblocked KV quant → which unblocked long context. If flash-attn doesn't work, none of the rest follows.

## Result

- Prefill peaking around ~580 tok/s (measured with 20K+ tokens already cached, not a cold-start number), staying in the 500-580 tok/s range across that phase — this is the number that actually matters for my use case, since TTFT at large context is what was hurting on Vulkan

- Native ROCm/HIPBLAS/ROCBLAS, no override, across every inference engine I tested — not just llama.cpp

- Flash attention stable, KV cache running q8_0, ctx up to 131072 confirmed stable

- Decode holds steady around ~22-23 tok/s on a 35B A3B model (IQ4_NL), flat even past 40K+ tokens in a session — decode wasn't the bottleneck I was chasing, but it stayed consistent throughout

- Ollama specifically had been running on Vulkan before this because ROCm wasn't cooperating — now runs native ROCm too

---

## Full launch config

For anyone trying to reproduce or compare against a similar setup, here's the exact `llama-server` command I run:

```bash

BASE="$HOME/Documents/Model LLM/Ornith-1.0-35B"

MODEL="$BASE/Ornith-1.0-35B-UD-IQ4_NL.gguf"

PORT=8082

~/Projects/llama2/build-rocm-test/bin/llama-server \

--model "$MODEL" \

--host 0.0.0.0 \

--port "$PORT" \

--n-gpu-layers 99 \

--threads 4 \

--threads-batch 4 \

--n-cpu-moe 22 \

--ctx-size 131072 \

--batch-size 1024 \

--ubatch-size 1024 \

--keep 20480 \

--cache-type-k q8_0 \

--cache-type-v q8_0 \

--swa-checkpoints 24 \

--checkpoint-min-step 2048 \

--embd-normalize 0 \

--no-kv-unified \

--kv-offload \

--jinja \

--reasoning-preserve \

--flash-attn on \

--parallel 1 \

--cache-ram 8192 \

--cache-idle-slots \

--temp 0.6 \

--top-k 20 \

--top-p 0.95 \

--min-p 0.1 \

--repeat-penalty 1.1 \

--repeat-last-n 512 \

--alias udinllama \

--log-verbosity 4

```

A few notes on the less obvious choices, in case they save someone else time:

- **`--n-cpu-moe 22`** — set based on how much fits in 12GB VRAM after everything else is accounted for, not an arbitrary number. Check your own VRAM headroom before copying this value directly.

- **`--threads 4` / `--threads-batch 4`** — this isn't a thermal-only choice. On my i5-11400F, the CPU-offloaded MoE compute is bandwidth-bound against dual-channel DDR4-3200, not core-count-bound. Above 4 threads I saw prefill get *worse*, not better — contention on memory bandwidth outweighs the parallelism gain. I tried 3 threads too (slightly faster on paper) but junction temps became unpredictable (spiking to 95°C occasionally vs a controlled 83°C peak at 4 threads), so I settled on 4 as the stable point, not just the fastest one on a spec sheet.

- **`--batch-size` / `--ubatch-size 1024`** — pushed to 1536/2048 in testing; didn't move prefill meaningfully but added ~10°C. Not worth it for this hardware.

- **`--cache-type-k q8_0` / `--cache-type-v q8_0`** — paired with `--kv-offload` to fit long context in 12GB VRAM with less quality loss than default f16 KV cache would cost in size.

- **`--no-kv-unified`** — only relevant because I run `--parallel 1` (single user, single active chat). If you're serving multiple concurrent sessions, you'll want KV unified on.

- **`--cache-ram 8192` / `--cache-idle-slots`** — this is a fallback safety net against OOM during inference, not something I expect to actually get fully utilized in normal use.

## Caveats / your mileage may vary (please read before trying this)

I want to be upfront about scope here, because I don't think this is a universal copy-paste fix:

1. **This was validated on Ubuntu 26.04 + TheRock's gfx1031 ROCm build specifically.** Other distros or ROCm sources may behave differently.
2. **llama.cpp commit drift is real.** I'm not confident this exact patch applies cleanly to every commit/fork out there. Some engines (Unsloth, in my case) auto-detect the llama.cpp commit and ROCm config already present and adapt; if that detection doesn't find a compatible match, it may fall back to Vulkan on its own.
3. **head_dim 512 is conditional, not universal.** It's tied to `amd_mfma_available(cc)` — meaning it only applies where the matrix-core path is relevant and where the tile kernel is actually hitting the shared memory limit this addresses. I wouldn't recommend hardcoding this blindly without checking whether your setup hits the same failure mode first.
4. **Hardware variance beyond just "same GPU model" matters.** Even identical GPU models can differ in memory bandwidth, clock offsets, and power limits depending on board partner/cooler design. My CPU/RAM (i5-11400F, 16GB DDR4 3200) also shapes some of the numbers above (especially decode speed, which on this setup is RAM-bandwidth-bound on the CPU-offloaded MoE portion).

**What I think *is* generalizable here isn't the number 512 — it's the method:** if you're on an AMD GPU architecture that's under-documented, using a known-stable backend (Vulkan, in my case) as a behavioral baseline, watching kernel activity in `btop` while running real workloads, and correlating that to the kernel dispatch logic in source, is a workable way to find your specific fix even without deep kernel-level theory knowledge.

Genuinely don't know why fixing this one file also fixed SGLang and vLLM, which don't share any code with llama.cpp/ggml. I was troubleshooting other ROCm-level things around the same time and didn't track every step carefully — so there's likely something else that got resolved in parallel that I can't identify. If anyone has insight into what else might explain this, I'd like to know

Happy to share exact configs (llama-server flags, ROCm build steps, benchmark logs) if anyone wants to try reproducing this on their own gfx1031 or adjacent-tier card. If anyone here actually understands the shared-memory-limit mechanics well enough to explain *why* this works, I'd genuinely appreciate the explanation — I found this by observation, not by theory.

If you have been using Qwen 3.8 27B lately, you must have observed that it thinks a lot, that is because it supports thinking effort embedded in its chat template and by default it is "xHigh". So I added a reasoning effort slider just like claude in TurboLLM so you can control it. For all other models it stays the old "Thinking Budget" where you can control number of tokens allowed for thinking. Go give it a try.

npx turbollm

I cannot fathom how this happened.

I am a solo game dev. Yesterday, a player on my Discord was asking the Google Search AI silly questions about my game, to laugh at some of the answers. But the answers sometimes contained surprisingly obscure info that should not be available for a small indie game like mine.

Out of curiosity, I chimed in and asked the player to ask the AI about future contents in my game. Somehow, the AI spitted out the exact & highly specific character name "Vantage Tripod", which I had never mentioned to anyone. As far as I know, the only place where the info exists in a digital form is inside one of my own Google Docs, and this was NOT me speaking to the AI! (The player even asked the questions in incognito).

My game is not big enough for there to be much noise out there to confuse Google's AI, so I guess that's why it managed to give actual, scarily real leaks on my game (without being confused by online speculations), and I have no idea how it knew all this.

The screenshots only included snippets of the conversation. This is the full conversation on Discord, in case you want to get your own opinion on what happened (The entire conversation with the player lasted for an hour, where the AI gave other upcoming content info such as things I'd only ever mentioned on Discord).

 

-----------------------------------------------
Edit: kallico from Discord has been sharing comparisons between her private GDD (never made into anything) and Google's responses when she asked for ideas based on her game title. Hers is a much more reproducible case where the responses matched her GDD to a concerning degree across multiple clean contexts, and did not feel like reasonable things an AI could have inferred from her prompt. When she tried the same questions with Claude (which shouldn't have access to Google Docs), the responses were far more reasonable and significantly less similar to the contents of her doc.

Hey everyone,

After ~5 days of tuning and ~$1.2k in compute I released a specialized Neuron-pruned IQ1_S quant of Kimi K3.

Key details:

- Size: ~308GB (vs the common ~594GB baseline)

- Every one of the 82,432 routed experts is present — no experts dropped

- HumanEval: 94.5% (matches full model 1:1)

- AIME: 92.5% (full ~96.1%)

- GSM8K: 95%

- MMLU: 79.49% (full ~85%)

- Throughput: 12.5492 tok/s average on 3× DGX Sparks (SparkInfer TP3 + custom speculative decoding patches; from ~2 t/s baseline)

This is my work (self-promo disclosure).

Links:

- HF (gated, request access): https://huggingface.co/vcruz305/Kimi-K3-GGUF

- SparkInfer patches + TP3 recipe + one-command deployment + benchmark receipts: https://github.com/vcruz305/kimi-k3-neuron-tp3-dgxspark-recipe

- Full announcement thread with more details: https://x.com/ViC305/status/2087609292442751209

Hardware notes: The reported 12.5 t/s is on 3 DGX Sparks via SparkInfer + my speculative decoding patches. It also has a llama.cpp fallback path. Optimized for multi-GPU / Blackwell-class setups.

If you have 3 Sparks (or 4× Blackwell / equivalent), please try it and share logs/results. Happy to answer questions on the neuron pruning, calibration, or multi-node setup. Feedback and PRs welcome!

We are AI researchers in the Denver area. We made an agentic MCP server so the AI chat bots can easily have updated fire data for anyone asking ChatGPT etc. or point any apps you are building to this server:

URLfirewatch-mcp.lovable.app

Hi r/LocalLLaMA,

Inspired by Reiner Pope's (MatX CEO, ex-Google TPU architect) whiteboard lecture, I built a serverless, interactive simulator to visualize LLM inference physics and KV cache economics.

👉 GitHub Repository: https://github.com/zhchin/llm_infra_visualizer

(It's pure HTML/JS. No server, no tracking, local-storage safe for your API keys.)

🛠️ Key Features:

• Interactive Roofline Model: Dynamically charts when your serving transitions from Memory-bandwidth bound (decoding) to Compute-bound (prefill).
• Automatic GPU Sharding: Input your model size/context, and it calculates the required Tensor Parallelism (TP-1 to TP-8) configurations for Blackwell, H100, A100, etc.
• MoE vs Dense Visualizer: Staggered purple wave animations for MoE routing bottlenecks vs synchronized cyan pulses for Dense models.
• KV Cache Economics: Compares real-world rental costs of keeping KV caches in HBM vs offloading to DDR/SSD vs Recomputation.
• AI Agent UI Control: Ask the built-in chatbot to "change batch size to 512" or "switch to MoE collapse scenario", and it will slide the UI knobs in real-time.

Check it out and let me know what you think! If it helps you size your deployments, please drop a ⭐ on GitHub!