I posted here about ToshLLM roughly two months ago, when I had just gotten local LLM inference working properly on Intel Macs with AMD GPUs by patching llama.cpp's Metal backend and writing an AMD-specific Flash Attention path.

A lot has happened since then.

What started as a way to make my RX 6700 XT useful for local inference has grown into a much broader AMD Metal optimization project, and one of the most interesting things has been seeing other people test it on hardware I don't own: W6900X setups, W6800X Duo, Vega II Duo, W5700X, Vega, Polaris, eGPUs, and multi-GPU Mac Pros.

The AMD backend itself has changed substantially.

Vega, Radeon VII, RX 400/500 and Radeon Pro Vega/WX

These cards now have dedicated Wave64 tuning instead of inheriting settings measured on newer RDNA cards.

On a Vega 64, depending on the model and quantization:

• generation improved by roughly 4-24%
• multi-conversation serving improved by 9-12%
• short cached follow-ups can be up to 39% faster
• MTP gained about 8%
• long-prompt processing also improved
• MoE models can experimentally use the wider matrix path previously used only on newer Radeon cards

Some measured examples:

Workload	Before	Current
9B Q5_K_M generation	31.2 t/s	38.8 t/s
8B Q6_K generation	35.5 t/s	41.4 t/s
7B / 64-expert MoE	136.0 t/s	155.3 t/s

The goal here isn't just "make it run" anymore. I'm increasingly tuning kernels separately for the different AMD generations.

RDNA has kept getting faster too

A number of quantization and model-specific paths have been optimized.

For example:

• IQ2_M generation on a 1.5B model: 92.1 -> 140.8 t/s
• Qwen3.6 14B-A3B: 62.2 -> 70.8 t/s
• Qwen3.5 4B + MTP: 70.0 -> 81.6 t/s
• Qwen3.5 4B + DFlash: 68.9 -> 82.6 t/s
• gpt-oss-20B prompt processing: 1093 -> 1292 t/s

Q4_K, Q5_K, Q6_K, q5_0/q5_1 and several other paths have received AMD-specific kernel work as well.

ToshLLM is becoming useful as a server, not just a chat app

Concurrent inference has improved quite a bit.

On a 9B Q5_K_M model:

• 2 simultaneous conversations: 56.4 -> 70.8 t/s
• 3 simultaneous conversations: 60.5 -> 81.0 t/s

There is now a browser UI based on llama.cpp's web interface, router mode can switch between models without restarting the server, multiple servers can run simultaneously, and there is a dedicated embeddings server for local RAG.

The API remains OpenAI-compatible, and I've started seeing people use ToshLLM's bundled llama-server as a backend for other software rather than only using the SwiftUI application. That was not something I expected when I started this.

Image generation has become much lighter on VRAM

This has been another area I've been optimizing.

For example, on my RX 6700 XT:

• Z-Image 1600x900: 8.2 GB -> 966 MB VRAM
• Z-Image 1024x1024: 2.4 GB -> 690 MB
• image generation is roughly 12-14% faster
• SD 1.5 768x768: 2690 MB -> 281 MB
• SD 1.5 768x768: 149 sec -> 73 sec

SD 1.5 is now particularly interesting for older 4-6 GB Radeon cards.

There is also experimental local video generation with Wan models now. It has improved substantially and Wan 2.2 5B can run in 12 GB VRAM, but I want to be clear that I still consider the video side experimental and it needs more work before I put it in the same category as LLM inference.

Community testing has become a big part of the project

This is probably the part I've enjoyed most.

People are now submitting benchmarks from configurations such as dual W6900X, W6800X Duo, W5700X, Vega and other machines I could never test myself.

There is a public benchmark database at:

https://toshllm.com/benchmarks

ToshLLM itself is still completely free and GPL-3.0.

No account, no telemetry, no cloud inference. The OpenAI-compatible server, chat, image generation, benchmarks, RAG/embeddings, etc. all run locally.

It's still pre-1.0 and the current DMGs are not notarized yet, so macOS requires Open Anyway on first launch. Notarization is something I'm working toward.

If anyone here still has a Mac Pro full of Radeon hardware, I'm especially interested in results from:

• W5700X
• W6800X / W6800X Duo
• W6900X
• Vega II / Vega II Duo
• Radeon VII
• multi-MPX configurations

At this point some of these old Intel Mac Pros have absurd amounts of usable VRAM for local AI, and I'm curious how far we can push them.

ToshLLM: https://toshllm.com

Source / releases: https://github.com/engeldlgado/toshllm

If you’re still using an Intel Mac with a discrete AMD GPU, you probably already know most local AI tools are built around Apple Silicon.

I built ToshLLM to make this hardware genuinely useful for local AI on macOS.

It started as a native app for running LLMs locally on Intel Macs with AMD GPUs using Metal, but it has grown a lot since the first public builds.

Recent highlights:

• Much faster prompt processing
• A big speedup in long conversations
• PDF chat with on-device OCR
• Experimental vision support
• Multiple servers at once
• Real GPU selection and experimental multi-GPU support
• Per-GPU VRAM monitoring
• Experimental local image generation inside the app
• Fully local, no cloud, no accounts

It’s still in development, but it’s already very usable on supported setups.

If anyone here is running a Mac Pro 2019 or another Intel Mac with Radeon hardware like RX 5000 series / 6600 / 6700 XT / 6800 / 6900 XT, I’d love test reports.

GitHub:

https://github.com/engeldlgado/toshllm

If you have an Intel Mac with an AMD GPU, you probably already know how frustrating local AI on macOS can be. Most tools are built around Apple Silicon, and on this kind of hardware llama.cpp under Metal usually does one of two things: it either falls back to CPU and becomes painfully slow, or it gives you corrupted output.

I ran into exactly that, so I started digging through llama.cpp’s Metal backend to understand what was actually breaking. That eventually turned into ToshLLM: a fully native SwiftUI app built specifically to make local LLMs usable on Intel Macs with AMD GPUs.

The main issues came down to two things. First, driver concurrency on these GPUs can cause race conditions and garbage text, so it has to be disabled. Second, standard Flash Attention depends on Apple Silicon-specific hardware support, which means compressed KV cache paths can silently fall back to CPU and destroy performance.

To fix that, I wrote a custom Metal Flash Attention kernel from scratch for AMD. That keeps both prefill and decode on the GPU instead of collapsing back to the CPU. On my RDNA 2 card, prompt processing is now about 8x faster than stock Metal. With an 8B model and compressed KV cache, performance went from 19 t/s to 33 t/s at 4k context, and it still holds around 22 t/s at 16k without falling apart.

I wrapped all of that into ToshLLM, a clean native app with zero external dependencies. It includes a patched llama.cpp backend, an experimental TurboQuant engine for much larger context windows on limited VRAM, a native chat UI, model VRAM estimates, Hugging Face search and download, MoE auto-tuning, Vision models compatible, and an OpenAI-compatible server.

It’s free, open source under GPL-3.0, and has no telemetry.

I’d especially love benchmark reports from anyone using AMD cards like RDNA 1, RDNA 2, Vega, or Polaris. Vega and Polaris are currently having some issues, but I’m working on support for them and hope to make them compatible as soon as possible.

GitHub repo and releases

Intel Macs with discrete AMD GPUs (the Radeon Pro / Vega / RDNA cards in Mac Pros, iMacs, and older 16-inch MacBook Pros) are a dead zone for local inference. Everything targets Apple Silicon now, and on these machines the tools either fall back to the CPU or hit the GPU and misbehave. I spent a while in llama.cpp's Metal backend figuring out why, and ended up building an app around the fixes.

The short version of why, since that's the interesting part:

• Metal command-buffer concurrency has to be disabled or the output is corrupted garbage on these GPUs. The simdgroup reductions race.
• Memory-mapping the model is about 6x slower than loading it resident (PCIe-bound), so you have to force no-mmap.
• Flash Attention is gated on simdgroup matrix-multiply support (the Apple7 family check) which these GPUs report as unavailable. So the moment you use a quantized or compressed KV cache, which requires FA, attention silently falls back to the CPU and generation falls off a cliff as context grows. The upstream "vec" FA kernel that doesn't need matrix units also miscompiles on RDNA under Metal (correct in isolation, garbage in the full kernel).

So I wrote a from-scratch Metal Flash Attention kernel for AMD. One float4 slice of the head per SIMD lane, simdgroups splitting the KV stream, online-softmax merge, validated against a CPU reference. It runs both prefill and decode on the GPU, supports head dims 128 and 256 and KV types f16/q8_0/q4_0 plus the heavy quantized cache. I just optimized it by splitting the KV across more simdgroups, which is a big decode-at-depth win: on an 8B with a compressed cache, generation went +42% at 2k context and +75% at 4k (roughly 19 to 33 t/s), and it stays around 22 t/s even at 16k instead of collapsing.

Rough numbers on my setup (RDNA2-class AMD card, 12 GB):

• Qwen3-8B Q4: ~101 t/s prompt, ~57 t/s generation
• Qwen3.6-35B-A3B (MoE, hybrid CPU offload): ~123 t/s prompt, ~18.6 t/s generation, ~25.7 with MTP speculative decoding
• Prompt processing is roughly 8x the stock Metal engine on these GPUs

The app is called ToshLLM. Native SwiftUI, no Electron or Python, nothing to install. It bundles two engines: the patched official llama.cpp, and an experimental one with aggressive KV-cache quantization (down to ~16% of f16) for long context on limited VRAM. On top of that there's a real chat UI with saved conversations and markdown, a model browser that estimates whether a model fits your VRAM before downloading, Hugging Face search, automatic MoE CPU-offload tuning, MTP speculative decoding, built-in benchmarks, and an OpenAI-compatible server so you can point any client at it.

It's free and GPL-3, no account, no telemetry. The AMD patches are in the repo so you can build from source. It's pre-1.0 and the DMGs aren't notarized yet (first launch needs Open Anyway).

If you've got one of these AMD Macs sitting around, I'd genuinely like testing reports, especially on cards I can't try myself (RDNA1 5500/5600/5700, the bigger 6800/6900, older Polaris/Vega) and how it holds up in long sessions.

Repo and releases: github.com/engeldlgado/toshllm

If you’re running an AMD RDNA2 card on a Hackintosh with NootRX, you already have working Metal — but stock inference engines still output garbage on these GPUs.

ToshLLM ships llama.cpp with AMD‑specific Metal patches (chunked staging transfers + concurrency disable) so you get correct output at full speed, in a polished SwiftUI app.

• Developed/tuned on RX 6700 XT 12 GB + NootRX
• Qwen3‑8B ~57 t/s gen / ~101 t/s prompt; 35B MoE ~26 t/s with MTP
• Model manager with VRAM/RAM estimates for your card, auto MoE offload, benchmarks, OpenAI API
• Brand‑new macOS 26 “Tahoe” Liquid Glass UI (falls back to materials on 14/15)
• Open source, GPL‑3.0, x86_64

Heads‑up: it’s beta, DMGs aren’t notarized yet (you’ll need “Open Anyway” once). On other AMD GPUs / driver setups behavior may differ — reports welcome.

https://github.com/engeldlgado/toshllm