




Two months later: ToshLLM has turned Intel Mac Pros + AMD GPUs into seriously capable local AI machines
I posted here about ToshLLM roughly two months ago, when I had just gotten local LLM inference working properly on Intel Macs with AMD GPUs by patching llama.cpp's Metal backend and writing an AMD-specific Flash Attention path.
A lot has happened since then.
What started as a way to make my RX 6700 XT useful for local inference has grown into a much broader AMD Metal optimization project, and one of the most interesting things has been seeing other people test it on hardware I don't own: W6900X setups, W6800X Duo, Vega II Duo, W5700X, Vega, Polaris, eGPUs, and multi-GPU Mac Pros.
The AMD backend itself has changed substantially.
Vega, Radeon VII, RX 400/500 and Radeon Pro Vega/WX
These cards now have dedicated Wave64 tuning instead of inheriting settings measured on newer RDNA cards.
On a Vega 64, depending on the model and quantization:
- generation improved by roughly 4-24%
- multi-conversation serving improved by 9-12%
- short cached follow-ups can be up to 39% faster
- MTP gained about 8%
- long-prompt processing also improved
- MoE models can experimentally use the wider matrix path previously used only on newer Radeon cards
Some measured examples:
| Workload | Before | Current |
|---|---|---|
| 9B Q5_K_M generation | 31.2 t/s | 38.8 t/s |
| 8B Q6_K generation | 35.5 t/s | 41.4 t/s |
| 7B / 64-expert MoE | 136.0 t/s | 155.3 t/s |
The goal here isn't just "make it run" anymore. I'm increasingly tuning kernels separately for the different AMD generations.
RDNA has kept getting faster too
A number of quantization and model-specific paths have been optimized.
For example:
- IQ2_M generation on a 1.5B model: 92.1 -> 140.8 t/s
- Qwen3.6 14B-A3B: 62.2 -> 70.8 t/s
- Qwen3.5 4B + MTP: 70.0 -> 81.6 t/s
- Qwen3.5 4B + DFlash: 68.9 -> 82.6 t/s
- gpt-oss-20B prompt processing: 1093 -> 1292 t/s
Q4_K, Q5_K, Q6_K, q5_0/q5_1 and several other paths have received AMD-specific kernel work as well.
ToshLLM is becoming useful as a server, not just a chat app
Concurrent inference has improved quite a bit.
On a 9B Q5_K_M model:
- 2 simultaneous conversations: 56.4 -> 70.8 t/s
- 3 simultaneous conversations: 60.5 -> 81.0 t/s
There is now a browser UI based on llama.cpp's web interface, router mode can switch between models without restarting the server, multiple servers can run simultaneously, and there is a dedicated embeddings server for local RAG.
The API remains OpenAI-compatible, and I've started seeing people use ToshLLM's bundled llama-server as a backend for other software rather than only using the SwiftUI application. That was not something I expected when I started this.
Image generation has become much lighter on VRAM
This has been another area I've been optimizing.
For example, on my RX 6700 XT:
- Z-Image 1600x900: 8.2 GB -> 966 MB VRAM
- Z-Image 1024x1024: 2.4 GB -> 690 MB
- image generation is roughly 12-14% faster
- SD 1.5 768x768: 2690 MB -> 281 MB
- SD 1.5 768x768: 149 sec -> 73 sec
SD 1.5 is now particularly interesting for older 4-6 GB Radeon cards.
There is also experimental local video generation with Wan models now. It has improved substantially and Wan 2.2 5B can run in 12 GB VRAM, but I want to be clear that I still consider the video side experimental and it needs more work before I put it in the same category as LLM inference.
Community testing has become a big part of the project
This is probably the part I've enjoyed most.
People are now submitting benchmarks from configurations such as dual W6900X, W6800X Duo, W5700X, Vega and other machines I could never test myself.
There is a public benchmark database at:
https://toshllm.com/benchmarks
ToshLLM itself is still completely free and GPL-3.0.
No account, no telemetry, no cloud inference. The OpenAI-compatible server, chat, image generation, benchmarks, RAG/embeddings, etc. all run locally.
It's still pre-1.0 and the current DMGs are not notarized yet, so macOS requires Open Anyway on first launch. Notarization is something I'm working toward.
If anyone here still has a Mac Pro full of Radeon hardware, I'm especially interested in results from:
- W5700X
- W6800X / W6800X Duo
- W6900X
- Vega II / Vega II Duo
- Radeon VII
- multi-MPX configurations
At this point some of these old Intel Mac Pros have absurd amounts of usable VRAM for local AI, and I'm curious how far we can push them.
ToshLLM: https://toshllm.com
Source / releases: https://github.com/engeldlgado/toshllm