Okay, here we are again :)
We received our second R9700 earlier this week, so we decided to jump on tensor parallelism.
We already had prepared a few things for this, and the TP kernels we already had in Paiton did their work.

We spent a lot of time (significant amount of GPU time (MI355x)) on the 3-bit quantization of this model and we are pretty happy with the accuracy.

What "3-bit" means here.

This is not a full 3-bit model. The 512 experts in every layer, that is where almost all the weights are, are stored at 3.125 bits per weight (3-bit codes, rotated, with a small scale per 64 weights). Everything else stays bigger: the attention and the shared parts are 8-bit, the speculative layer is 4-bit, and the vision part is plain bf16.
So the big part is 3-bit, the sensitive parts are not, and that is why the accuracy holds up. Counted over all weights it is a bit above 3 bits on average. Our own GPTQ run, no shortcuts.

Accuracy against the bf16 model (same evaluation corpus, text and assistant positions; two correct bf16 implementations against each other are the floor):

KL to bf16	KL to bf16, same expert routing forced	Top-1 agreement with bf16
bf16 vs bf16 (the floor)	0.0077	n/a
This 3-bit model	0.060	0.041

Long context check	Result
Needle test, 131,072 tokens	40 / 40, same answers as bf16
Needle test, 200,000 tokens	40 / 40, same answers as bf16

The whole language model (experts, trunk, speculative layer, draft head, vision tensors) sits in the VRAM of the two cards; only the model's n-gram embedding table lives in pinned system RAM. No expert offload, no weights streamed from the host during decoding.

And again, like before: we did not build a custom engine for this. It is still regular vLLM, with our Paiton plugin and our own kernels inside. ;)

Performance.

BetterBench 0.6.0, standard profile, measured on our own machine against our OpenAI-compatible server, with the exact image and launcher we publish. Decode mode: 98K context, speculative decoding on. Prefill with exact bf16 activations, no compression.

Decode, single request	tok/s
Weighted over the 8 categories	216.3
chat	202.1
code	214.3
file edit	236.0
json	259.2
math	255.5
prose	183.3
reasoning	190.1
summarization	239.8

Concurrent requests	Aggregate tok/s
1	204.6
2	315.1
4	449.2
8	565.1 (48 of 48 requests ok)

Prefill, prompt length	tok/s
2K	7,925
8K	8,321
16K	8,311
32K	8,232
64K	8,161
128K	7,928

Latency
Time to first token, short prompts (p50)
Update p99 (the longest gap between two tokens)

Long context. We tested 200K. The launcher has a 200K mode, and prefix caching is an opt-in flag that gives exactly the same output on a hit.

200K mode	Result
Needle test at 200,000 tokens	40 / 40, same answers as bf16
Single-stream decode after a 32K / 100K / 190K prompt	105 / 101 / 100 tok/s, flat
Time to first token, 190K prompt	25 s
Prefix cache hit, 64K repeated prompt	9.6 s cold, 0.35 s on the hit
Prefix cache hit, 128K repeated prompt	20 s cold, 0.41 s on the hit

Overall, not bad for a first release :)
Oh, right, of course we generated an "AI Slop" image as well. (we know some of you like them (some of you don't ;-)))

https://preview.redd.it/836rjbm2cmuh1.png?width=1672&format=png&auto=webp&s=033741b4cfe64c03fdd966a400d847150ca275de

Weights
Image and launcher: github.com/Eliovp-BV/paiton-vllm-plugin (directory models/Qwen3.8-Flash-Next)

This is our first release of this model, more will follow.
Next is the RAM and SSD cache tier, like we did for the 27B model, so cached prompts can live outside the VRAM and much longer contexts become possible.
After that the 4-bit version, with even better accuracy.
We keep tuning, we keep optimizing.
A longer blog post with all the details is being worked on as well.

And of course, feedback is very welcome!

TL;DR: Qwen3.8 27B on a single AMD Radeon AI PRO R9700, with 3-bit weights and speculative decoding, now keeps 569,878 tokens of reusable cache in its new coding mode.
Every request still gets 262,144 tokens of context, and two full-length requests fit at once. Coding agents resend the whole conversation on every turn; now only the new part is read.
Later agent turns start up to 12× sooner, whole agent sessions finish about 6× faster, and a 258K-token document the server has already read comes back in 2.7 s instead of 134 s. Decode speed and accuracy are within noise of our previous release.

We read your comments and we listened

You asked for more context. You got more than half a million tokens of cache on one card, with multi-hour accuracy runs on every configuration (GSM8K, HumanEval, MMLU-Pro, needle tests), at essentially the same speed as our previous release.

	Previous release (2 Oct)	This release, --mode long-kv4
Reusable (prefix-cached) tokens in the 4-bit mode	0 (no prefix cache)	569,878
Tokens per request	262,144	262,144
Full-length requests at once	1	2
Decode, single stream	174.3 tok/s	173.4 tok/s (−0.5 %)
Aggregate, 8 requests	462.4 tok/s	478.5 tok/s (+3.5 %)
Time to first token, short prompts (p50)	86.7 ms	87.1 ms
GSM8K / HumanEval / MMLU-Pro	95.53 / 92.07 / 60.57	95.45 / 92.68 / 59.93

Our previous release's FP8 --mode long already had a prefix cache, holding 281,665 tokens. The new 4-bit coding mode caches about twice as much.

Run it

git clone https://github.com/Eliovp-BV/paiton-vllm-plugin && cd paiton-vllm-plugin
# set the PAITON_* model paths as shown in the README (already set up? just `git pull`)
bash models/Qwen3.8-MXFP4-DFlash2/run-3bit.sh --mode long-kv4

Point your agent at http://127.0.0.1:18982/v1
The model name is Qwen3.8, any API key works, and the context window is 262,144 tokens. Already running our 3-bit weights? No new download. The launcher pins the new container and Docker pulls it on first start. Every mode is in the README.

No new engine

This is stock vLLM with a plugin: the same OpenAI-compatible server, the same API, and the same tools and workflows you already use. Nothing to relearn, nothing to migrate.

We will keep publishing new ready-built containers that follow upstream vLLM. Each one goes through the same full validation (speed, accuracy, long-context) before it ships. You get the upstream improvements without lifting a finger. One command and you're serving.

What it does for coding agents

Without a prefix cache, the server rereads the whole prompt on every turn: at 250K tokens that is about two minutes before the first token. Now finished requests stay cached until the space is needed, turn 20 reads only what changed, and in our sessions 91–92 % of prompt tokens came from the cache.

Workload (3-bit, thinking off)	Previous 4-bit mode (no prefix cache)	--mode long-kv4	Faster
20-turn conversation growing from 50K to 253K tokens, whole session	1,237 s	204 s	6.1×
… average wait for the first token, turns 2–20	63.6 s	7.8 s	8×
3 agents sharing a 100K-token repo, 5 turns each, whole session	655 s	111 s	5.9×
… average wait for the first token, later turns	84 s	7.0 s	12×
New question about a 258K-token document already read	134 s	2.7 s	49×

The cached answer to the 258K-token document matched the cold read exactly.

Speed details (BetterBench, full 20-pass run): decode within 0.5 % single-stream; +3.5 % with 8 requests (−2.5 % at 4); time to first token unchanged.
A new long prompt reads about 3 % slower, and short prompts 7–13 % slower (a fraction of a second). That's the price of ending each prefill step where a later request can pick up from the cache.

Accuracy (greedy, paired per question): the coding mode is within noise of the previous release, and the 512K mode is within noise of the coding mode.

	MXFP4	Previous release (3-bit)	--mode long-kv4	--mode long-512k
GSM8K 5-shot (1,319)	95.68	95.53	95.45	95.53
HumanEval pass@1 (164)	95.12	92.07	92.68	94.51
MMLU-Pro subset (1,400)	62.57	60.57	59.93	60.64

Opt-in: 524,288 tokens in one request

--mode long-512k extends the position range with the model's official long-context scaling. That scaling applies to every request in this mode, so use it only when a single request needs more than 262K.

• It found 4/4 planted facts at 300K and 4/4 at 500K tokens.
• A cold read takes 158 s at 300K and about 6 minutes at 500K. Follow-up questions take 2.8–4.4 s from the cache.
• The cache holds 594,290 tokens, and weighted decode is within 0.6 % (in one run the chat category was 16 % slower).

Unchanged

MXFP4 (run-mxfp4.sh) is still there and still the most accurate option. --vision works in the default 65K mode and in --mode long (images with up to 245K tokens of context).
The previous release is one --image flag away (see the README).

Caveats, honestly

• System RAM: the coding and 512K modes pin 2.4 GiB of system RAM for the embedding table, and 16 GB of RAM is enough (tested). Below about 13.5 GiB of total RAM, --mode long-kv4 keeps the table on the GPU and caches 451,879 tokens. You still get 262K per request and the prefix cache. --mode long-512k needs the RAM.
• Cold reads: the first read of a 258K-token prompt still takes over two minutes. The cache helps from the second request on, as long as the start of the prompt stays the same (same system prompt, no timestamp at the top). usage.prompt_tokens_details.cached_tokens shows every hit.
• Vision isn't in the coding or 512K modes yet. Use --mode long --vision for images with long context.
• RAM/SSD cache tier: the experimental system-RAM tier for the prefix cache (--host-cache-gib, plus an SSD tier behind it) spills even more context off the card. For now it works with the FP8 --mode long; support for the 4-bit coding mode lands in the next release. The coding mode's 569,878-token GPU cache works today.
• The agent-session numbers and the accuracy columns were measured on pre-release builds of these configurations; the README has every number.

Our testbench is ancient, slow CPU, limited and slow memory (16GB), so your results will most probably be even better!

What's next: more GPUs

More R9700s are on the way, and we're going after tensor parallelism next.
Others are working on multi-card setups too, so expect some healthy competition on that front. Good for everyone running AMD at home!

More: Paiton

We said we were done, but we couldn’t help ourselves.

A follow-up to our previous R9700 post: the updated image reaches 154.42 tok/s weighted decode, up 5.16% against a fresh run of the released image

JSON reaches 218.1 tok/s, code 180.6, and summarization 138.8. We measured broad gains across decode, concurrency and prefill: weighted decode is up 5.2%, with 4.4–5.6% gains at the larger prefill depths. Weighted decode, category medians and prefill are separate metrics.

We combined the native target and DFlash2 improvements into one container and ran the full benchmark against a fresh release control. Every updated number below comes from that same packaged-image run, with unchanged checkpoint, corpus, context and target sampling settings. The tables also show our previous post; percentage changes use the fresh matched release control.

Decode category	Previous post tok/s	Fresh released image median tok/s	Updated Release median tok/s	Change
Chat	123.6	123.2	121.3	-1.53%
Code	169.1	168.7	180.6	+7.01%
File edit	185.7	185.5	181.2	-2.29%
JSON	216.0	215.8	218.1	+1.06%
Math	182.5	182.7	184.3	+0.88%
Prose	75.4	75.4	79.8	+5.85%
Reasoning	112.4	112.5	117.8	+4.75%
Summarization	115.0	115.1	138.8	+20.61%

Chat and file editing were lower in this run; both are included. The weighted score uses the benchmark’s fixed category weights.

Concurrent requests	Previous post tok/s	Fresh released image tok/s	Updated tok/s	Change
1	115.96	115.43	122.62	+6.22%
2	202.85	200.55	207.18	+3.30%
4	301.75	300.15	308.17	+2.67%
8	406.49	410.71	421.20	+2.55%

Concurrency is aggregate throughput over each complete 48-request mixed workload, including prompt handling and the final requests finishing.

Nominal prefill depth	Previous post input tok/s	Fresh released input tok/s	Updated input tok/s	Change
2,000	3510	3501	3501	-0.00%
8,000	3532	3535	3692	+4.44%
16,000	3535	3540	3694	+4.37%
32,000	3389	3394	3584	+5.58%
64,000	3116	3118	3293	+5.61%

BetterBench’s nominal depth labels correspond to actual median prompt lengths from 1,516.5 to 47,016.5 tokens. Prefill throughput uses actual input tokens divided by time to first token.

Upstream vLLM 0.29 / ROCm 10, the Paiton plugin, Unsloth Qwen3.8 NVFP4 converted to MXFP4, DFlash2, FP8 KV cache, a 65,536-token maximum context and up to eight active requests.
Thinking and APC are off.
Seed 42, temperature 0.7, top-p 0.95, top-k 20.
APC remains an optional setting; no cached-prefix speedup is included here.

Both arms completed 290/290 benchmark requests, following the same 76-request preparation. Five scored decode requests per category, eight scored prefill requests per depth, and 48 requests per concurrency level. The updated image matched all 12/12 greedy checks before timing; both arms repeated their 12/12 outputs after the complete workload. No unhandled serving errors or scored code-cache changes were found.

Sampled continuations and acceptance work can differ across configurations.
These are measured serving-throughput results, not identical-output timing or a broad model-quality evaluation.
An earlier complete combined-runtime run measured 154.05 tok/s weighted decode; this post uses the subsequent packaged-image run throughout.

I think we're done now :)

Pull the new package.
Setup and full results.