r/LocalLLaMA

Qwen just released Qwen-AgentWorld-35B-A3B — a 35B-parameter MoE with only ~3B active parameters per token.

The interesting part: this is not positioned as a standard chat/instruction model or a full autonomous agent. It is a language world model trained to predict what an environment would return after an agent takes an action.

It covers seven agent interaction domains:

MCP / tool calling

Search

Terminal

Software engineering

Android

Web

Operating-system GUI interactions

The intended use seems to be simulating the environment side of an agent loop: given the action history and a new tool/GUI action, predict the next observation/state. That could be useful for agent training, offline evaluation, synthetic trajectories, testing tool-use workflows, or building sandbox-like environments without constantly running the real tools.

huggingface link

To anyone who needs AIR…

Spent a while treating layer placement, KV format and llama.cpp itself as experimental variables. 159 logged experiments. Numbers first, caveats after.

Hardware: AMD Ryzen AI MAX+ 395 (Strix Halo, 128 GB unified) + RTX 3090 Ti on an eGPU link. One llama.cpp process, AMD on Vulkan, NVIDIA on CUDA, one 27B model split across both.

Baseline: 9.474 tok/s. Target-only, no speculation, AMD side alone.

Now, code-shaped generation:

• 32K context: 153.32 tok/s
• 200K context: 87.74 tok/s

HumanEval, 164 problems, driven by a real coding agent (pi-agent), graded by executing the official tests:

• Local: 159/164 in 29.7 min
• Remote 2x RTX 3090 running vLLM TP2: 157/164 in 42.4 min

Long-context retrieval suite: 15/15, 352 s local vs 551 s on the remote.

Things that actually moved the needle, roughly in order of surprise:

1. The chat template. Swapping to a terser template (Qwen-Sharp) cut wall time 44% and output tokens 51%, with no accuracy change. Bigger than several weeks of GPU work. It just stops the model narrating.
2. KV cache format as a placement lever. Going q8_0 -> q4_0 on both K and V freed exactly 2,176 MiB, which was enough to move every full-attention layer onto the fast card at full 262K. +28% prefill, +20% generation. Qwen3.8 is 3:1 Gated DeltaNet to full attention, so only 16 of 64 layers have KV at all, and those are the only ones whose cost scales with context. Which layer sits on which GPU matters enormously.
3. --spec-type draft-mtp,ngram-mod. The flag accumulates. n-gram on top of MTP is worth +72 to +140% on code-shaped output, about -1% on prose, and costs nothing in VRAM. Free win if you generate code.
4. A one-line llama.cpp patch: --spec-draft-ubatch. The speculative context was inheriting the target's 512 micro-batch and reserving a 2.2 GiB compute buffer to draft 4 tokens. Setting it to 64 freed 1,039 MiB for a 1.78% prefill cost, which bought another layer of placement.
5. MTMD_BACKEND_DEVICE=Vulkan1 to put the vision encoder on the idle iGPU. Undocumented as far as I can tell. Default puts it on the first GPU-type device, which is the already-full 3090 Ti, and it dies allocating 884 MiB.

Things that did NOT work, so you don't have to try them:

• External draft models (0.8B, 4B): 5.06-5.77 tok/s, worse than no speculation. Sequential drafting latency across PCIe dominates.
• Multi-lane parallel drafting: down to 1.62 tok/s. The 27B verification pass is the bottleneck; anything that widens the verification graph loses.
• Q6_K: 2.3x the wall time, marginally lower accuracy (because if hit the 64K output limit wall twice). Only 15.6 of its 20.5 GiB fits on the card, so 4 attention layers get pushed to the iGPU instead of 1.

Caveats, because this is r/LocalLLaMA and you'd find them anyway:

• 153 tok/s is code-shaped generation at 32K where n-gram does heavy lifting. Same config on prose at 200K is 35.8. Different workloads, kept separate on purpose.
• The remote comparison is deployed-stack vs deployed-stack, not hardware isolated: it serves an AWQ-MTP fine-tune, local serves Q4_K_M of the base.
• I asked for standalone functions rather than the canonical prompt+completion format, so 3 of the 5 HumanEval failures are NameErrors on helper functions that the official harness would have kept in scope. 159/164 is probably a slight undercount. Same undercount applies to every arm.
• q4_0 keys are a real quality change. Passed my 15/15 gate; validate on your own workload before trusting it.

Full writeup with every flag, patch and failed branch: https://definedrr.medium.com/qwen3-8-27b-9-tokens-per-second-to-153-ee3781f4a3f5?sharedUserId=definedrr

https://www.reddit.com/r/LocalLLaMA/comments/1j7r47l/i_just_made_an_animation_of_a_ball_bouncing/

This post inspired me to make that test after a year ;)

That is one of my many tests I make comparing output quality.

What is more interesting using a PI Agent results are much better than an Opencode using a Qwen 3.8 27b ?!

Seems PI Agent is much better in the agent environment somehow... Not counting uses less tokens , do not have a hard limit of 32k output tokens, is faster, do not freezing, compressing context far less than Opencode. For instance if you have context in the Opencode output 32k and all context 100k then the compression is starting at 67k context ... PI is starting at 90k context even if you have set output context 64k or more.

My config for RTX 3090

llama-server with ini config -> which is exposing API to Opencode and PI agent.

llama-server.exe --models-preset 1_preset.ini --models-max 1 --direct-io

config ini

[Qwen3.8-27B_dense_c-100k]
model = models/Qwen3.8-27B-Q4_K_M.gguf
mmproj = models/mmproj-BF16-Qwen3.8-27B-UD-Q4_K_XL.gguf
reasoning-format = deepseek
flash-attn = on
n-gpu-layers = 99
reasoning = on
ctx-size = 100000
temperature=1.0
top-p=0.95
top-k=20
min-p=0.0
presence-penalty=0.0
repeat-penalty=1.0
mmproj-offload = false

ONE MORE IMPORTANT THING:

Always use a VISION module as the model is using vision to asses the output quality!

I am offloading it to a RAM as we do not need an extremely fast vision for a code.

A screenshot processing on a GPU 0.3s vs a RAM 3s do not make a big difference on a few screenshots during a code generation / debugging ;)

I wanted to just test the unsloth 1bit quant of qwen 3.8 27b as I have just 8gb vram and ngl it gave me a good laugh

Finally see a new voice conversion model. MeanVC2 supports cross-gender and cross-language voice conversion. 3x realtime on CPU with audio.cpp.

Disclaimer: The converted voice quality of MeanVC2 is decent; the noise comes from my rough demo engineering, not the model itself. This is only a quick demo to show MeanVC2 running in real time.

https://huggingface.co/ASLP-lab/MeanVC2

From the Digg article (https://digg.com/tech/3pf3046j)

“Google Gemma posted that the family of open models has achieved 1 billion downloads. The account is hosting an exclusive evening in San Francisco on August 20 to honor open-source builders, researchers, and contributors. Space at the event is limited and applications are required for attendance. Demis Hassabis shared the announcement from the official account. Clément Farabet replied that he will attend and is excited to celebrate the milestone. The post frames the achievement as driven by the community advancing the Gemmaverse forward.”

Seems like it would be the perfect venue to announce a new model, but could just be wishful thinking on my part

Thoughts? Speculation? Too busy testing Qwen 3.8 27b to care?

Would be a good time for Google to drop a 120b. Pull a Meta Glimmer and be king of the hill for a week or so before Qwen countered with a new mid-size model.

I pre-trained a 1.02-billion-parameter on Kimi K3 replica trained on 5.00 billion decontaminated tokens for $250.

This model has 1.02 billion parameters, of which 145 million are active per token.

It is roughly one two-thousandth of K3 by total size. It saw 5,000,003,584 tokens, which is a rounding error against the corpora frontier models are trained on.

It has never been instruction-tuned, and it has only ever done one thing: predict the next token.

What it does have is K3's architecture:

- Kimi Delta Attention, Gated MLA, Attention Residuals

- LatentMoE with the same aux-loss-free balancer

- Same activation function with the same two constants

- K3's own 163,840-token tokenizer, unmodified. 

I report a 33.4% HellaSwag which beats the GPT-2 124M score of 28%

Read the entire tutorial here: https://books.vizuara.ai/book/pretraining-a-mini-k3

Presets is an open-source toolkit for optimizing inference with agents, which we build at dstack. Here's one example of using it on a single MI300X.

Qwen3.8-27B went from 311 to 495 tok/s, +59%, at the full 1M context with p50 TTFT under 1.5s and four concurrent users at 10k in / 1.5k out.

The gains came from linked optimization sessions and source-level patches to SGLang's AITER attention backend.

What comes out is a portable preset that deploys on any AMD cloud, Kubernetes cluster, or bare-metal fleet: https://dstack.ai/blog/presets/

Dense 27B, meet DFlash2. On my Flow Z13 (Ryzen AI Max+ 395, Radeon 8060S, 128GB), Qwen3.8-27B hits 31 t/s burst at 80W and runs 26-29 t/s through an entire 32k-token generation, with every token paying for all 27B parameters. No MoE shortcuts this time.

Three things got it there: Inco AI's DFlash2 drafter, n-max 4, and a chat template that stops the model from drowning in its own reasoning. The last one surprised me most. The stock template's runaway thinking didn't just waste tokens, it collapsed draft acceptance at depth; fix the style and the speed decay disappears.

Last week's DeepSeek V4 Flash guide (https://www.reddit.com/r/LocalLLaMA/comments/1vlmh0b) covered the MoE side of this hardware. This post is the dense half.

Note: the writing is AI-assisted editing; the research, debugging, and every number are from my own runs on this machine.

Edit (Aug 20): the recommended config was revised after the burst table below was measured. Three findings landed on top of the initial numbers: n-max 4 beats 7 by 25% on long generations, the Sharp chat template fixes a runaway-reasoning failure mode and lifts sustained decode another 26%, and q8_0 drafter KV is a free +1.6% with bit-identical output. The launch command reflects all three; sustained expectations are now 26-29 t/s from 1k to 32k output tokens.

Caveat: these are early benchmarks. The sustained matrix (both quants to 8k, Q5 to 64k), the Unsloth v2/v3 A/B, and full-corpus PPL have landed; more workloads and Q4 long-horizon runs are still in progress.

TL;DR

• Daily driver: UD-Q5_K_XL + incoai DFlash2 Q4_K_M drafter (n-max 4) + Nathan's v0.6.5, f16 KV, drafter KV q8_0, ubatch 4096, Sharp chat template. ~30 t/s burst, 26-29 t/s sustained out to 32k tokens.
• DFlash2 over the built-in MTP is +40% burst (30.2 vs 21.6 t/s at 70W) and +60% sustained (25.9 vs 16.3 t/s at 8k tokens). Lossless: output is bit-identical to unspec decode at temp 0, verified.
• Q5_K_XL decodes faster than Q4_K_XL with DFlash2 on short generations (31.4 vs 28.5 t/s at 80W burst; Q5's 55.9% vs 44.9% acceptance pays for the extra bandwidth). On long generations the ranking flips: at 8k tokens Q4 + Sharp beats Q5 + Sharp (28.8 vs 27.5 t/s). I run Q5 for the quality at ~1-2 t/s sustained cost; all-long-form-code workflows should take Q4 + Sharp.
• Pin your Unsloth revision: every number here is the pre-Aug-19 v2 file. Unsloth re-uploaded all UD quants as Dynamic v3 under the same filenames. v3 is better quality (PPL 13.84 vs 14.51) and faster on burst, but collapses at depth: 19.7 vs 27.5 t/s at 8k under the recommended config. The v2 download link is in Sources.
• n-max 4, not 7. At depth, deep drafts get rejected and waste verify bandwidth: n4 is +25% over n7 on long generations, costs ~1.5% on short ones. Nathan's v0.6.5 independently landed on the same number, and my re-sweep under Sharp (where higher acceptance could have shifted the optimum) still says 4.
• The Sharp template is the hidden lever. The stock template burned entire 4k and 8k budgets on reasoning with zero output on my agentic test prompt. With Sharp: the model answers, and sustained decode runs 25.7-29.0 t/s from 1k to 32k tokens, peaking at 32k.
• Skip the FP4 builds: a full PPL tier worse than Q4_K_XL, slower than this stack anyway, and riding a one-off engine (gotcha #3). Tested --spec-draft-p-min 0.7 too: -3% here, an MTP-era trick that doesn't transfer to block drafting.

Quick start

1. Grab Nathan's v0.6.5: Nathanw1014/strix-halo-llamacpp. I benchmarked the identical payload when it was a dev tag (dev-20260819-0b0f35d); v0.6.5 is the same code, now gfx1151-validated.
2. Download the target: Qwen3.8-27B-UD-Q5_K_XL (20.6GB). Pin revision 408fcc1807ab (pre-Aug-19 v2, the file I benched): plain main now serves Dynamic v3, which trades sustained speed for quality (details in the note below).
3. Download the drafter: incoai/Qwen3.8-27B-DFlash2-GGUF, Q4_K_M (1.1GB). Caveat: the upstream DFlash2 PR is still in review, so if the drafter format changes before merge you may need to re-download.
4. Download the template: chat_template.jinja from the Sharp repo, save as sharp.jinja.
5. Launch:

llama-server \
  -m Qwen3.8-27B-UD-Q5_K_XL.gguf \
  -md Qwen3.8-27B-DFlash2-Q4_K_M.gguf \
  -ngl all -ngld all -fa on \
  -ctk f16 -ctv f16 -ctkd q8_0 -ctvd q8_0 \
  -c 65536 -np 1 \
  -b 4096 -ub 4096 \
  -t 16 -tb 32 \
  --spec-type draft-dflash --spec-draft-n-max 4 \
  --chat-template-file sharp.jinja \
  --jinja --host 127.0.0.1 --port 8080 --metrics

The -ctkd/-ctvd q8_0 flags quantize only the drafter's KV cache: +1.6% t/s, and I verified the output stays bit-identical. No boot params needed; the model is small enough that GTT defaults just work.

Results

Config-comparison table: 4-task streaming bench (python, math, JSON, long-prompt; short 100-200 token generations, warm runs), stock template, same engine build throughout.

Spec method	n-max	@70W	@80W	Acceptance
plain (no spec)	-	~14 t/s	-	-
MTP n5/p0.50	5	21.6 t/s	-	66.4%
DFlash2, Q4_K_XL	7	27.3 t/s	28.5 t/s	44.9%
DFlash2, Q5_K_XL	7	30.2 t/s	31.4 t/s	55.9%

Read those as burst rates. The number that matters for real work is sustained generation, measured with a fixed agentic coding prompt at n4 with drafter-KV q8_0, 70W:

quant + template	2k tokens	8k tokens	32k tokens
Q5_K_XL, stock	20.6 t/s	19.8-20.2 t/s	-
Q5_K_XL, Sharp	25.9 t/s	27.5 t/s	29.0 t/s
Q4_K_XL, stock	23.4 t/s	20.4 t/s	-
Q4_K_XL, Sharp	26.6 t/s	28.8 t/s	not tested

Two lessons in that table. Burst belongs to Q5 (its distribution sits closer to the drafter's training, so acceptance climbs and pays for the extra weight bytes); at 8k the ranking flips to Q4, because acceptance converges between the quants and Q4's ~2.3GB fewer weight bytes per pass wins. Q5 stays my daily driver for the PPL and burst; Q4 + Sharp is the pick for long-form code.

Second, and bigger: with the Sharp template, decode gains speed with depth instead of decaying.

The "spec decode collapses at long context" story turned out to be half a template problem. Stock-template output drifts into sprawling reasoning that the drafter can't predict, so acceptance falls from ~56% to ~24% and t/s follows.

Sharp's terse style keeps acceptance climbing as code accumulates: 0.57 at 8k, 0.72 by 32k, and the rate curve rises the whole way (25.7 at 1k up to 29.0 t/s at 32k). Past ~40k the classic long-context decay does return: a full 64k generation averages 24.8 t/s with the trailing window near 16, even though acceptance still holds 0.62. So Sharp moved the horizon from 8k to ~32k tokens; it did not make it infinite, and the late decay is attention cost, not draft mismatch.

Acceptance percentages are not comparable across methods (DFlash2 drafts blocks in parallel, MTP sequentially), so judge by t/s. Per-task burst spans 21 to 40 t/s depending on workload; that spread is content, not noise.

Prefill: what to expect at depth

Shallow (3k prompt): ~298-302 t/s at ubatch 4096, both spec methods identical. Ubatch 4096 is the knee: 8192 measured flat, 2048 leaves ~5% on the table.

Deep context, f16 KV, llama-bench:

Model	pp32768	pp65536
Q4_K_XL	265.1 t/s	222.1 t/s
Q5_K_XL	251.5 t/s	212.2 t/s

Why the deep numbers hold up: Nathan's fork carries a flash-attention prefill series (dequant-once scratch, KV contiguize, coopmat1 tuning), and the gains grow with depth. I ported that series onto the ROCmFPX fork to A/B it properly: +59% at 32k with f16 KV (165 to 263 t/s). At 64k the unpatched engine didn't just slow down, it threw vk::DeviceLostError on every model I tried, while the ported one ran 199-219 t/s.

The port is here with full numbers, and the findings went upstream.

Why not FP4? The reddit hype, checked

The ROCmFP4 builds were everywhere last week with 30+ t/s headlines, so I ran the whole stack. What holds up: the FP4 engine's Vulkan path genuinely is fast, and it beat the MTP-era configs (23 vs 21.6 t/s). What doesn't: it's slower than Q5+DFlash2, it measures a full PPL tier worse than Q4_K_XL, and the engine is a dead end.

Same-engine, same-method short-corpus perplexity (90KB wikitext-2 slice, c8192, n512, 2 chunks, f16 KV):

Quant	Short PPL
Q5_K_XL	53.85
Q4_K_XL	57.26
ROCmFP4_FAST	59.06

The engine is also a one-off: the pinned build the benchmarks ran against was force-pushed out of the repo's history, and the prebuilt crashes at 64k f16 KV. Fun demo, not a daily driver.

Dynamic v3 note (updated Aug 20): Unsloth re-uploaded every UD quant with Dynamic v3 under the same filenames on Aug 19, so all numbers in this post are the v2 files. I A/B'd both Q5s under the recommended Sharp config, same session, 70W. v3 wins two of three: PPL 13.84 vs 14.51 (wikitext-2 full corpus, ~4x the error bars) and burst 30.6 vs 29.2 t/s at 73.7% acceptance.

The third column is why I still recommend v2: sustained 8k collapses to 19.7 t/s vs 27.5 (acceptance decays to 0.40 at depth). For agentic coding that's the whole game; for short generations and maximum quality, take v3 knowingly.

Reproducing my numbers means pinning the revision. The v2 files live at commit 408fcc1807ab (Aug 14, last before the re-upload): Q5 direct is huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/408fcc1807ab/Qwen3.8-27B-UD-Q5_K_XL.gguf, or hf download unsloth/Qwen3.8-27B-GGUF Qwen3.8-27B-UD-Q5_K_XL.gguf --revision 408fcc1807ab. My Q4_K_XL numbers are v2-era too; current main serves a different Q4.

Gotchas (learned the hard way)

1. The stock template can burn your entire budget on thinking. On my agentic test prompt it produced zero content tokens at both 4k and 8k max_tokens: 8192 reasoning tokens, no answer, six minutes of GPU time. That failure mode, not hardware, is the biggest speed killer this model ships with.

The Sharp template fixes it and lifted sustained decode 26% on top.

2. n-max is a depth knob. 4/7/8 are within ~1.5% on short generations, but at 8k tokens n4 beats n7 by 25% (19.8 vs 15.8 t/s). Deep drafts get rejected as acceptance decays, so the long tail just costs verify bandwidth. Nathan's v0.6.5 release notes independently measured the same crossover (width 4: 21.1 t/s vs width 7: 16.3 at 32k), and re-sweeping under Sharp confirmed 4: n5 is par on burst and -2% at 8k.

3. DSpark is broken for this model in llama.cpp right now. The community drafter gives 15-16% acceptance and no speedup, and it's not the drafter's fault: I verified the GGUF metadata and bisected the failure into llama.cpp's DSpark draft generation itself. Details in ggml-org/llama.cpp#25618; use MTP or DFlash2 until it lands.

4. p-min does not transfer. A commenter reported --spec-draft-p-min 0.7 lifting DS4+MTP acceptance dramatically; on DFlash2 here it cost 3%. Block drafting changes the economics, pruning just shrinks the block.

5. First bench run is always cold. First DFlash2 pass read 24 t/s; warm runs 27+. The drafter warms up across requests, so run the workload twice before believing a number.

6. Dense vs MoE is the real choice. Qwen3-Coder-30B-A3B (3B active) prefills at ~1900 t/s on this box vs ~300 here; that's arithmetic, not tuning, so huge-prompt short-answer workloads should use the A3B. For agentic coding where answer quality is the product, this stack is the strongest thing I've run locally.

Sources

• Engine: Nathanw1014/strix-halo-llamacpp v0.6.5 (DFlash2 validated on gfx1151; I benched the identical payload as dev-20260819-0b0f35d)
• DFlash2: https://inco.ai/blog/dflash2/
• Drafter: incoai/Qwen3.8-27B-DFlash2-GGUF
• Template: peculiar-ragdoll/Qwen-Sharp-Chat-Templates
• Target (v2, pinned): unsloth/Qwen3.8-27B-GGUF @ 408fcc1807ab UD-Q5_K_XL + UD-Q4_K_XL; current main = Dynamic v3, see the note above
• DSpark broken draft path: ggml-org/llama.cpp#25618
• My FA-prefill port + ROCmFPX findings: aic0d3r/ROCmFPX port-nathan-prefill, charlie12345/ROCmFPX#86
• Prior work on this machine: DeepSeek V4 Flash guide: https://www.reddit.com/r/LocalLLaMA/comments/1vlmh0b (full guide: https://www.reddit.com/r/LocalAiCore/comments/1vkq5kj)

Happy to answer questions. More Strix Halo benchmarks and guides at r/LocalAiCore.

Hey everyone! We’re releasing new Qwen3.8-27B GGUFs with 10% higher accuracy for the same size. This uses a new version of Dynamic v3.0

Unsloth Dynamic V3 outperforms others by >10% on Div-300, KLD & more benchmarks.

We also release 1-bit quants that retain 77% accuracy. Run on 8GB RAM.

Some of you already saw we updated our quants a few hours ago. No, nothing was broken, nothing needed fixes (I don't know why people even said this since it's a complete fabricated story). This was purely an update to make them EVEN BETTER.

We do not train on the imatrix calibration dataset, and we do NOT use QAT or QAD. Everything is done through post-training quantization. Our imatrix file used is available for the community to test, evaluate, and use. We encourage researchers and developers to create variations and fine-tunes of Qwen3.8 using our Unsloth quants/imatrix. You can read our over fitting analysis as well.

Blog with all details and more benchmarks: https://unsloth.ai/docs/basics/dynamic-3.0-ggufs

GGUF: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

Enjoy! We also will be doing a new Unsloth Desktop update today: https://github.com/unslothai/unsloth

We had A LOT of updates and will be introducing auto compaction, allowing external APIs to do tool calling and more.