Some interesting Qwen3.8-27b findings
For context: All tests Unsloth Q8_K_XL, F16KV, on a single RTX Pro 6000 Blackwell, Max-Q (the other one was sticking its tongue out at me)
Oh boy... this entire Qwen madness has been driving me nuts. So i put together my own benchmark. Small, honest. testing all sorts of combinations, and because of ONE big glitch, went down an actual rabbit hole and proved that this thing can INDEED rip. batches and batches of 300, 500 even 800+ tokens per second. But funny enough, that's not the most important bit. The Quality of the results mattered more. R$ed bars are failed generations (JS errors, doesn't matter how small), Yellow are positioning hiccups but otherwise ok), Green are fully successful.
While i still do more testing - now moving from raw curl tests to an actual harness, to evaluate my findings on long context horizon, tool calling, etc, i'll leave you with these images.
A note on token counts: these aren’t short-completion benchmarks. Prompt 1 was deliberately open-ended and the model routinely generated ~48k–70k tokens while reasoning and building the artifact. Prompt 2 is different: it feeds a previously generated artifact back into context and asks Qwen to work from/reproduce that existing structure, so the actual generated completion is typically ~20k–23k tokens. That’s where ngram speculation gets enough reusable material to become completely ridiculous.
The throughput numbers in the chart are full-run generation averages reported by llama.cpp, not the 3-second peaks. During the fastest P2 runs, tg_3s repeatedly reached 300–700+ t/s; those bursts are interesting, but the headline result is still 234.41 t/s averaged across the entire ~20k-token generation. And Even P1 runs (the one marked p1, s666) had bursts of 300-812 (peak)
The last 5 runs are the important bits:
P1 / seed 666: 86.72 t/s
P2 baseline / 666: 116.15 t/s
P2 baseline / 667: 125.84 t/s
P2 ngram-mod / 666: 163.11 t/s
P2 ngram-mod / 667: 234.41 t/s
Because prompt 1 (create a simple dashboard, one file, self contained), at some point generated some ludicrous segments of 300-800 tokens per second, i went down investigating that. Tested cold vs hot runs (funny enough, that had nothing to do with it), to reproduce, and it did. Then, on a hunch, i changed the prompt that ngram had enough reproducible content to absolutely RIP through content. see the last 2 runs.
"qwen38-ngram-mod":
env:
- "CUDA_VISIBLE_DEVICES=0"
cmd: >
/app/llama-server
-m /models/qwen3.8-27b/Qwen3.8-27B-UD-Q8_K_XL.gguf
--mmproj /models/qwen-mmproj/mmproj-F16.gguf
--port ${PORT}
-ngl 99
--ctx-size 131072
--batch-size 2048
--ubatch-size 512
--cache-type-k f16
--cache-type-v f16
--spec-type ngram-mod
--spec-draft-n-max 8
--spec-draft-p-min 0.8
--flash-attn on
-np 1
--jinja
--chat-template-file /models/qwen-mmproj/chat_template.jinja
--reasoning-format deepseek
--reasoning-preserve
--temp 1.0
--top-p 0.95
--top-k 20
--min-p 0.0
--presence-penalty 0.0
--repeat-penalty 1.0
Chat template from here: https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
Github repo here: https://github.com/mihaiCroitoru/qwen38-bench
The repo includes the exact prompts/configs, raw benchmark harness, timing methodology, Chrome artifact validation, and full per-run reporting. I’ll push all preserved run artifacts/results once I finish the harness-level tests.
Feel free to run your own tests, tell me what i missed about mine!