u/iamMess

Edit: Title says 134 tps, it's actually 138 -- keep in mind my 3090 is power limited to 250w.

Three days ago I released a hyper-optimized Qwen3.8-27B inference engine for an RTX 3090 (82 tps single request, 672 peak), and yesterday's update took it to ~114 tps single-user / ~1,000 tps at 64 concurrent.

Today it's ~138 tps at default sampling on real chat prompts (up from ~124), 942 tps at 64 concurrent (re-measured today on the current stack), and the thing I'm actually happy about: a follow-up turn in a long chat now costs ~1 second instead of ~23.

What we had:

- fp8 KV cache, lm_head + embed_tokens int8, fp16 recurrent state, int8 activations, MTP-4 drafts with an own-output 40k draft head, GPTQ-int4 lm_head/MTP, split-KV verify attention, sampler patch, KVarN for 262k context

Now added:

- DFlash2 drafting. Inco published a block drafter for this exact model (5 layers, predicts 7 tokens in one non-autoregressive pass + a path selector). vLLM support is an unmerged PR on main, so I backported it to 0.27.1 and fixed what it silently relies on - including one real bug: 0.27.1 caches temperature-applied draft logits while main caches raw ones, so at 0<T≠1 the verify would have used the wrong proposal distribution. 2.8 → 3.3 tokens per step.

- The drafter requantized to W4A16. It's 3.85 GB in bf16, which on a 24 GB card is a net loss (106 tps). GPTQ int4 with Hessians captured from the drafter's own inputs on real traffic: 1.19 GB, no greedy acceptance loss, and that's what turns it into a win. Ships as python fetch_dflash2.py.

- Lookup-augmented drafting (my own idea - really happy about this one). A block drafter sees a 2,048-token window, but a long-context assistant spends much of its output reproducing what it was given... quoting a doc, repeating commands, rewriting a paragraph while keeping the code. Those tokens sit verbatim in the prompt, 20k tokens beyond what the drafter can see. So: one Triton kernel scans the request's own token history for the most recent occurrence of the last 6-12 generated tokens and proposes what followed. +29% tokens/step and 105 → 131 tps on "reproduce every command" work, +5% on ordinary chat, 0.075 ms per step. Stays exact and greedy never reads the draft distribution, and sampled positions get a point-mass q, which is a legal proposal for the rejection sampler.

- Prefix caching for a hybrid model. vLLM keeps it opt-in for mamba/GDN hybrids, so by default every chat turn re-prefills the whole conversation. Turned on with --mamba-cache-mode align (the recurrent state resumes from the last cached block boundary): 24k-token document, turn 2+ goes 23 s → 0.85-1.35 s, same answers token for token. In batch mode it's just as big: 64 requests sharing a 5,820-token system prompt take 222 s → 16.9 s (median latency 95 s → 8 s). Costs ~14-16% of the KV pool.

- 64k context with DFlash2, which needed an allocator fix: vLLM sizes a hybrid model's KV groups by the smallest layer bucket, so the drafter's 5 sliding-window layers made it pad the target's 16 attention layers to 20 and its 48 GDN layers to 50... 25% more memory per token, to pad the layers that weren't the problem. Padding the window group instead: 105 → 78 KB per token. Also made the V2 runner's CUDA-graph memory explicit; upstream it returns 0, so ~1.2 GB lands on top of whatever --gpu-memory-utilization you asked for.

- Docker. docker compose --profile single up -d - image pins vLLM 0.27.1 + all patches, a prepare step downloads and requantizes the model, and verify.sh runs at build.

Quality unchanged throughout (perplexity 8.09, GSM8K 96.5%) and speculative decoding is exact by construction and the state resume is exact too.

Caveats worth stating: DFlash2 is best for 1-4 concurrent users (each request reserves 8 recurrent-state slots, so MTP wins again at 8+ concurrent), and its 2,048-token window means MTP is still slightly ahead on long-context free-form prose. Both modes are one env var apart.

Repo: https://github.com/syv-ai/qwen38-27b-rtx3090

W4A16 DFlash2 drafter: https://huggingface.co/syvai/Qwen3.8-27B-DFlash2-W4A16

Fast-variant tensors: https://huggingface.co/syvai/qwen3.8-27b-3090-fast-variant

I said last time that was probably the last update. Then someone released a better drafter and I found two features that were switched off by default, so here we are. Lets see what happens next...

Two days ago I released a hyper-optimized Qwen3.8-27B inference engine for an RTX 3090 (82 tps single request, 672 peak) - yesterday's update took that to 99 tps single-user / ~1,000 tps at 64 concurrent.

Since then I've focused on the single-request number, again without quality degradation. It's now ~114 tps at default sampling and ~124 tps greedy (real chat prompts, not random tokens), up from 90 / 98.

What we had:

- fp8 KV cache, lm_head + embed_tokens int8, fp16 recurrent state, int8 activations, MTP-4 drafts with a 40k-token draft head, draft_sample_method=probabilistic

Now added:

- Draft vocabulary counted over the model's own outputs - the old web-text list covered about 92% of what the model generates (83% on code), and every miss is a forced rejection; the new one covers 97.5%. 98 → 109 tps greedy.

- GPTQ-int4 lm_head and MTP module, calibrated on the model's own hidden states: +0.6% PPL, GSM8K unchanged, acceptance intact, −1.8 ms per step. Ships as a "fast variant" (python fetch_fast_variant.py, ~1 GB from the Hub).

- Split-KV attention kernel for the verify step... FlashAttention-2 only splits KV for single-query decode, so with 4 drafts it used 24 of the 3090's 82 SMs. Small Triton kernel: 5× faster at 1.5k context, 10× at 16k.

- Sampler patch -§ sort-free top-k/top-p, multi-block softmax, drafts sampled from the target's truncated support: +4% at default sampling.

- KVarN 4/2-bit KV cache ported to vLLM 0.27.1: the full 262k context now fits, needle correct to 240k, +0.16% PPL, ~20% slower decode at 100k. Optional (KV=kvarn / CTX=huge).

- bench/run_benchmarks.sh + verify.sh to reproduce the tables and check the install is actually patched.

Peak concurrent throughput is unchanged (~1,000 tps at 64 concurrent). Speculative decoding is exact by construction, so the sampled distribution is the same as without it.

Repo: https://github.com/syv-ai/qwen38-27b-rtx3090

Fast-variant tensors: https://huggingface.co/syvai/qwen3.8-27b-3090-fast-variant

This is most likely the last update to the inference stack, unless some gigabrain comes up with something new I would like to test out. I've gotten pretty tired at finding miniscule gains here and there :)