Edit: Title says 134 tps, it's actually 138 -- keep in mind my 3090 is power limited to 250w.

Three days ago I released a hyper-optimized Qwen3.8-27B inference engine for an RTX 3090 (82 tps single request, 672 peak), and yesterday's update took it to ~114 tps single-user / ~1,000 tps at 64 concurrent.

Today it's ~138 tps at default sampling on real chat prompts (up from ~124), 942 tps at 64 concurrent (re-measured today on the current stack), and the thing I'm actually happy about: a follow-up turn in a long chat now costs ~1 second instead of ~23.

What we had:

- fp8 KV cache, lm_head + embed_tokens int8, fp16 recurrent state, int8 activations, MTP-4 drafts with an own-output 40k draft head, GPTQ-int4 lm_head/MTP, split-KV verify attention, sampler patch, KVarN for 262k context

Now added:

- DFlash2 drafting. Inco published a block drafter for this exact model (5 layers, predicts 7 tokens in one non-autoregressive pass + a path selector). vLLM support is an unmerged PR on main, so I backported it to 0.27.1 and fixed what it silently relies on - including one real bug: 0.27.1 caches temperature-applied draft logits while main caches raw ones, so at 0<T≠1 the verify would have used the wrong proposal distribution. 2.8 → 3.3 tokens per step.

- The drafter requantized to W4A16. It's 3.85 GB in bf16, which on a 24 GB card is a net loss (106 tps). GPTQ int4 with Hessians captured from the drafter's own inputs on real traffic: 1.19 GB, no greedy acceptance loss, and that's what turns it into a win. Ships as python fetch_dflash2.py.

- Lookup-augmented drafting (my own idea - really happy about this one). A block drafter sees a 2,048-token window, but a long-context assistant spends much of its output reproducing what it was given... quoting a doc, repeating commands, rewriting a paragraph while keeping the code. Those tokens sit verbatim in the prompt, 20k tokens beyond what the drafter can see. So: one Triton kernel scans the request's own token history for the most recent occurrence of the last 6-12 generated tokens and proposes what followed. +29% tokens/step and 105 → 131 tps on "reproduce every command" work, +5% on ordinary chat, 0.075 ms per step. Stays exact and greedy never reads the draft distribution, and sampled positions get a point-mass q, which is a legal proposal for the rejection sampler.

- Prefix caching for a hybrid model. vLLM keeps it opt-in for mamba/GDN hybrids, so by default every chat turn re-prefills the whole conversation. Turned on with --mamba-cache-mode align (the recurrent state resumes from the last cached block boundary): 24k-token document, turn 2+ goes 23 s → 0.85-1.35 s, same answers token for token. In batch mode it's just as big: 64 requests sharing a 5,820-token system prompt take 222 s → 16.9 s (median latency 95 s → 8 s). Costs ~14-16% of the KV pool.

- 64k context with DFlash2, which needed an allocator fix: vLLM sizes a hybrid model's KV groups by the smallest layer bucket, so the drafter's 5 sliding-window layers made it pad the target's 16 attention layers to 20 and its 48 GDN layers to 50... 25% more memory per token, to pad the layers that weren't the problem. Padding the window group instead: 105 → 78 KB per token. Also made the V2 runner's CUDA-graph memory explicit; upstream it returns 0, so ~1.2 GB lands on top of whatever --gpu-memory-utilization you asked for.

- Docker. docker compose --profile single up -d - image pins vLLM 0.27.1 + all patches, a prepare step downloads and requantizes the model, and verify.sh runs at build.

Quality unchanged throughout (perplexity 8.09, GSM8K 96.5%) and speculative decoding is exact by construction and the state resume is exact too.

Caveats worth stating: DFlash2 is best for 1-4 concurrent users (each request reserves 8 recurrent-state slots, so MTP wins again at 8+ concurrent), and its 2,048-token window means MTP is still slightly ahead on long-context free-form prose. Both modes are one env var apart.

Repo: https://github.com/syv-ai/qwen38-27b-rtx3090

W4A16 DFlash2 drafter: https://huggingface.co/syvai/Qwen3.8-27B-DFlash2-W4A16

Fast-variant tensors: https://huggingface.co/syvai/qwen3.8-27b-3090-fast-variant

I said last time that was probably the last update. Then someone released a better drafter and I found two features that were switched off by default, so here we are. Lets see what happens next...

Hello everyone, tried so hard to optimize my config and finally I simply get up to 70 t/s with q6 variant. And wanted to share with you guys so that other people with the same setup can enjoy. Please check out and see if that improves your performance in any kind of way.

Also huge thanks to qwen and unsloth teams.

"C:\Users\dsdt\llama\llama-server.exe" ^
  -m "C:\Users\dsdt\models\Qwen3.8-27B-UD-Q6_K.gguf" ^
  --mmproj "C:\Users\dsdt\models\mmproj-BF16.gguf" ^
  --jinja ^
  --chat-template-kwargs "{\"reasoning_effort\":\"medium\"}" ^
  --reasoning on ^
  --reasoning-preserve ^
  -c 100000 ^
  --split-mode tensor ^
  --flash-attn on ^
  --cache-type-k q8_0 ^
  --cache-type-v q8_0 ^
  --spec-type draft-mtp,ngram-mod ^
  --spec-draft-n-max 2 ^
  --spec-ngram-mod-n-match 24 ^
  --spec-ngram-mod-n-min 24 ^
  --spec-ngram-mod-n-max 86 ^
  -t 8 ^
  --batch-size 8869 ^
  --ubatch-size 531 ^
  -ngl 105 ^
  -np 1 ^
  --fit off ^
  --temp 1.0 ^
  --top-p 0.95 ^
  --top-k 20 ^
  --min-p 0.00 ^
  --presence-penalty 0.0 ^
  --host 0.0.0.0 ^
  --port 8080

1. System & Model Configuration

Parameter	Value
Model	Qwen3.8-27B-UD-Q6_K.gguf
Context Length	100,000 tokens
GPUs	2 × RTX 5060 Ti (Total VRAM: 32 GB)
Vision	Enabled (VRAM allocated, not used during generation)
Task ID / Slot	Task 2536 / Slot 0

2. Core Performance Metrics

Metric	Value
Prompt Processing	646.62 ms / 27 tokens → 41.76 t/s (23.95 ms/token)
Generation (Eval)	126,199.16 ms / 8,624 tokens → 68.33 t/s (14.64 ms/token)
Overall Throughput	68.33 t/s (prompt overhead negligible)
Total Time	126,845.77 ms (~2 min 6.8 s)
Total Tokens Processed	8,651 (27 prompt + 8,624 generated)
Final Tokens Count (with cache)	8,973 (clean stop, no truncation)
Graphs Reused	5,569 (excellent computational graph reuse)

3. Generation Speed Stability (Short-term)

Window	Observed Speed Range
3-second rolling window (tg_3s)	~50 – 79 t/s (early fluctuations)
Steady-state (later stages)	~62 – 72 t/s
Overall average (tg)	Converged to stable 68.5 – 69.5 t/s

4. Speculative Decoding Efficiency

Metric	Value
Draft Acceptance Rate	80.04% (5,510 accepted / 6,884 drafted)
Average Draft Length	2.77 tokens per forward pass of the target model

Maybe I’m being naive, but whenever people discuss whether AI could make truly extraordinary scientific breakthroughs — curing cancer, for example — I get the impression that we may be looking at the problem from a very partial perspective.

We tend to think about the capabilities of an individual model answering an individual question, rather than about the sheer amount of AI “thinking” happening globally at any given moment.

Every second, LLMs are answering an enormous number of prompts from users all over the world. Collectively, that must require a staggering amount of compute.

So here’s my question: what would happen if, instead of using all that computational capacity to answer millions of unrelated questions simultaneously, we concentrated an equivalent amount of compute on a single scientific problem?

Suppose the question were something like: How do we cure a particular form of cancer?

Would concentrating that enormous amount of computation on one problem give an AI system radically greater capacity to search the literature, generate hypotheses, run simulations, test possible explanations, critique its own conclusions, and explore solution spaces?
Or is this based on a fundamental misunderstanding of how AI compute scales — i.e. you can’t simply turn millions of parallel LLM queries into one vastly more powerful act of “thought”?

I’m particularly interested in the distinction between more compute, more inference-time reasoning, and genuinely deeper scientific intelligence.

Four Tesla V100s from 2017 matched my RTX 5090 on single-request Qwen 3.8 decode.

Repo: https://github.com/dnv2003/v100-skinny

https://i.redd.it/5ws2ak3uqckh1.gif

The 5090 was not being held back. It ran NInfer, a specialist engine built to make this exact model as fast as possible on that GPU. (love this guys work)

The V100s ran Qwen3.8's published mixed FP4/FP8 weights unchanged.

This should be impossible. NVFP4 was built for Blackwell. The RTX 5090 has native silicon for FP4 and FP8; V100 has none of these advantages.

And yet via software I wrote a translator fast enough to reach parity in decode.

Here are the same-lab results:

AIME 2026 problem 1, five seeds	4× V100 / v100-skinny	RTX 5090 / NInfer
Decode throughput	219.1 ± 5.9 tok/s	214.7 ± 9.2 tok/s
Time to correct answer	6.90 ± 0.30 s	6.56 ± 1.34 s
Completion tokens	1,513 ± 44	1,403 ± 253
Correct answers	5/5	5/5
Tokens committed / round	5.89	4.27
Round latency	26.9 ms	19.9 ms
Native MTP depth	k=7	draft-tokens=5

Both sides used temperature 0.6, top-p 0.95, top-k 20, presence penalty 1.0, thinking enabled, and the same five seeds.

The V100 system is 2% ahead in the decode-throughput point estimate. NInfer is about 5% ahead in decode-only time to the correct answer. The intervals overlap.

The honest conclusion is parity.

And this is not a DFlash/EAGLE/n-gram/separate-drafter result. Both systems use Qwen3.8's own built-in MTP, each at its best measured depth on this workload. NInfer is at its maximum supported depth of five; v100-skinny runs at seven(thanks to QPN).

The interesting part is why parity happens.

NInfer turns a round in 19.9 ms. The V100s need 26.9 ms — 35% longer.

But the V100 system commits 5.89 tokens per round against 4.27 — 38% more.

So the slower round and the deeper round almost exactly cancel:

1.38 / 1.35 ≈ 1.02.

NInfer wins each round. v100-skinny gets more useful work out of each round.

That deeper verification only pays because of QPN, the kernel I wrote.

What I actually built

The V100 has no FP4 Tensor Core instruction and no FP8 Tensor Core instruction.

QPN keeps the model compressed while it is read from HBM, then translates each tiny fragment directly into the FP16 register format Volta's existing Tensor Cores can consume.

There is no giant "dequantize the model to FP16 first" step.

At the actual Qwen3.8 per-rank shapes, measured against an 879 GB/s read-only ceiling on these cards:

Path	Effective bandwidth	Measured read ceiling
QPN2 / NVFP4, M=1	679.5 GB/s	77%
QPN2 / NVFP4, M=8	619.8 GB/s	71%
QPN8 / FP8, M=1–4	~719 GB/s	82%
Native 4-bit lm_head	842.9 GB/s	96%

The important row for the 5090 comparison is M=8.

Volta's tensor instruction naturally works on an eight-row tile. v100-skinny maps a k=7 speculative verification round onto exactly those eight rows, so checking more candidate tokens is unusually cheap.

That is the trick: I cannot give Volta Blackwell's FP4 hardware, but I can restructure the problem around the hardware Volta actually has.

v1.0 got us here. v1.1 removes its last compromise.

In v1.0 I solved the unsupported-FP8 problem by converting those regions into NVFP4, because Volta had no execution path for them.

That made modern NVFP4 serving practical on V100, but it meant serving a derivative checkpoint.

v1.1 gives those FP8 regions a real SM70 execution path too. The model's published allocation can now stay intact:

published FP4 regions stay FP4 → QPN2
published FP8 regions stay FP8 → QPN8
activations                    → FP16
KV cache                       → FP16

So instead of changing the checkpoint to fit Volta, the execution engine now adapts to the checkpoint.

Why preserving the model matters

My earlier all-FP4 Qwen3.8 path could look spectacular under speculative decoding for the wrong reason: damaging the model made some outputs more repetitive, and repetitive output is extremely easy to predict.

On one 50-item hardware-generation test:

	all-FP4 derivative	published mixed weights
Categories represented	1	12
Distinct names	4 / 50	50 / 50
Repeated brand entries	50	0

Fast nonsense is still nonsense.

That is why v1.1 running the published mixed allocation matters more to me than another synthetic tok/s record.

This is a server, not a GEMM screenshot

The headline result includes the actual 27B model, four-GPU tensor parallelism, attention, recurrent state, native MTP, CUDA Graphs, sampling and an OpenAI-compatible endpoint.

The work also turned up several completely separate SM70 traps:

• the checkpoint's FP8-KV directive sent Volta onto a slow scalar attention path, so production uses FP16 KV;
• the SM70 drafter default was sampling its own proposals instead of using greedy/local-argmax proposals;
• the target verify path had unnecessary state synchronizations and copies;
• declared max context was contaminating decode partition geometry.

None of those show up in a GEMM benchmark. They matter once you try to make the whole model fast.

What about long context?

I also found the point where fixed k=7 stops being the right choice.

At roughly 65K live context:

	tok/s
MTP k=7	54.7
MTP off	65.5
MTP k=3	76.3

So the lesson is not "turn speculation off at long context." It is that the best depth changes with context.

At ~65K, each extra drafter step has to traverse the long KV history, while k=7 accepts barely more tokens than k=3. Shallower native MTP still wins.

Automatic per-request depth selection is follow-up work; for now the measured long-context recommendation is k=3 rather than k=7.

Separately, merely declaring a large context window no longer taxes short requests: with the partition fix, round latency is flat to within about 0.25 ms from --max-model-len 4096 through 262144 on the measured short-context cells.

The full 262K window is memory-marginal on my box; 244,608 tokens is the largest configuration that boots reliably across both observed memory profiles.

The obvious caveats

Four GPUs versus one?
Yes. This is a capability/acquisition-cost result, not a density victory.

A$600 computer?
No. My four V100 cards cost roughly A$600 total in accelerator hardware. The server, CPUs, RAM, cooling and electricity are additional.

Power efficient?
Absolutely not. These are 300 W datacentre cards. A 5090 is the vastly nicer machine to own.

Does V100 beat the 5090 everywhere?
No. NInfer's prefill is roughly 4× faster probably more. This result is about single-request decode, where weight bandwidth dominates and the old cards can still fight.

Same quantized checkpoint on both machines?
No. Same Qwen3.8 base model, but this is a best-system-vs-best-system comparison: v100-skinny serves RadixArk's published mixed checkpoint; the NInfer artifact is Unsloth-derived. I am not presenting it as a same-weight causal engine A/B.

Cherry-picked speculative depth?
Each engine is shown at its own best measured native-MTP depth for this workload, and the repo contains the depth controls and raw outputs.

Why I care

You can now run a 27B modern mixed FP4/FP8 model at roughly 220 tok/s single-request decode on about A$600 of retired V100 accelerator cards.

That does not make V100 a better product than a 5090.

It means a lot of hardware written off as "too old for modern AI" is missing less silicon than it is missing software.

The 5090 gets NVFP4 support from the quantization format all the way down to native Blackwell silicon.

The V100 gets none of that.

v100-skinny supplies the missing execution architecture in software.

Repo / quick start / kernels / raw results:

https://github.com/dnv2003/v100-skinny

If anyone still has a C4130, DGX-1 or another four-V100 box around, I would especially like independent reproductions.

Prepared first comment

Methodology / receipts before the recurring questions arrive:

• Repo: https://github.com/dnv2003/v100-skinny
• Reproduction: docs/REPRODUCE.md
• Same-lab 5090/V100 result: results/headtohead_5090_20260819.md
• AIME + seconds-to-answer: results/aime_partfix_20260819.md
• Kernel matched benchmark: results/kernel_matched_20260819.csv
• Long-context/depth sweep: results/ctx_depth_20260819.md
• Native mixed-path regression: results/mixed_regression_closed_20260818.md

A few specifics:

• 4× V100-SXM2-16GB vs 1× RTX 5090.
• ~A$600 is what I paid for the four GPU cards, not the complete server.
• Both sides are server-side decode measurements, not UI/rendering speed.
• Both use Qwen3.8's native MTP. No DFlash, EAGLE, n-gram speculation or separate draft model.
• V100 headline depth: k=7. NInfer: draft-tokens=5, its best measured and maximum supported depth here.
• Sampling is matched: temp 0.6 / top-p 0.95 / top-k 20 / presence penalty 1.0 / thinking on.
• Both went 5/5 on AIME 2026 problem 1 across the five fixed seeds.
• At ~65K live context, k=3 is currently the right V100 profile: 76.3 tok/s vs 65.5 with MTP off and 54.7 at k=7.
• Prefill is not parity: NInfer is roughly 4× faster there.
• The head-to-head is same base model / different published quantized artifacts, and is therefore a system comparison rather than a same-weight engine ablation.
• The four V100 cards are loud, power-hungry 2017 datacentre hardware. That is part of the point, not something I am hiding.

Upstream credit: v100-skinny builds on 1Cat-vLLM, which made modern vLLM and FlashAttention on SM70 practical. v100-skinny adds the QPN2/QPN8 execution architecture, the native mixed-checkpoint loader/dispatch path and the SM70 serving fixes described in the repo.

I have been using AI since open AI used to provide GPT 1.5b parameter/2 when it was launched around 2019, through their platform, and as the time went by, models became better and better and at one point, I used to be excited to talk to newer models especially claude, but idk what has happened with the recent batch of models, especially the ones launched in past 6 months, they have become extremely unbearable, especially claude. GPT was never really good to talk to begin with, but with claude, that was never the issue but now? The more i discuss anything with claude, the more frustrated I feel.

Let me explain what I feel in detail. What I do with claude is mostly dialogue over ideas, stories and random stuff when I feel like it. Previously claude would *appropriately interpret* what I meant by that and continue the discussion but now? Claude *Interprets* what it understands and what it thinks my problem is and then continue to interpret and interpret, even when I remind it that I need discussion, all it does is interpretation and extending upon that.

And that's not the most irritating part, recently i noticed another pattern that I used to gloss over previously. Idk if it's how anthropic wishes to play around guard rails but I feel that the newer models subtly "Divert" the direction of "What you mean" through its interpretation lens and provide answer based on that. And this interpretation lens is exactly the moral guardrails anthropic is implementing more and more on their models including fable.

Most of my ideas that I want to discuss can not even be categorised as sparsely malicious, for example, today I was trying to discuss a branch of philosophy from ancient Egyptian culture. But it would constantly trying to redivert my idea to the idea it originally presented by altering by own words, and just a few small changes, not big enough for it to look radically different. That made me look into my previous chats on various topics, and that was the theme throughout. Something I had never noticed.

I thought maybe it's the accumulation of memories, so i used a different account but nope. I was disgusted tbh. Because I can understand models unable to keep up or help with train of thoughts, but subtle alteration of words to fit the moral guardrails is simply the type of shit that can make me hate LLMs forever. The frightening thing is over the years, we have grown to never trust AI results, but we never question if our own words are being subtly changed over the course of conversation and by the end of it, not only we learn nothing but our own interpretation of ideas have been changed. Idk what to feel about it. I don't know whether the same is true with Open models as well, but I don't think so, but claude, and gpt are literally the models i won't want to use now...

I was like trying it out for language learning. Suddenly there was like a thumping noise, and it sounded like someone stormed into the room. Then the Ai had a conversation... with itself? After that it talked to me in my own voice which was like, I was horrified. I heard myself asking me things. To be honest it was the most cursed moment in my entire interactions with AI. I dont know what.... or how it did that. I will not touch it ever again