u/kristiyanstoyanovAI

Been diving into speculative decoding lately and wanted to share some real numbers instead of just theory, since most explainers stop at "draft model proposes, target model verifies" without showing what happens when you actually run it.

Setup: Qwen3.5-122B-A10B (hybrid INT4/FP8 checkpoint) on a single DGX Spark, tested against the same prompt across all methods, vLLM logs for acceptance rate + throughput.

Baseline (no spec decoding): ~37 tok/s

N-gram: Actually came in below baseline (~30 tok/s). No neural component — it just matches repeated token sequences in context. On general prompts the acceptance rate was low enough that the drafting overhead cost more than it saved. Would probably do better on highly repetitive tasks like doc summarization, but not a general win.

MTP (native multi-token prediction head): This is where it got interesting. The model ships with an MTP head, but the INT4 checkpoint I used doesn't register those weights in its index file by default — needed a patch to get vLLM to actually load and use it (shoutout to the recipe author for documenting that). Once patched:

1 token: ~44-46 tok/s, ~85% acceptance

2 tokens: ~48-50 tok/s peak, this was the sweet spot

4 tokens: dropped back down — acceptance rate falls off hard the further out the head predicts, so the overhead of drafting/verifying/discarding stopped being worth it

EAGLE3: Couldn't find a good EAGLE3 drafter for the 122B model specifically, so I ran this on a smaller Qwen (30B-A3B) instead. At 1 speculative token, went from ~33 tok/s baseline to ~38-40 tok/s. Pushed it to 12 speculative tokens just to see — acceptance rate dropped to 8-11% and throughput roughly halved, even though the drafter itself was cranking out 85+ tok/s in draft speed. Perfect illustration of why raw draft speed doesn't matter if acceptance is low.

DFlash: The most novel of the four — instead of autoregressive drafting, it uses block diffusion to generate a whole chunk of tokens (I ran 16) in a single non-autoregressive pass. In theory this should sidestep the "overhead compounds with more speculative tokens" problem entirely. In practice, it was volatile — swung between 30-52 tok/s on general prompts, and even on a coding-focused test (where I expected structured/predictable output to shine) it peaked in the mid-40s rather than the ~80 tok/s I'd seen in some published benchmarks. My take: it's a genuinely exciting technique in an early state — MTP is more consistent/production-ready today, but DFlash's approach (non-autoregressive drafting) seems like the more scalable long-term direction.

Here are my results on all the tests ran.

Final Results

Some of my takeaways from the tests:

• Acceptance rate is everything. More speculative tokens are not always better. The acceptance rate at each position compounds — even a single bad position in an autoregressive chain invalidates all subsequent drafts.
• DFlash's parallel drafting breaks the serial constraint. By generating blocks in a single forward pass, DFlash eliminates the per-token cost overhead that limits autoregressive drafters at high K. This is the architectural reason its peak throughput is so high.
• Task type matters. Structured output (code, JSON, math) consistently benefits more from speculative decoding than open-ended generation, because the conditional distribution is sharper and easier for any drafter to model.

Curious if anyone's gotten more consistent DFlash numbers, or found a good EAGLE3 drafter for larger MoE models like the Qwen-122b-a10b.

I noticed I was wasting a lot of tokens by using my agent like a very patient junior engineer: I’d ask for the same kind of thing multiple times, and every time it would go off, search around, reason through the steps again, and eventually get there.

What’s worked better for me is treating recurring tasks differently. If the problem is already understood, I try to turn it into a small script or tool, verify it, and then let the agent reuse that instead of re-figuring it out every session.

The basic idea is: use inference for decisions, not repetition.

That alone has made a noticeable difference in token usage, speed, and reliability for me. The agent is still useful for deciding what to do, but it doesn’t need to burn context on how to do something that’s already solved.

Feels obvious in hindsight, but I think a lot of us are still overusing intelligence where simple automation would do the job better.

Any other cool and low-hanging fruit optimizations you have noticed?
Any