I built a zero-VRAM speculative decoding engine that runs 1.2x faster on consumer GPUs — no second model needed
Hey everyone,
I've been working on a speculative decoding engine called Structspec that makes local LLMs generate code faster without needing a second model in VRAM.
The idea is simple: instead of loading a draft model, it mines token patterns from a code corpus and combines them with syntax-aware rules (indentation,
brackets, keyword transitions). These propose draft tokens that get verified in a single pass against the real model.
Tested on Qwen2.5-Coder-7B with an RTX 4050:
- ~1.2x wall-clock speedup
- 100% draft acceptance on some prompts
- Zero extra VRAM used
The part I'm most excited about is something I called SymbolicMotifCache — it abstracts code patterns across variable names. So `current = current.next`
and `node = node.left` get recognized as the same underlying pattern. I think this could be useful beyond just code generation but I'm still figuring out
the limits.
I have a few ideas to push this further — better pattern generalization, support for more languages, and combining this with quantization-aware
techniques. Still learning a lot about the inference optimization space.
If this sounds interesting, a star on the repo would mean a lot — I'm a student trying to build up my portfolio and every bit of visibility helps.
Repo: https://github.com/neerajdad123-byte/zero-vram-spec
Would love to hear feedback or suggestions. Happy to answer any questions about how it works.
