I ported vLLM's serving stack to C++20: 66 MiB binary, no Python at inference, output checked token-for-token against vLLM
I'm the author, so discount the enthusiasm accordingly. This is an unaffiliated community port, not endorsed by the vLLM project, which it uses to verify its correctness.
What started it: I love vLLM, but a vLLM install here is 9.1 GiB of virtualenv, and I wanted to embed inference inside other software, on machines where having an interpreter in the process is a problem. And, honestly, Python dependencies have a different deployment story, in term of security (supply chain attacks), and bloat of Python itself. So vllm.cpp is vLLM's serving stack written from scratch in C++20. Nome TBD yet, calling it vllm.cpp until I have a better name.
Continuous batching, block-paged KV, automatic prefix caching, speculative decoding, an OpenAI-compatible server. It builds to a 66 MiB binary with no Python and no PyTorch at runtime.
The gate matters more to me than the size does. Every architecture is checked token-for-token against a pinned vLLM oracle on the same workload, and upstream's own test module gets ported in the same commit as the code. The ids have to match. 25 or so architectures so far. And yes, this project does extensive use of AI. I'm prepping follow-ups on how this is architectured (this is a port, which in some parts deviates, like support of MLX, Radix Attention, and such)
Speed, since it is the first question. You can see in the image that we are almost ties with vLLM on high concurrency. I've tested only on DGX Spark, Thor, and AGX Orin. Qwen3.6-27B NVFP4 on a DGX Spark (GB10), against vLLM in its production graphed config, medians of 3 interleaved reps, 1024 in / 128 out:
| concurrency | vllm.cpp | vLLM | ratio |
|---|---|---|---|
| 1 | 86.05 | 82.32 | 1.045x |
| 2 | 159.68 | 158.03 | 1.011x |
| 4 | 292.34 | 290.31 | 1.007x |
| 8 | 508.77 | 505.46 | 1.007x |
| 16 | 801.76 | 789.16 | 1.016x |
| 32 | 1095.01 | 1076.25 | 1.017x |
Nominally ahead everywhere, but our run to run noise is 0.5% and five of those six sit inside 1.7%. That is one win at c1 and five ties, and I would rather say it than have someone work it out in the comments. Output is identical at every point. Memory is the less ambiguous axis: peak GPU 40,996 MiB against 70,531, though vLLM pre-reserves a fixed fraction up front and we allocate what the workload needs, so it is a difference in footprint rather than a cheaper KV.
Some other numbers people usually ask for: 1.18x llama.cpp's prefill on the same GGUF file on CPU aarch64 with decode a tie, 97.6% of MLX-LM warm total on an M4, and DeepSeek-V4-Flash in 2-bit GGUF on one Spark at 18.69 tok/s, which is 1.14x the fastest GGUF engine I could find for it.
Speculative decoding is in: MTP takes c1 from 9.97 to 15.10 tok/s, DFlash from 10.16 to 29.32, both landing on top of vLLM running the same speculator.
It loads safetensors and GGUF, does NVFP4, k-quants and i-quants, fp8, bf16. CUDA sm_80 through sm_121a, CPU with AVX-512 and Arm i8mm, Metal, Vulkan partially. Model list is in the repo rather than pasted here. There are also some pieces of sglang, and ideas I always wanted to see in a cpp engine, such as radix attention and LPM aware cache scheduling.
What does not work: many things have to be built yet, model architectures, hardware support, no multi-GPU on real hardware (tensor parallel is proven equal to tp=1 on CPU, I have one box), LoRA is not wired through the server, multimodal runs in the CLI and library but not over the HTTP API, no embedding or reranking models, no ROCm. It is also under heavy development, so flags and internals move between commits. There is a stable surface, which is the versioned C ABI.
Help from the community to port to new architectures is welcome!
To start with it, build is cmake and nothing else:
cmake -S . -B build && cmake --build build -j # CPU
cmake -S . -B build-cuda -DVLLM_CPP_CUDA=ON -DVLLM_CPP_TRITON=ON # CUDA
cmake --build build-cuda -j
Apache 2.0. https://github.com/mudler/vllm.cpp
Benchmarks, methodology, and the rows we lose: https://github.com/mudler/vllm.cpp/blob/main/docs/BENCHMARKS.md
Happy to answer anything!
