Would vLLM help in my setup? (Single GPU + CPU offload, multi-user)
I have an RTX 3060 12GB and 16GB DDR5 RAM. Since my GPU only has 12GB VRAM, I usually offload part of the model to the CPU.
I mainly use llama.cpp with **Qwen3-36B-A3B-MXFP4 (Unsloth quant)**and MTP enabled. I currently get around 17–20 tok/s , and about 25 tok/s after disabling Flash Attention (IDK seems unusual) and tweaking a few flags (though my setup is probably still not well optimized).
My workload is mostly:
Daily AI assistant use
Coding
Multi-user inference (typically 4–5 concurrent users at peak)
Given that I’m using a single RTX 3060 with CPU offloading ,would switching to vLLM provide any real benefit for multi-user serving? I’m fine with INT4 or other quantized weights if supported.
From what I understand, vLLM is great for batching and concurrent requests, but I’m unsure whether that advantage still applies when a significant portion of the model is CPU-offloaded due to limited VRAM.
Has anyone compared llama.cpp vs vLLM in a setup like this?