Running Qwen3.8-27B dense fully on a single RTX 5060 Ti 16GB — ~45–47 tok/s
Pretty happy with this one.
I managed to get Qwen3.8-27B running fully on GPU on an RTX 5060 Ti 16 GB, while keeping a 32K context and MTP enabled.
Current setup:
● GPU: RTX 5060 Ti 16 GB
● Model: Qwen3.8-27B dense
● Quant: custom Unsloth IQ4_XS + importance matrix
● GGUF: Qwen3.8-27B-IQ4_XS-MTP-Q4_K.gguf
● Size: ~14.60 GiB
● Runtime: llama.cpp CUDA, sm_120a
● Full GPU offload
● Flash Attention
● CUDA Graphs
● --fit off
● Context: 32K
● KV cache: Q4
● MTP KV: Q4
● MTP-2
● Single slot
The only surgical modification I had to make to fit the whole thing was quantizing:
blk.64.nextn.eh_proj.weight
from Q8_0 → Q4_K, saving roughly 25 MiB. The rest of the target quant remains IQ4_XS.
Performance:
● No MTP: ~25.7 tok/s
● MTP-1: ~40.0 tok/s
● MTP-2: ~47.4–47.6 tok/s
● After a ~30K token prefill: ~45.7 tok/s
● 30K prefill speed: ~455 tok/s
So MTP-2 is giving me roughly an ~85% uplift over normal decoding.
What blows my mind is that this is not an MoE. It’s a 27B dense model, fully resident on a 16 GB consumer GPU, running at genuinely interactive speeds.
Given how strong Qwen3.8-27B seems to be in the early benchmarks, this is probably the first local setup I’ve had where I don’t feel like I’m making a major capability sacrifice just to stay local.
Curious to see what numbers other Blackwell 16 GB users are getting, especially with different quants / MTP settings.