Qwen3.8-27B on a 2019 Quadro RTX 6000 24GB: 24.36 tok/s Q4, Q8 does not fit, ngram 110+ is a lie, tested Cloud API too 155 toks/s!
I ran Qwen3.8-27B on the box a lot of us actually own or can still buy used: a 2019 Quadro RTX 6000($830-$950). Turing SM75. 24GB. No max clocks. This is the floor, not the flex.
Why bother
If you only read 119–200 tok/s threads you will think 27B is a 5090 toy. It is not. It is a 24GB dense model. I wanted the number for people on a 6000 / 3090 / used 24GB, not another screenshot from a card that costs more than the rest of the PC.
Recipe
llama.cpp
Qwen3.8-27B Q4_K_M, 15.40 GiB
-ngl 99 -fa on -b 512 -ub 512 -c 32768 -np 1
llama-bench tg128, thinking off: 24.36 tok/s
That is decode, not a chat anecdote. 24 tok/s is usable. It is not “type and wait.” It is also not the 5090 story. If someone tells you this card should do 80, they are mixing boxes.
Q4 vs Q6 vs Q8 on 24GB
Q4_K_M is the one I would actually leave loaded. 15.40 GiB leaves room for OS, context, and the card not dying.
Q6_K: 17.95 tok/s, 20.56 GiB. Quality bump. Speed tax. Still -ngl 99.
Q8 will not -ngl 99 on this card. Do not fight it. You will start offloading, decode falls off a cliff, and you will blame the model. It is the 24GB ceiling.
If you have 24GB and you want the model to feel like a daily driver, stay on Q4 or Q6. Q8 is a different class of card.
The number people will quote that is fake
Keep CUDA graphs on.
Ngram speculative at 110–201 tok/s on this box is a repeated-prefix cache hit. It is not a win. It is the bench agreeing with itself. If you post that as tok/s you are lying to the next person with a 6000.
I almost posted it. Then I looked at the prompt. Do not be me.
Who this is for
You have ~24GB and you want 27B local without buying a 5090 this week. 24 tok/s Q4 is the honest “yes, keep it loaded” line.
Who this is not for
12GB / 4070 class: Q4 plus RAM offload will boot. It will not feel like the model you just watched someone run at 100+ tok/s. You will hate the drop for the wrong reason.
If you have no GPU and you just want to poke the same 27B before you download 15GB
Same model on OrcaRouter, my numbers:
thinking off ~155 tok/s warm decode
thinking on 66.3, and most of that is reasoning tokens, not answers
later calls sat 37–118s in queue before token 1
That wait is not tok/s. Do not put it in a speed table.
Disclosure
I was passed a new-account $10 voucher, about 30M tokens, first 500 people. Base and uncensored. I get nothing if you use it. Not a referral. If you already have a 24GB card, ignore the link and run the GGUF.
https://www.orcarouter.ai/redeem/I-LOVE-ORCAROUTER
If you have a 3090 / 4090 / 6000 and a different quant, post the recipe. The 5090 thread does not need another “same.” This one does
Here’s my recipe for this old Turing RTX6000:
https://github.com/vcruz305/Qwen3.8-27B-Turing-RTX-6000
Here is my Turing RTX6000 repo with all the models and recommendations for it that I’ve tested!