u/Stock_Ad9641

I am doing a lot local AI, mostly creatively in audio and image subjects.
I currently rely on two 3090 cards that I bought used below 800$ each.

I think it was a good choice but other options exist, the 4090 is faster than both 3090 on paper. Though I did not see one offered fair priced.

There is a lot, 5090 and 60ti, DGX, Macs

I wonder what do you guys have, and would you go that path again?

I read it everywhere: fp8 quantization and fp8 kv cache are being used.
This is good for Gemma but Qwen will run perfectly fine with 4-5 bit.

I ran it in fp, 8bit and 4 bit and it is behaving very similar.

The difference is night and day - at fp8 you can barely fit 27B on a 4000$ gpu.
At 4bit it runs on a 3090 - and due to the smaller context size it runs faster , less vram IO needed

So do yourself a favor and run Qwen 3.6 in smaller bit quantizations