u/DontWinFrensWthSalad

I asked my Q3 K6V4 qwen 27b model to make me a human head in WebGL from scratch with no libraries and this was the result. What a time to be alive. I did this with two 3060tis.

I was having FOMO when 3.8 dropped so spent the weekend figuring out how to get two 3060tis (8gb each) to fit a 27B quant.

I started with Unsloth IQ4_XS GGUF in llama.cpp. A single card with partial offload was only ~2.4 t/s. Dual GPU got me ~7–8 t/s initially, then ~17.8 t/s once I figured out that tiny tensor-split changes mattered enormously. A 1.02,0.98 split was enough to fit all layers at 32K where 1,1 could not.

That taught me lesson #1: with tiny VRAM margins, “16GB total” is misleading. The limiting GPU and memory placement matter more than aggregate VRAM.

Then I tried EXL3 3.5bpw with ExLlamaV3/TabbyAPI. That was the real breakthrough because the model size was smaller so I could fit more context. Also tensor parallelism, despite one GPU being stuck in a PCIe x2 slot, gave ~25–30 t/s instead of ~18.

I initially assumed 3.5bpw would be a substantial quality downgrade from IQ4_XS, but EXL3 isn’t equivalent to uniformly quantizing every tensor to 3.5 bits. It uses non-uniform/mixed allocation and protects sensitive tensors, so the effective quality is much better than the number suggests. In my initial coding/reasoning spot checks I couldn’t see an obvious regression from IQ4_XS, but I’m still doing proper head-to-head benchmarks before making stronger quality claims.

MTP was even crazier: ~55–60 t/s on coding workloads. Unfortunately Qwen’s hybrid Gated DeltaNet architecture + MTP + multi-GPU exposed recurrent-state/TP bugs. It benchmarked beautifully but was too fragile for a persistent agent: recurrent-slot failures, freezes, and synchronization wedges. Also with MTP context window got absolutely crushed down to 40k.

Without MTP, here are my results.

80K K6/V4 ~30 t/s

100K K6/V4 ~24 t/s

112K K6/V4 ~24.7 t/s

I wired it up to Pi and asked Qwen to make me an animated SVG antfarm. Here is a still shot, but the ants do in fact carry food into the anthill and gracefully fade into darkness, and re-emerge without food. The clouds move too. The only obvious issue is the shade is on the wrong half of the hill. It took about 50k tokens to plan and execute.

All in all I'm pretty impressed even with my gimped hardware.