1 trillion parameters, 49B active, definitely chonky! If you don’t love the model you gotta at least love the humor in the name - Le Chonk
I run vLLM on 8x 3090s (Qwen 3.8 Flash Next) mostly so my partner and family could stop pasting personal stuff into ChatGPT. Getting the model serving was the easy part; the hard part turned out to be the last mile: a chat UI that non-technical people would actually use.
What I learnt getting there:
Concurrency is felt immediately. With 2+ people chatting at once, per-user tok/s drops and they notice before they can articulate it. MoE helps (Flash Next stays responsive), but plan your VRAM budget around peak household load, not single-user benchmarks.
Non-technical users judge latency differently. They don't care about tok/s, they care that the first token shows up fast and the UI doesn't look like a dev dashboard. Open WebUI is powerful but the knob density alone made my family bounce.
Self-hosted voice/search is a deployment tax. STT + TTS + SearXNG are three separate services to deploy and wire. That friction is why most "local AI for the family" projects die at setup.
So I ended up building my own UI layer: chat-first, bundles Parakeet STT / Kokoro TTS / SearXNG and connects them via the installer, talks to vLLM's OpenAI-compatible endpoint. Native clients everywhere: web, Android, iOS, and desktop (macOS Apple Silicon/Intel + Linux). It's been in daily use by multiple non-technical users for months.
One thing I didn't expect to need but now use daily: Agent Connections. You can hook Codex, Claude Code, OpenCode, or Hermes to it through the Host Connector, so the coding agents run on your own machine (or an SSH host) and you drive them from the same chat UI, with plans/tool calls/approvals visible. My family gets the chat, I get my agents, same server.
MIT licensed, no telemetry: https://github.com/yoloyash/overtchat
Happy to answer questions about the serving setup (3090s, Flash Next on vLLM, concurrency behaviour) or the UI decisions. What do you all use for the last mile in your own setups?
Hey all, I've been building Glyd, a lossless compression layer for model weights on NVIDIA GPUs. Qwen models are what I test on most, so this felt like the right place to share it.
The idea: a bf16 weight only carries about 11 bits of real information, so you can store the exact same model in about a third less GPU memory and get every weight back exactly. It's not quantization and nothing gets rounded. There's also an exact mode that matches bf16's outputs bit for bit.
What it gets you with Qwen (all measured, logs are public):
- Qwen3-8B runs on a 16 GB card (bf16 can't load it)
- Qwen3-32B fits on one 48 GB GPU instead of two
- Qwen3-8B on an L4 with vLLM: 1.59x the requests/sec vs bf16 (weights plus our lossless KV cache, 2.64x the KV tokens)
- Qwen3-14B on an A100 40GB: 1.28x req/s
- Qwen2.5-72B on 2x H100: 4.07x req/s, 12.65x the KV cache
Try it:
```
curl -LsSf https://getglyd.com/install.sh | sh
glyd run Qwen/Qwen3-8B
```
Or with vLLM: `vllm serve Qwen/Qwen3-8B --quantization glyd`
Honest caveats: it's Linux + NVIDIA (Ampere or newer) and bf16 checkpoints only. On GH200 it's a bit slower than bf16 at full load right now, and MoE (Qwen3-30B-A3B) is still slower at full load; a fix for that is coming in the next release. The codec is open source. The GPU part ships compiled and is free for personal and research use (business source license).
GitHub: https://github.com/surya-koritala/Glyd
Benchmarks + logs: https://getglyd.com/benchmarks
Would love feedback, especially which Qwen models or GPUs you'd want numbers on next.