u/anitamaxwynnn69

I run vLLM on 8x 3090s (Qwen 3.8 Flash Next) mostly so my partner and family could stop pasting personal stuff into ChatGPT. Getting the model serving was the easy part; the hard part turned out to be the last mile: a chat UI that non-technical people would actually use.

What I learnt getting there:

Concurrency is felt immediately. With 2+ people chatting at once, per-user tok/s drops and they notice before they can articulate it. MoE helps (Flash Next stays responsive), but plan your VRAM budget around peak household load, not single-user benchmarks.

Non-technical users judge latency differently. They don't care about tok/s, they care that the first token shows up fast and the UI doesn't look like a dev dashboard. Open WebUI is powerful but the knob density alone made my family bounce.

Self-hosted voice/search is a deployment tax. STT + TTS + SearXNG are three separate services to deploy and wire. That friction is why most "local AI for the family" projects die at setup.

So I ended up building my own UI layer: chat-first, bundles Parakeet STT / Kokoro TTS / SearXNG and connects them via the installer, talks to vLLM's OpenAI-compatible endpoint. Native clients everywhere: web, Android, iOS, and desktop (macOS Apple Silicon/Intel + Linux). It's been in daily use by multiple non-technical users for months.

One thing I didn't expect to need but now use daily: Agent Connections. You can hook Codex, Claude Code, OpenCode, or Hermes to it through the Host Connector, so the coding agents run on your own machine (or an SSH host) and you drive them from the same chat UI, with plans/tool calls/approvals visible. My family gets the chat, I get my agents, same server.

MIT licensed, no telemetry: https://github.com/yoloyash/overtchat

Happy to answer questions about the serving setup (3090s, Flash Next on vLLM, concurrency behaviour) or the UI decisions. What do you all use for the last mile in your own setups?

Hey everyone, I need 7 testers for my app, overtchat. It is a lightweight, open-source chat client for self-hosted LLMs.

I will test your app back immediately. Just leave your links in the comments below. How to join:

1. Join Group: https://groups.google.com/g/overtchat-testers2. Web Invite: https://play.google.com/apps/testing/com.overtchat.mobile3. Play Store: https://play.google.com/store/apps/details?id=com.overtchat.mobile

I see a ton of terrible Corsair support stories on here. I wanted to share a different experience I had just to balance it out.

Had a Corsair Vengeance RGB 64GB (2x32GB DDR4) that was gifted to me years ago. One stick gave my pc reboots since day #1 so I was using it on and off. Always thought it was some compatibility issues with my mobo. Got to know about MemTest and tested it out a few weeks ago - it would abort at 58% from sheer error count lol. Kit was probably 3-4 years old easy. The person who gifted me this was no longer a part of my life and there was no way to reach out to them for receipts.

Submitted a ticket in late April with no expectations.

Timeline:

• Apr 22: Submitted ticket, explained everything honestly.
• Apr 23: Was asked for some pictures and some other information. RMA approved within 24 hours.
• Apr 25: Agent Wayne L (GOAT) confirmed everything, and kept me in the loop
• Shipped the bad kit back somewhere around Apr 28
• May 4: Replacement shipped
• May 7: Delivered at my door (3 days from Taiwan to WA, US lol)

Genuinely hope y'all also have good experiences like this 😸 (not sponsored)

https://preview.redd.it/8o43bjhe9d1h1.png?width=5346&format=png&auto=webp&s=1c87c2ee8b8ffff43495f543266056b0e26d3947

In another post I had someone ask me about the power draw of the 4x 3090 setup so I'm sharing a a full test I conducted to understand the efficiency curve. Used this blog post (not mine) as a reference.

Setup:

• GPUs: 4x RTX 3090 (Dell OEM, EVGA XC3, 2x ASUS Strix)
• PCIe Topology: Gen 3 (Bifurcated: x16 / x8 / x8 / x4)
• Model: Qwen3.6-27B (FP16)
• Backend: vLLM v0.20.2 (TP=4)

Power Limit (W)	Output (t/s)	Prompt Processing (t/s)	Total Throughput (t/s)	Efficiency (t/joule)
350/390 (Unrestricted)	29	239	269	0.77
300	29	238	268	0.89
275	29	236	265	0.96
250	29	232	261	1.04
220	27	220	248	1.13
200	24	196	221	1.11

Takeaways:

1. The 220W Sweet Spot: Peak efficiency (matches the blog's findings)
2. Diminishing Returns: Increasing the limit beyond 250W provides diminishing returns

Hope this helps someone. Happy to answer any questions.

I'm VERY satisfied with Qwen 3.6 27B as a daily driver, but I would still like to know if there are any better/bigger models I can run on this setup. My understanding is that the best I can do is DSv4 at Q2 - not sure if it's fully supported yet though.

Additional context: it's an open build on a generic mining frame. I'm cooling it with 10x TL-C12C-S (5 on each side of gpus perpendicularly). I finished building this very recently so I'm open to suggestions on how to improve it.

Edit: Added prompt processing to the table