Built a self-hosted, always-on gateway to compare Claude/GPT/Gemini side-by-side (Open WebUI on a NAS)
I wanted to type one prompt and see several models answer side-by-side, on something always-on the whole household could use. Off-the-shelf tools either charged, wouldn't take my own key, or couldn't show answers side-by-side.
What I ended up with:
- One OpenAI-compatible endpoint in front of several models
- Open WebUI for side-by-side + merge (Mixture-of-Agents)
- Both in Docker on a Synology NAS, so it survives my laptop being closed
- Keys stay server-side, never sent to the browser
- Rule-based routing: only foreign-model domains go through a proxy; domestic ones connect directly
Two real gotchas: a nightly crash that turned out to be the NAS hitting its inotify instance limit, and dead proxy nodes I now auto-detect by pointing health checks at a real model endpoint.
Honest caveat: bridging consumer subscriptions into an API is a ToS gray area — I keep it personal/internal. Curious how you all run always-on multi-model setups?