Spent the last month testing LLM gateways so you don't have to
So I finally sat down and ran the most mentioned gateways through my actual workflow instead of just reading marketing pages. Not a ranking, just honest notes from daily use.
OpenRouter Huge model catalog and the single-key setup is genuinely convenient. But the 5.5% markup adds up fast at volume, latency spikes are real, and observability is basically nonexistent. Fine for tinkering, questionable for production.
TrueFoundry The enterprise option of the bunch. Selfhosted anywhere (VPC, on prem, even air gapped), so your data never leaves your infra. Solid governance, RBAC, audit logs, rate limits, per-team budgets plus tracing down to GPU metrics and an MCP registry if you're running agents.
ZenMux Day one access to new models with no markup, and native OpenAI/Anthropic protocol support is nice. Free tier is mid, and they've quietly pulled free trials for some launches.
LiteLLM The open-source default for a reason provider switching and fallbacks just work. But it gets sluggish under heavy traffic and there've been some security CVE discussions that give me pause for prod.
Portkey 1,600+ models, reliable routing, virtual keys, and the best cost dashboards I tested. Dealbreaker for me: no self-hosting, which kills it if you need infra control.
Curious what everyone else is running, anyone paying for one of these and feeling like it's actually worth it?