I’m working on a multi-model API proxy for teams that use LLMs heavily, and I’d love to get feedback before we onboard our first batch of users later this month.

The idea is to provide one API layer for accessing major current models, with routing, fallback, usage controls, observability, and developer-friendly tooling on top.

I’m especially interested in hearing from:

• small and mid-sized companies using AI APIs in production
• teams building internal AI tools
• people heavily using agentic coding tools
• dev teams switching between multiple model providers
• anyone dealing with cost, latency, reliability, rate limit, or quota issues

A few things I’m trying to understand:

1. Which models would you need supported on day one?
2. Do you care more about cost, latency, quality, context window, or automatic fallback?
3. What is the most painful part of your current AI API setup?
4. Would task-based routing be useful? For example, cheaper models for simple tasks, stronger models for coding/reasoning, fallback models when a provider is down.
5. What kind of observability would you want? Logs, traces, cost per user, cost per project, prompt/version tracking, evals?
6. For agentic coding workflows, what matters most: tool calling reliability, context window, latency, model choice, rate limits, or something else?
7. If you’re an SMB, what would make you trust a proxy layer enough to put it between your app and the model providers?

Not trying to make this a sales pitch — we’re still shaping the product and want to build around real workflows instead of guessing.

If you use AI APIs heavily, especially for coding agents or production workflows, I’d really appreciate hearing what your ideal setup would look like.

Hi,

We’re designing a self-hosted LLM gateway setup in an educational/research context, with local vLLM backends and tools like OpenWebUI for human-facing chat.

Right now, the easiest option seems to be LiteLLM, but we’re checking whether there is a fully free and open-source alternative with similar features.

What we need is not only model routing, but the gateway/governance layer:

- OpenAI-compatible API gateway
- Authentication and API key management
- Single Sign-On support, ideally OIDC/SAML
- User/team quotas or budgets
- Cost tracking or token accounting
- Metrics and logs
- Load balancing between multiple vLLM instances serving the same model
- Support for local/private model endpoints
- Compatibility with OpenWebUI
- Reasonable setup and maintenance for a small education/research team

The architecture would probably expose several endpoints through the gateway: local vLLM models, possibly other compatible providers, and maybe a semantic-routing endpoint for automatic model selection when users do not choose a specific model.

Has anyone here deployed something like this in education or research?

We’re also considering whether Portkey or Bifrost could fit this role, so experiences with those tools would be very welcome.

Is there a fully self-hosted, open-source stack that gets close to LiteLLM’s feature set, or is LiteLLM currently the most practical option?

Thanks