u/No-Archer0007

Yk the ones "Your writing is lovely and clearly accomplished, but I didn't connect with the project the way I'd need to in order to represent it effectively in a competitive market."

Translation: the sentences are fine, the book didn't grab me, good luck

I think I preferred the form rejections, honestly. At least those don't make you sit there going like okay so if the prose is good and you still don't want it, what's wrong with the BOOK

a form rejection you can blame on volume, on luck, on the agent having a full list. A personalized one that praises the writing and passes anyway points straight at the thing you can't see, which is that the story itself isn't doing what it needs to do, and now you have to figure out what that is with basically no information.

Thirty one queries in. Four of these "lovely prose" passes now. I'm starting to think my problem isn't the writing line by line, it's something structural in the premise or the stakes that isn't landing in the pages, and a polished query plus polished sample pages is actually hiding the problem instead of revealing it.

any suggestions are honored, busy much but still gonna read em all

Once you're calling more than one LLM provider, you eventually hit the question of whether to put a gateway (or router, same idea, the terms get used interchangeably) in front of them. I resisted this for a while, it felt like premature infrastructure, until a provider had a rough hour, our app just sat there serving 500s with no fallback path, and support tickets piled up for most of a day. That's usually the moment teams actually go looking. Here's how the main options compare in 2026.

First, what an AI gateway actually does, so the comparison makes sense. It sits between your app and the providers, gives you one endpoint for many models, lets you swap models without code changes, handles fallbacks when a provider fails, and centralizes cost tracking and budget limits. Some go further into governance and org-wide routing policy. The differences below are mostly about how far past "one endpoint for many models" each one goes.

OpenRouter

The easiest possible on-ramp. You point your existing OpenAI client at their endpoint and it basically works, plus you get access to a huge catalog of models, more than you'll probably ever use. Pricing is usage-based with a small markup on top of provider costs, or bring your own keys. For getting started, prototyping, or hobby work, it's hard to beat on speed-to-first-call.

Where it stops short is the control plane. It's more "here's access to every model" than "here's how my org governs LLM usage." Budget caps per team, fine-grained routing policy, governance, those aren't its focus. Fallback exists but it's defined per-request rather than as an org-wide policy.

Best for: fast model access, prototyping, broad model coverage.

LiteLLM

The open-source workhorse, and probably the most widely deployed thing on this list in practice. If you want a self-hosted, code-level abstraction across providers, LiteLLM is the one most teams reach for first. It normalizes the API across 100+ providers so you're not maintaining provider-specific request formats, runs as a proxy or a library, and has an active community and fast provider support, new models tend to show up quickly. The proxy also does key management, budgets, and rate limits, more than people assume it can. It's free as open source, with a paid enterprise tier for SSO, audit logs, and support.

The tradeoff is the usual open-source one, you own it. You're operating the proxy, handling upgrades, and building your own dashboards and higher-level governance around it. Great if you want control and have the ops capacity, more work if you don't, and that maintenance cost is easy to underestimate on day one.

Best for: teams wanting an open-source, self-hosted abstraction with full control and broad provider support.

Portkey

A real step up into managed production controls. Portkey has proper request-level reliability, configurable retries and fallbacks (you set conditions on what counts as a failure and where it routes next), semantic caching that genuinely cuts repeat-query cost, guardrails, and solid observability. Pricing is usage-based with a free tier to start, which lowers the barrier to trying it. If your main pain is per-request reliability and cost on a service or two, it's mature and well-built, and its failure-condition config is the most granular here.

Where it felt limited to me was at the org level. When you've got many services and want one routing policy applied everywhere, it leans more toward per-request configuration than "define the rule once, apply it across the whole organization."

Best for: teams that need strong request-level reliability, granular fallback control, and caching.

Orq.ai

Orq's gateway (you'll still see it called the Router, the rename to Gateway is recent so both terms are floating around) is the most policy-oriented of the group. Budget controls and fallback chains are defined centrally, so when a provider degrades, the fallback applies across every service routing through it without per-app changes. If you're running several apps or teams through one layer and want consistent routing policy, governance, and cost control in one place, that's the angle it's built for.

The bigger differentiator is that the gateway isn't a standalone product, it's part of a full LLMOps platform, so routing sits next to prompt management, evals, and observability rather than being a separate tool you reconcile with everything else. It's managed, and the compliance story (SOC2, GDPR, EU AI Act, EU-based company) is real if you're in a regulated space.

The honest tradeoffs: it's a broader platform, so if all you want is a thin routing layer and nothing else, it's more than you need, and Portkey gives you more granular per-request failure knobs. It's also a newer, smaller player than the incumbents, so the community and battle-tested-at-extreme-scale factor isn't the same as something like LiteLLM yet. Worth weighing if you're purely optimizing for a minimal gateway rather than a platform.

Best for: teams wanting centralized routing policy across multiple apps, or a gateway that's part of a broader lifecycle platform.

The short version

If you just want model access fast, OpenRouter. If you want open source and you're happy owning it, LiteLLM, which is also the safest bet for broad provider coverage and community support. If your pain is granular request-level reliability and caching on a few services, Portkey. If you're routing multiple apps through one layer and want centralized policy plus prompt management, evals, and observability in the same place rather than a standalone gateway, that's where Orq.ai fits best.

The real fork is whether your problem is "give me models" or "help me govern how my org uses them." The first points to OpenRouter or LiteLLM, the second to Portkey or Orq, and which of those two depends on whether you're solving reliability on a few services or policy across many. Pick based on the problem you actually have today, not the one you might have at 10x scale, you can always move up the list when you get there.

one thing i've noticed over the last year is that teams seem willing to accept failure modes from LLMs that they'd never accept anywhere else in the stack.

if an external API became unavailable for an hour and took down a critical user flow, nobody would shrug and say "well, that's just how APIs are." we'd ask why there wasn't a timeout, a fallback, or a circuit breaker.

replace that API with an LLM and suddenly those questions disappear.

same thing with prompts. i've seen prompt changes go straight into production with less review than a one-line config change, even though that prompt is effectively part of the application's behavior. then something regresses and nobody can confidently answer what changed or when.

none of this is really an AI problem. it's the same distributed systems discipline we've spent years learning, except now one component happens to be probabilistic instead of deterministic.

sometimes it feels like the excitement around LLMs has convinced people that the old rules don't apply anymore, when in reality they matter even more. the model is the new part. everything around the model is still just engineering.

curious if other people who've been building systems for a while are seeing the same thing, or if i'm becoming the guy who starts every sentence with "back in my day" lol

Edit: the prompt-changes-going-to-prod-with-no-review bit is the one that gets people. we treated prompts as config for too long, anyone could push, then something regressed and nobody knew what changed. fix was just treating them like code, version control + a review step before prod. we use OrqAI for it but tbh git and a held-out set does most of the same job. point is prompts deserve the same review bar as a one-line config change, not less.

still one of the worst moments i've had working with LLMs.

the feature itself wasn't unusual. user asks a question, we retrieve the relevant docs, the model answers from those docs. we'd repeated one sentence so many times internally that nobody even questioned it anymore: "it only answers from our knowledge base."

except... it didn't.

the incident wasn't caused by some wild hallucination out of nowhere. retrieval came back with weak matches, but not completely empty ones. the model took those scraps of context, filled in the gaps on its own, and returned an answer that sounded just as confident as every correct answer we'd ever shipped. the customer had no reason to think this one was any different.

the part that still bothers me is nothing looked broken. there were no crashes, no errors, nothing that would've made us think the system had gone off the rails. it behaved exactly the way we'd built it to behave. the bad assumption was ours.

we assumed retrieval automatically meant grounding. it doesn't. all retrieval does is give the model context. it doesn't guarantee the model will stay inside that context.

after that incident we stopped treating grounding as something the model would just do. if retrieval is weak, the system says it doesn't know. and before an answer goes back to the user, we verify it's actually supported by the retrieved context instead of trusting the model's confidence.

if you're shipping RAG into healthcare, finance, legal, or anywhere a wrong answer actually matters, i'd strongly recommend testing one thing: what happens when retrieval comes back with almost nothing?

i have a feeling that answer would make a lot of teams uncomfortable.