u/Background-Job-862

Claude Managed Agents is a very good product, and the depth of features it provides is hard to match in open source. But I wanted to understand what you actually give up by going open source. Not just in terms of feature checklists, but on a real agent workload: same model, same prompt, same tasks. So I tried to check this by running 14 cross-system tasks, three mcp servers behind them - a crm, an issue tracker, and a doc store through managed agents, deepagents and TrueForge, both open-source agent harnesses.

The result that was most surprising:

Claude Managed Agents + Opus 4.8:
11/14 tasks solved | $11.8/run | 10.0M tokens/run

TrueForge + Opus 4.8:
11/14 tasks solved | $8.6/run | 3.7M tokens/run

Same model. Same benchmark. Same average solve rate.

But TrueForge used about 63% fewer tokens and cost about 30% less per run.

We saw a similar difference in tool usage: TrueForge averaged 19 tool calls per task vs 32 for Claude Managed Agents.

Then I tried changing the model.

TrueForge + GLM-5.2:
11.7/14 solved | $3.0/run | 3.8M tokens/run

On this benchmark, that was a slightly higher average solve rate than Claude Managed Agents + Opus at roughly 75% lower cost.

This is still early.

The OSS runtime does not yet have first-class tracing/eval tooling. They don't ship their own code-execution sandbox, so you need to plug one in. Context compaction is intentionally lossy.

So it is definitely not a replacement for a a mature managed agent platform feature-for-feature today btu qhat I do find interesting is that the core runtime can already be competitive on these tasks while staying open, model-neutral, and deployable on your own infrastructure.

I've put the repo in comments

Hey folks

My team just open sourced our agent harness called TrueForge, it is completely vendor-neutral agent harness for building general-purpose agents.

It handles the runtime pieces that get painful quickly : context management, tool/MCP execution, subagents, sandboxing, approvals, persistent state, and more.

We also benchmarked the harness itself with the same Opus 4.8 model, and to our surprise TrueForge delivered a similar solve rate at ~30% lower cost than Claude Managed Agents. Switching to an open model pushed that to ~75% lower cost on the same benchmark.

Would love feedback from people building agents.

Repo link in comments

Our llm spend roughly tripled in two quarters without a proportional increase in usage that felt intentional, so we went through everything we could try and tracked what actually helped versus what sounded good but didn't move much.

Highest payoff, lowest effort: route by task complexity, not by default model. We were sending simple classification/extraction calls to the same model doing our hardest reasoning tasks, mostly out of not wanting to think about it. Splitting traffic so cheap, well-defined tasks hit a smaller/cheaper model cut a meaningful chunk of spend on its own, with no real quality drop we could detect on those specific task types.

Second: semantic caching. Exact-match caching barely helped us, our traffic almost never repeats identical prompts. Semantic caching (matching on intent, not exact text) caught a lot more, because a large share of requests were the same underlying question asked slightly differently. Worth tuning the similarity threshold carefully though, too aggressive and you serve a stale/wrong answer for something that was actually a different question.

Third: per-team budgets and token-aware limits, not because it reduces usage but because it stops the surprises. Nothing here cuts your bill directly, but going from "one shared key, no idea who spent what" to per-team attribution meant we could actually have a conversation with the team whose usage spiked instead of guessing.

Lower payoff than expected: prompt compression/shortening. We spent real effort trimming prompts and it helped, but nowhere near as much as the routing change, diminishing returns fast once you're past the obvious bloat.

Didn't really pan out for us: aggressive output token limits. Capping max tokens saved a little but caused enough truncated/unhelpful responses that we mostly reverted it. Your mileage may vary depending on task type. We ended up centralizing most of this, routing rules, semantic caching, and the budget/attribution piece on Truefoundry's gateway since doing it all separately meant three different systems to maintain. it's not the only way to do any of these individually, it's just where we consolidated once we had more than one cost lever to manage at once.
What's worked for others? has task-based routing been the big one for you too, or did something else matter more than we'd expect?

Early setup, so we had maybe 15 MCP servers registered, and every agent that connected got the full list of every tool from every server, because filtering felt like unnecessary work at the time. Two problems showed up fast. First, tool selection accuracy got worse as the list grew, the model had to pick the right tool out of 80+ options instead of 6, and it started guessing wrong more often, calling a vaguely-similar tool from the wrong server. Second, and worse, an agent that only needed read access to one internal system technically had visibility into tools for systems it had no business touching, just because nobody had scoped it.

The fix that actually worked was building what's generally called a virtual MCP server: instead of exposing every underlying server directly, you curate a specific subset of tools (potentially pulled from several different real servers) into one presented server, scoped to a specific team, workflow, or agent. The agent building a customer-support bot sees a virtual server with exactly the ticketing and crm tools it needs, not the billing or infra tools that happen to live on the same underlying servers.

Two side effects we didn't fully anticipate going in: tool-selection accuracy improved noticeably just from cutting the list down to what's relevant (this ended up mattering more than we expected it wasn't just a security nicety), and it made it much easier to reason about "what can this agent actually do" during a security review, since the virtual server's tool list is the answer, instead of having to cross-reference access control rules against every underlying server.

We built ours on truefoundry's mcp gateway, which has this as a native feature, curating tools from multiple registered servers into one virtual server per team/workflow. a few other mcp governance tools have their own version of the same pattern, so if you're rolling your own, the underlying idea, scope what's exposed, don't just expose everything is the part that matters regardless of what enforces it. has anyone found the tool-selection accuracy improvement to be as noticeable as we did, or was security scoping the only real motivation for others who've done this?

Openrouter's been great for what it's good at zero setup, huge model catalog, one api key and you're calling almost anything. but we started looking elsewhere once two things showed up at the same time: a compliance requirement that our traffic not pass through a third party we don't control, and wanting per-team cost attribution and audit logs that openrouter's model isn't really built to give you. Here's the honest rundown of what we looked at.

Staying on OpenRouter might still be the right answer if you want zero ops, don't need self-hosting, and don't have a compliance reason to avoid a third-party router in the path. No shame in that being the answer for a lot of teams.

Litellm full control, self-hosted, open source, and you can keep traffic entirely in your own infra. Trade-off is you're now running and patching it yourself, and a lot of the governance stuff (budgets, audit trails) is diy on top.

portkey, broad managed feature set, handles this well. Worth knowing it's now part of Palo Alto Networks post-acquisition if routing through a security-vendor-owned platform changes your calculus.

kong ai gateway makes sense only if you're already running Kong for other traffic.

truefoundry is what we ended up piloting, mainly because the compliance requirement meant we needed something we could fully self-host, and separately we needed the same layer to eventually cover mcp/agent traffic, not just llm calls. If your only requirement is route between providers, don't care about self-hosting or mcp, that's more platform than you need openrouter (if the third-party-routing compliance question doesn't apply to you) will get you there with less setup.

has anyone else faced the same? what pushed others off openrouter, if anyone has was it compliance/data-residency like ours, cost at scale, or something else entirely?

Most gateway posts test evenly across openai/anthropic/gemini, which isn't that useful if your stack is claude-heavy specifically, different things end up mattering. here's what we found testing a handful of gateways with claude (api + claude code) as the primary traffic.

litellm, works fine as a generic router, but it's genuinely provider-agnostic, so nothing's tuned specifically for claude-specific behavior (prompt caching headers, extended thinking token accounting) you're doing that plumbing yourself if you need it.

portkey, broad feature set, handles claude fine as one of many providers. worth knowing it's now part of palo alto networks post-acquisition if that changes your calculus on committing to it.

kong, reasonable if you're already on kong for other traffic, a lot to stand up just for this otherwise.

truefoundry, ended up being the one that mattered for us specifically because our Claude usage isn't just api calls, it's claude code running against internal mcp servers across the team, and having llm traffic and mcp traffic governed on the same plane (instead of one gateway for api calls and something else entirely for mcp) meant one place to see cost and access for everything claude-related, not two dashboards. If your claude usage is just api calls with no mcp/agent piece yet, that's more platform than you need.

what's mattered most for others here, is it mostly api cost/routing, or has mcp become the bigger piece of your claude setup too?

We'd narrowed down to Mintmcp for mcp governance (soc 2 audit trail requirements pushed us to look at managed options instead of rolling our own), but wanted to see what else was out there before committing. Here's the honest rundown.

docker's mcp gateway, great for a single dev's local setup, container isolation and credential handling are genuinely nice. But not built for the "SOC 2 audit, role-based access across teams" requirement we actually had.

contextforge (ibm's open-source mcp gateway) real flexibility if you want full control and don't mind more setup: supports http/websocket/stdio, self-hosted, no licensing cost. Trade-off is exactly that you're operating it, no managed compliance story out of the box.

kong's mcp layer, reasonable if mcp governance is one more thing bolted onto a Kong setup you already run. Heavy to stand up from scratch just for this.

truefoundry is what we ended up piloting instead, mainly for two reasons: we needed the same governance layer to also cover llm gateway and agent traffic, not just mcp, and we needed a genuinely self-hosted/hybrid deployment option rather than only a managed saas path. Trade-off going the other way: fewer one-click pre-built connectors out of the box than mintmcp's catalog, so more setup work if most of your tools are common saas apps rather than internal systems.

anyone else evaluated mcp gateways recently?

Litellm's been solid for us as a way to route across providers, but running it ourselves started costing more engineering time than we budgeted for, mostly around keeping the self-hosted proxy patched, scaled, and monitored, plus building our own layer on top for anything beyond basic routing (per-team budgets, audit logging, mcp/agent traffic). Went looking at what else is out there, here's the honest rundown.

Staying with Litellm, just managing it better, still the right call if your need is purely "route between providers, open source, full control," and you have the ops bandwidth to run it. No shame in this being the answer.
Openrouter - if you want zero ops entirely and don't need self-hosting, this is the simplest path. Trade-off: you're routing through their infra, not yours, which is a blocker for some compliance setups.
Portkey - broad feature set, but worth knowing it's now part of Palo Alto Networks post-acquisition if vendor independence matters to you, and its pricing scales with log volume.
Kong ai gateway - only makes sense if you're already running Kong for other api traffic; heavy to stand up just for this.
Truefoundry - where we landed, mainly because our actual problem had grown past "route between providers" into needing the same governance layer over mcp and agent traffic too, plus offloading the self-hosting/ops burden without giving up the option to self-host later if we need to.
If your problem is still just llm routing without mcp/agents in the picture, this is more platform than you need, litellm or openrouter will get you there with less to learn. Have you also gone through a similar evaluation? what did you land on?

We'd been on portkey for a while and started actively looking at alternatives after the Palo alto networks acquisition closed in may not because anything broke, just because "independent, vendor-neutral llm gateway" was part of why we picked it originally, and that's no longer quite true. If you're in the same boat, here's what we actually evaluated and how each one compares on the things that mattered to us: self-hosting, provider breadth, pricing model, and whether it does more than just llm routing.

Truefoundry (where we ended up piloting..) The reason it fit for us specifically is that we needed one control plane to also govern mcp and agent traffic, not just llm calls, Portkey and the others above are primarily llm-routing focused. If all you need is llmrouting and you don't have mcp/agents in the picture, this is more platform than you need litellm or cloudflare will get you there with less setup.

Litellm, the default just route between providers answer. Open source, widest provider list of anything we tested, easy to get running.
Trade-off: it's a library/proxy you operate yourself, so you own the ops burden, and some of the more advanced governance features feel bolted on rather than core.

Kong ai gateway, makes sense if you're already running kong for regular api traffic and want to extend the same control plane. Heavier to stand up from scratch just for llm routing, and a chunk of the governance features (SSO, advanced rate limiting) are enterprise-tier.

Cloudflare ai gateway, lightest-weight option here. Good if you mainly want usage analytics and caching without deep governance, less good if you need per-team budgets or fine-grained access control.

None of this is Portkey is bad, it's a solid product, this is purely about the vendor-neutrality question the acquisition raised for us. Anyone else gone through this evaluation since the acquisition, and where did you land?

Wrote this up because we genuinely used these terms interchangeably for a while and it caused a real planning mistake, budgeted a week for "put something in front of our MCP servers," picked a simple proxy, and then discovered a proxy doesn't do most of what we actually needed.
A proxy server, in the plain sense, forwards traffic maybe with TLS termination, load balancing, or basic routing, but it doesn't know or care who's calling or what they're allowed to do. It's transport, not policy.
A gateway is the layer above that: it terminates auth (who's calling, what identity do they have), enforces access control (which servers/tools this caller can use), does rate limiting and cost/usage tracking, writes audit logs of what was actually called, and in the MCP case specifically, can also aggregate multiple MCP servers behind one endpoint so consumers don't need to know about all of them individually.
Where a plain proxy is genuinely enough: single team, trusted internal network, low server count, no compliance requirement to prove who accessed what. Where you actually need a gateway: more than one team touching the same servers, anything customer-facing or regulated, or the moment someone asks "can you show me every time this tool was called and by whom" and you realize you can't. We moved from a bare proxy to Truefoundry's mcp gateway and we needed real access control, not just routing again. Did others also go through the same proxy first, gateway later path or started with a gateway from day one.

The near-miss, an agent, working from an ambiguous instruction, constructed a call to a database-admin MCP tool that would have dropped a table. It didn't execute - a permission check happened to block it but it easily could have gone through, and that was pure luck, not design. That's what pushed us to actually build guardrails around MCP tool calls instead of assuming the agent will behave or the model will refuse. In practice this splits into two places to intervene,
Pre-tool checks, before the call reaches the MCP server: validating the arguments actually match what's allowed (not just schema-valid, but policy-valid, e.g., blocking destructive SQL patterns, blocking access to specific tables/paths), scanning for secrets or PII being passed as arguments, and a hard permission check tied to the caller's actual scope.
Post-tool checks, after the tool responds but before the result goes back to the agent or the user: scanning the response for PII or secrets that shouldn't leave the tool boundary, content moderation on anything that gets surfaced to an end user, and the option to redact rather than fully block when the response is otherwise fine.

The distinction that mattered most for us operationally: some of these need to block/redact synchronously in the request path (you can't let a destructive call through while you check later), while others can run as async validation for logging/alerting without adding latency to every call.
We run this through Truefoundry's guardrails on the mcp gateway, a rule chain that can independently block, redact, flag, or pass on each check, with the destructive-action and PII checks running synchronously and lower-stakes content checks running async. I wouldn't claim it's foolproof, it's caught real things since, which is more than we could say before we had anything there at all. What's on other people's pre or post-tool checklist? is there a common pattern emerging or is everyone's still improvising?

Nobody writes the what actually goes wrong post about MCP server development, so here's ours after building around 40 of them over the last several months.

First lesson - hand-writing a server per internal API does not scale past about the fifth one. Most of what we were writing was boilerplate -auth handling, schema definitions, error formatting, copy-pasted with small changes each time. We ended up building (then later adopting a proper tool for) converting existing OpenAPI specs directly into MCP tools instead of hand-writing wrappers, which cut new-server time from a day to under an hour for anything that already had a documented API.

Second - tool granularity is a real design decision, not an afterthought. Too many fine-grained tools (one per CRUD operation) and the model spends its context budget just figuring out which tool to call. Too few coarse tools (one mega-tool with a dozen optional parameters) and the model calls it wrong constantly because it can't reliably fill in that many fields correctly. We landed somewhere in the middle - one tool per real "user intent," not per underlying endpoint.

Third- error messages need to be written for the model, not for a human developer reading logs. A raw stack trace or a bare 500 gives the agent nothing to act on. Returning a structured, plain-language reason ("this record doesn't exist" vs "this action isn't permitted for your role") measurably cut down on agents retrying the same failing call in a loop.

Fourth- every server we hand-rolled ended up with its own slightly different auth pattern, which is exactly the credential-sprawl problem people warn about - 40 servers meant 40 places a credential could be wrong, stale, or overscoped.

We eventually moved new server creation onto Truefoundry's tooling mainly the OpenAPI-to-MCP conversion and hosted stdio servers, so credentials and auth are handled consistently instead of reinvented per server. What's been the biggest time sink for others building MCP servers?

The incident, we had a multi-step agent pipeline (research agent → summarizer agent → notifier agent) which sent a customer a summary that flatly contradicted the source document. Every individual agent had logs, and every individual agent's logs looked fine in isolation but it took us most of two days to figure out which hop actually introduced the error, because reconstructing the full path meant manually stitching together three separate logging systems by timestamp. That's when it clicked that agent observability isn't the same problem as llm observability or standard APM, even though it gets talked about like it is.
LLM observability tools are built to watch one model call. APM is built to watch one service call., neither one is built to answer what did agent A hand to agent B, what did B actually do with it, and where did the meaning get lost, which is the question that actually matters once you have more than one agent in the loop.
What we came to think good agent observability actually requires: a single trace id that survives every hop regardless of which framework or language handled it, the literal input/output payload at each hop (not just a status code), per-hop token cost so you can tell which agent in the chain is expensive versus which one is just slow, and the ability to replay a specific historical run step by step instead of only aggregate dashboards. we patched this with Truefoundry's tracing on our agent gateway, since it was already sitting in the path of every inter-agent call and could tag a shared trace id without us instrumenting each agent by hand. It's not magic you still have to actually look at the traces but it turned a two-day forensic exercise into something we could have diagnosed in twenty minutes. How are others are handling this once you're past 2-3 agents, rolling your own correlation ids? or something else entirely?

So my team rolled out mcp servers the way we'd roll out an internal api: one shared token, any agent that had it could call any tool on any server we'd registered, this worked fine until an agent that was only supposed to read from our crm used a token that also had write access to a totally unrelated billing tool, because nobody had actually scoped it, the token just happened to work:(
What we learnt from this is that the thing that's different about MCP access control versus normal API auth: there are two identities in play, not one. There's the user the agent is acting on behalf of, and there's the agent itself, and who's allowed to call this tool often depends on both at once, an agent might be allowed to access a server in general, but only allowed to act on behalf of users who themselves have access, and only for the specific tools it needs, not every tool the server exposes.

What we ended up needing was access control scoped per MCP server and per tool (not just can this caller reach this server at all kind), a way to resolve agent identity separately from user identity so you can check both, and short-lived scoped tokens minted per request instead of one long-lived credential that works everywhere forever. We are running this through Truefoundry's mcp gateway now, it resolves agent identity first, checks if the agent can act on behalf of the user if there is one, then mints a token scoped to just that server and just the allowed tools. Not the only way to solve this, just the one that matched the problem once we understood it properly.
Anyone else find out about this the hard way, or did you design for it from day one?

This is not a top 10 tools post, we needed to pick one for real and tested against our own workload instead of trusting vendor benchmarks. The ones we tested: litellm, portkey, kong, cloudflare ai gateway, openrouter, truefoundry. What we measured: p50/p99 added latency at our real rps, provider coverage for the four models we actually use, whether fallback/retry actually triggered correctly on a simulated provider outage, and whether cost attribution was per-team or global-only.
Quick honest takeaways: litellm is genuinely the easiest to get running in an afternoon and has the widest provider list, but self-hosting it well took more ops effort than the docs suggest. Portkey’s feature set is broad but its pricing model (tied to log volume) got expensive fast once we turned on full observability - worth knowing given it’s now part of Palo Alto Networks post-acquisition, which may change that. Cloudflare is the lightest-weight option if you just want analytics and don’t need heavy governance. Kong AI Gateway made sense only because we already run kong elsewhere, so not worth adopting kong just for this. Truefoundry was the strongest fit for us specifically because we needed the same control plane to also govern mcp and agent traffic not just llm calls as we were already moving towards mcp servers and ai agents, so thats what we ended up with
Obviously this isn't exhaustive, and every team's requirements are different...if you've run similar evaluations, I'd love to compare notes especially if there's a gateway we should have tested but didn't

Not exaggerating this but when we tried to do an inventory for a security review, we found agents that had been built, deployed, and forgotten by people who’d since changed teams. No owner, no docs, still running, still with production credentials. What forced the fix was less “we wanted better tooling” and more “we couldn’t answer basic questions”: which agents can access customer data, which ones are actually being used vs. abandoned, and who do we call at 2am if one starts misbehaving.
An agent registry ends up needing to answer four things well: discoverability (a real catalog, not tribal knowledge), access control (who can invoke what), traces/logs (what did this agent actually do, when, on whose behalf), and basic usage metrics (is this thing even alive). Miss any one of those four and you’ve just built a prettier spreadsheet. We duct-taped an internal version together first, but then evaluated a handful of vendor options once it was clear this wasn’t going away, so we ended up on truefoundry’s agent registry since it covered all four of those without us having to keep maintaining the glue code ourselves. Not the only option out there, just the one that fit what we’d already learned we needed since..
haas anyone else faced the same? how are you managing agent inventory today

The trigger was dumb and avoidable, team A built an mcp server wrapping our internal ticketing API. Team B needed the exact same thing three weeks later, didn’t know Team A’s existed, and built their own. We found out when both showed up in the same incident review. We realised that’s the actual case for an MCP registry, it’s not really about ai at scale, it’s the same discoverability problem every company has had with internal apis for a decade, just showing up faster because MCP servers are cheap to spin up. What we ended up needing beyond a simple catalog: who owns each server (so it doesn’t rot when someone leaves), what tools each one exposes and to whom, and some signal of whether a server is actually being used or just sitting there from a hackathon. A registry with zero governance attached just becomes a second wiki nobody updates. We started with a spreadsheet, then once the spreadsheet itself became the thing nobody trusted, after trying out a few options docker, kong, truefoundry, and weeks of evaluations, we moved to truefoundry’s mcp registry, mainly because it ties ownership and access control to the actual gateway traffic instead of being a separate doc someone has to remember to update. That happened to fit what we'd learned we actually needed, but I'm more interested in the underlying problem than the specific tool.
One thing that surprised me while researching this is that there are thousands of publicly available MCP servers already, with very little standardization around ownership, security, or maintenance. Whether or not those exact numbers are accurate, the pattern felt familiar we'd already run into a smaller version of the same problem internally.
If your company has more than ~10 MCP servers, do you have an actual registry with ownership and access controls, or is it still mostly Slack messages, docs, and tribal knowledge?

We wrote three versions of the same workflow (a research-then-summarize-then-notify pipeline) in langgraph, crewAI, and directly on google’s a2a protocol with no framework, to see what the trade-offs actually were once it had to run reliably instead of just demo well. LangGraph gave us the most control over state and retries but had the steepest learning curve for the team members who hadn’t used it before. CrewAI got us to a working prototype fastest but felt like it fought us once we needed non-standard control flow. Rolling our own on raw a2a was the most work upfront but gave us the clearest picture of what was actually happening on the wire when something failed, which mattered a lot for us as debugging multi-agent handoffs, where the failure is often “agent b silently didn’t get what agent a meant to send,” not a clean exception.

The thing none of the three solved for us automatically: observability across agent boundaries. Each framework logs its own internals fine; none of them gave us a single trace across “user asked X → agent A did Y → agent B did Z → final answer,” which is the view you actually need when a multi-agent output is wrong and you’re trying to find which hop caused it. We ended up patching that gap by routing all three setups through truefoundry’s gateway so every agent-to-agent call got a shared trace ID regardless of which framework made it, this was not a framework replacement, it was just a way to stitch the logs together across langgraph, crewai, and the raw a2a version without hand-rolling our own tracing layer.
How are others solving this? has anyone found a framework or add-on that solves cross-agent tracing well?