u/Future_AGI

An allowlist decides one thing: whether an MCP tool is allowed to run. It does not count how many times that tool runs, and it does not read the arguments the model fills in. So an approved tool is approved for every call, including the ones you were not thinking about when you added it.

An agent in a retry or planning loop calls a deploy or a delete_rows tool in production. The name is on the allowlist, so call two, call five, call twenty all pass the same check and go through. The allowlist was never counting.

The arguments are the other half. A name check approves delete_rows, but the model writes the arguments, and that is where a drop table or a where 1=1 shows up. The allowlist never reads that far.

This is the same gap behind the recent Figma client-name allowlist pushback: a name is a label the caller reports about itself, and a label does not limit what the caller does next.

We run a gateway in front of the tool calls. Every call gets checked against the allow and deny lists and a per-tool rate limit, the arguments get scanned on each one, and a single request can only fire so many calls before the gateway stops it.

What are you using to cap this today: a per-tool rate limit, a hard call budget per turn, human approval on writes, or nothing yet?

If you follow the agent threads here, you have watched the "do agents need memory or just documentation" argument go in circles. Most of that, we think, is that "memory" covers three separate problems, and each camp is defending a different one.

First is keeping state inside one session. In LangGraph that is the checkpointer, keyed by a thread_id, and it lets a run pause and resume with state intact. The edge that catches people: when a node hits an interrupt and you resume, the whole node runs again from the top. Any tool call or write before the interrupt fires again on every resume, which is why the docs insist anything before an interrupt be idempotent.

Second is remembering across sessions. That is the Store, separate from the checkpointer, namespaced per user, searchable later. It is the piece the "you need a memory layer" people mean, and where the cost hides: as it grows, a naive setup spends tokens reading and refreshing it even on tasks that never needed it, and a fact true three months ago still sits there with nothing marking it stale.

Third is what fits in the context window each call. LangGraph treats this as context management, separate from the two memory types above, though the argument lumps it in. Trim or summarize old turns too hard and the agent re-asks for a detail the user gave two turns back, because the summary dropped it.

Split those three and the argument reads differently. The documentation camp is mostly on the third problem and part of the second. The memory-layer camp is almost always on the second. Same word, three problems, which is why it keeps circling.

So which of the three has you stuck? On the cross-session one, the contested piece, what breaks first: store bloat that makes simple tasks pay for memory they never needed, or stale entries with no clean way to retire them? We're curious what people have landed on.

Running 15+ MCP servers can burn 60,000 tokens before your agent reads a message.

A lot of teams run 15 or more MCP servers. Each one exposes 20 to 30 tools. Multiply that out and the agent burns through 60,000 tokens on every single request loading tool schemas.

More servers should mean more capability. Agents get slower, more expensive, and worse at picking the right tool.

Anthropic published research on this. Their agents connected to thousands of tools consumed hundreds of thousands of tokens before reading a single request. Two problems appear: tool definitions fill the context window, and intermediate results burn even more tokens as they pass through the model.

One developer deleted every MCP server in his setup and got 40% of his context window back. The agent performed better with fewer tools because it could focus on what mattered.

Grouping servers works better than running one per service. Code and CI tools go in one grouped server with GitHub, GitLab, and Buildkite. Internal comms go in another with Slack, Discord, and email. Data stores go in another with Postgres, Redis, and BigQuery. Product tracking goes in another with Jira, Notion, and Linear.

The agent picks the right group first, then uses the tools inside it. It doesn't see GitHub pull request tools when answering a Slack question.

This cuts context usage in half. Response quality goes up because the agent stops sorting through irrelevant tool definitions.

You can check your own numbers. Count your MCP servers. Multiply by the average number of tools per server. Multiply by 150 tokens per tool definition. That's your baseline context cost per request. If it's over 30,000 tokens, you're paying for tools your agent rarely uses.

The tradeoff: MCP gives you a standardized protocol for connecting agents to services. That standardization is worth the token cost for remote or authenticated services. For local tools that get called frequently, the overhead is not justified.

Other teams do similar things. Claude Code switched to lazy loading. The agent decides which tools to load instead of dumping everything at once. For high-frequency local calls, some developers skip MCP entirely and use direct code execution. Anthropic measured a 98% token reduction when agents write code to call tools instead of loading every schema.

Four well-designed grouped servers beat fifteen scattered ones.

What's your MCP server count? Have you measured what it costs in tokens?

Most of us start with the obvious two: latency and cost. They are easy to graph and easy to explain to amanager.

But those two rarely tell you the thing you actually care about, which is whether the output was any good.
A response can be fast, cheap, and completely wrong.

So we are curious what the rest of you track once something is live. A few we have seen people mention:

• How often the model refuses or goes off-topic.
• Whether answers stay grounded in the retrieved context.
• Tool-call success and retry rates for agents.
• Drift, when the same prompt slowly gets worse over weeks.

What is on your list, and what did you add only after it burned you once?

An agent gets approval to deploy a staging change. It reaches the deploy API, gets a timeout, and resumes thirty minutes later. The earlier approval and plan are still available. In that gap, the artifact, target environment, or rollout window may have changed. 

For consequential actions, an approval should describe the action that was approved, rather than becoming a standing permission. A practical grant can bind a specific action and resource, the relevant arguments, an expiry, a replay rule, and an audit record. At execution time, the question is more specific than "was this approved?" It is "does this approval still cover this call against this resource with these parameters?"

Four checks make that concrete: 

Scope: Record the exact tool operation and resource. deploy is too broad if the target is a particular service and environment. 

Expiry: Set when the grant stops applying. A short window can cover a deploy confirmation without surviving the rest of a long-running run. 

Replay protection: Decide whether the same approval can authorize a second attempt, a changed argument, or a different resource. That should be an explicit choice. 

Audit record: Preserve what was approved, for which resource, when, and what actually executed. A run ID and an approval record give you somewhere to start when the answer is disputed. 

Retries are the awkward part. The agent may have created a release but not reached the deploy API. Or it may get an ambiguous response after issuing a database migration, a force-push, a destructive write, or a production configuration change. Retrying could be correct, could duplicate the side effect, or could need a fresh approval because the original grant expired or surrounding state changed. 

There is no universal retry rule. The workflow needs an explicit state for "unknown outcome," plus an explicit decision to inspect state, retry, request a new approval, or stop. Quietly treating an old grant as authority for the next attempt hides the decision that matters. 

At Future AGI, the gateway applies policy to each MCP tool call and its arguments, which gives us a place to keep a grant tied to the action that was approved. 

“How are you handling retries after an approval has expired, especially for deploy, migration, or force-push actions?”

We think the first question around GPT-6 Astra should be smaller than “is it smarter?”

Can it pass the jobs the current model already handles?

A production model upgrade can change more than final-answer quality. The regressions that hurt are often tool choice, tool argument shape, JSON validity, refusal behavior, retrieval judgment, and the sequence an agent takes through a workflow.

We would treat a GPT-6 Astra migration like a release candidate.

Keep a pinned slice of real tasks. Include known bad cases, not only clean demos. Capture expected tool calls or state changes when those are part of correctness. Review failures from the trace instead of inspecting only the final response.

A benchmark can make a model worth testing. An eval harness has to tell us whether it is safe for a particular workflow.

What is the first regression test you would run before moving a production workflow to GPT-6 Astra?

A coding agent should never treat a codebase graph as current unless it can name the Git commit it represents. 

A graph can be useful and still be unsafe as edit context. If it cannot say "I was built from commit abc123 at this time," there is no way to compare it with the repository the agent is about to change. A current graph is a graph tied to a known revision. 

The freshness contract we want is small:

• Every graph response carries the repository commit SHA and graph build time. 
• Changes to files, imports, symbols, or tests mark related graph regions dirty. Unrelated regions can remain intact. 
• If the graph SHA does not match the target revision, the query returns the Git diff first and labels affected relationships stale. 
• Before editing, the agent verifies the relevant files and symbol definitions against the checkout as well as the graph.

A graph can identify where an agent should look. The direct check is where the agent establishes that a reference has not moved, been deleted, or changed its callers. An MCP server exposing repository context could emit based_on_commit, built_at, and stale in every result. 

Git already gives the comparison primitive: the difference between the graph commit and the working target. The indexer needs a policy that translates that diff into dirty regions. A changed test can be its own dirty region. It should stop the agent from treating test coverage or a call path as settled. 

That gives stale context a useful failure mode. When freshness is unknown, show the diff and ask the agent to inspect the affected files. Do not make it guess from a cache. 

Which do you prefer for codebase indexing: incremental rebuilds after each change, explicit rebuilds before an agent session, or Git-diff fallback when the graph is behind? What is the first stale-context failure your coding agent caused?

One allowed tool call can change the next allowed MCP tool call.

allowed read tool -> untrusted tool output -> allowed write tool with changed arguments

Server and tool allowlists answer identity questions: which server and which tool names may run. They do not preserve authorization intent after an allowed read returns text that changes the arguments of the next call.

This is the narrow MCP security problem. A result can contain instructions. If the agent follows them, prompt injection has crossed a boundary without selecting an unapproved server or tool. The next call can still match every configured allowlist.

For agent authorization, the runtime needs a second-hop decision before dispatch. Inspect the output-derived argument as an argument for the destination tool. Was it formed from untrusted text? Does it request a capability the user authorized? Does it fit the destination tool's expected scope?

The destination tool may be allowed for the task. Its identity tells us little about why this invocation has these arguments. A prior read tool may have returned data the agent should use as data, then that result changes a later action. Many agent guardrails only evaluate tool identity and miss this transition.

The input-side check needs the argument's provenance, including which values came from the prior tool result and which came from the user or agent plan.

In Future AGI’s Apache-2.0 gateway, we keep checks at the MCP boundary for configured server and tool policy, tool arguments, textual tool results, and per-tool limits.

That boundary gives us a place to examine the result and the next call. The next call still needs a decision that considers its arguments and their origin before execution.

Where do you enforce that second-hop check: runtime, MCP boundary, policy layer, or executor?

Feels like half of us just grab the Q4_K_M and move on. But we hardly ever see anyone say where their model falls apart once you push lower.

From what we have seen, the usual 4-bit GGUF quants mostly hold up. Go below 4-bit and it gets rough fast, and that's where quantization-aware training starts to matter more than plain post-training quantization. At 3-bit and under, folks often pair QAT with distillation to get the quality back. Part of why Gemma shipped QAT checkpoints, and why the BitNet ternary models train at low bit-width from the start.

And it stays confident the whole way down, so the bad answers read just like the good ones.

Easiest way to check yourself: run one fixed set of prompts at Q8, Q4, and Q3 on the same model and see where the answers start to drift.

So which model, and what bit level did yours give up at? What broke first?

If you added tracing to an agent this year, odds are you mapped your spans to OpenTelemetry's GenAI semantic conventions. They are becoming the default vocabulary: gen_ai.operation.name for the lifecycle (create_agent, invoke_agent, execute_tool, retrieval, plan), token usage attributes, and a span tree for the whole run instead of one span per LLM call.

Most teams treat these as locked. They are not. Every gen_ai.* attribute in the registry still carries the "Development" badge, and the spec is mid-flight. The repo split earlier this year, and the MCP tool conventions moved into the same GenAI repo, so tool calls now share the agent's trace vocabulary. The names you instrument against today can still change under you.

So you pick which cost to pay: follow the spec now and eat the churn when names change, or keep your own span schema until it settles and migrate later anyway.

For anyone tracing agents in production: are you following gen_ai.* as it moves, pinning a version, or holding your own schema for now? And how much has that churn cost you?

If you give an agent memory across MCP tool calls, it has to live somewhere, and the store you pick shapes how the agent behaves. We spent a month on two of them: a Git-backed store and a vector DB. Both worked. Then each broke where the other held up.

The Git side was easy to live with. Every memory was a plain file we could open, diff, and roll back. When the agent wrote something wrong, we could open the exact entry and see why. Retrieval was the weak spot. String and path matching only goes so far, so it missed relevant memories that shared no keywords. And with nobody pruning, the store rotted into stale notes the agent still treated as current.

The vector DB fixed retrieval. Recall was good. It surfaced related context the Git side never found. But we lost any way to eyeball what the agent knew. Writes went in as opaque vectors, so when a memory looked off, there was nothing to read. And one bad embedding would skew later lookups, with no signal until answers started drifting.

So we're stuck. One we can read but it barely retrieves. The other retrieves fine but we can't see what's inside. For now we treat the Git store as source of truth, with the vector index as a helper on top.

Which do you trust as your source of truth? And has anyone landed on running both, with one as the index over the other?

The same input gives a different output every run, so half the bad answers people hit can't be reproduced on purpose. Someone files a bug with the exact input attached, you run it, and it comes back fine. Run it again, fine again. The failure showed up the one time nobody was recording.

This breaks the loop every developer grew up on: reproduce it, fix it, run it again to prove it's gone. All three steps assume the same input lands the same output. With LLM calls that assumption is gone, your tests go flaky, and most debugging tooling still acts like it holds. People hit this even at temperature 0, where the output is supposed to be deterministic and isn't.

The move most teams land on is to capture the failing run the instant it happens, exact input and exact output, because a day later you cannot get back to it. Even then you are stuck proving a negative. It passed twenty times after the fix. It also passed twenty times before.

So the honest question: how are you reproducing a non-deterministic failure well enough to fix it? Or has everyone accepted you catch these live or not at all?

A guardrail fires and blocks an output. The eval that flagged it lives in another tool, the trace in a third, so to see what happened you line all three up by hand. That is normal once an agent is in production: you run four things, tracing, evals, runtime guardrails, and a gateway in front of the models, usually four separate tools.

There's a real argument for keeping them separate. Each goes deeper in its own lane. Langfuse and Phoenix are strong at tracing, Ragas and DeepEval are real eval frameworks, Guardrails AI and NeMo handle policies, and Portkey and LiteLLM are solid gateways. Nothing locks you in, and you can swap any piece the week a better one ships.

The cost shows up later: four dashboards, four logins, data that never joins. Spend sits in the gateway, quality scores in the eval tool, the guardrail's decision elsewhere, nothing keyed the same way. The all-in-one bet is the opposite: the layers share context, so a trace, its eval score, and its guardrail decision sit in one record.

We build one of these, Future AGI: it runs tracing, evals, runtime guardrails, and the model and tool gateway in one Apache-2.0 stack you can self-host, so a blocked call never leaves and the trace lines up with the eval. It is still a nightly build with rough edges, and the honest reason to run it this way is fewer moving parts, not any single piece beating the dedicated tool.

So when a guardrail blocks something, is it on the same trace as the eval that flagged it, or are you piecing it together from separate tools? And if you consolidated, did it ever cost you on depth, where the bundled piece was weaker than what you gave up?

It's wild how fast this snuck up on all of us. A year ago, running a 100B-class model at home was a fantasy unless you rented a datacenter card. This month people are loading 20GB+ mixture-of-experts models onto a 16GB consumer GPU and getting usable speed out of them. The model launches grab the headlines, but the runtime work underneath them is what changed this year.

None of it is one magic trick. It's a stack of open, community-built engineering: aggressive quantization, offloading MoE experts to CPU and system RAM, compressing the KV cache, speculative decoding, and the llama.cpp/GGUF-style runtimes tying it all together. All of it open, all of it moving week to week.

For most of us, the frontier-model race matters less than this. A slightly worse model you can actually run beats a slightly better one you can't touch.

So, over to you all: what's the most surprising thing you've gotten running on your own hardware this year, and which trick made it fit? And where's the wall still too high?

Wednesday check-in. This one's for the stuff that passed every check and was still wrong.

The tool call that came back 200 with a useless body. The span that never fired, so the trace read as a clean success. A retry that "worked" but handed back stale data. The dashboard sitting green while the answers quietly got worse.

What's the one that fooled you this week, and how did you catch it? Wrong first guesses count too.

And if you just want to drop what you're building or where you're stuck, do that here.

RAG evaluation always comes down to two bad options. Either you hand-label question and answer pairs forever, or you point an LLM judge at every run and watch it burn your budget and hit the provider's rate limits. We sat in that spot for a while before admitting the second option was our own fault.

The mistake was scoring retrieval and generation as one blob. Once they are fused, the only tool that can grade the whole thing is a judge model, so you pay a judge on parts that never needed one. Retrieval is not a judge problem.

Here is how we split it.

Separate retrieval eval from generation eval. Grading them as one record is what forces a judge onto everything. The two stages fail in different ways and want different tools.

Retrieval is arithmetic, not judgment. Label the right source for a question once, then recall@k, MRR, and nDCG tell you whether the retriever found it and ranked it well. No model calls. That answers "why would you need an LLM to test retrieval."

Use the judge only for faithfulness and groundedness. That is the one place with no deterministic ground truth, whether the answer actually stands on the retrieved text. Scoping the judge there cuts judge calls by an order of magnitude, and the rate-limit problem goes with it.

Keep the golden set small and one-time. Thirty to fifty labeled examples are enough to start. You pay the labeling cost once per example, and every later run reuses the same set.

Label by source page and line range, not chunk index. Then you can change chunk size later without relabeling. The same logic says score each sub-step of a multi-hop pipeline, so one run gives you several checks.

Curious what people reach for on the judge half, ragas, DeepEval, something local, and where plain retrieval metrics were enough on their own.