Hey everyone. I'm experimenting with agent tooling and trying to understand a problem before building around it.
I'm seeing a recurring pattern where adding more MCP servers/tools eventually creates more problems:
tool definitions eat a lot of context,the model has more similar tools to choose between, tool selection becomes less reliable keeping every tool loaded seems wasteful when most aren't relevant to a given task.

I'm curious how people actually handle this in production.
When an agent has a large toolset, do you:
Load everything into context?
Manually scope tools for each agent/workflow?
Use a tool router/search layer?
Dynamically load tool definitions only when needed?
Something else?
And more importantly: has this actually caused you measurable problems? cost, latency, wrong tool calls, reliability, etc?

I'm particularly interested in real examples rather than what should work theoretically. Cheers :)

I tried LangChain Academy but their courses lack depth. Also, I prefer learning through books/written content. Please suggest resources for learning LangChain and Agentic AI.

We blamed our model for weeks because our RAG assistant kept giving confident policy answers that missed one exception clause. Too much chunk overlap meant retrieval kept pulling near copies of the same paragraph and the exception never made it into context. We were paying extra tokens to make the wrong evidence look unanimous.

We inspected the retrieved chunks inside Braintrust traces and compared chunking runs on the failed queries. Embedding similarity showed why basic top k kept selecting copies. Adding deduplication helped, then reranking against the full question pulled the exception clause above the repeated policy text. Citation precision and groundedness both improved when the evidence set stopped repeating itself.

Those failed queries also became regression cases for us. We now score retrieval coverage separately from answer groundedness, because a model cannot cite a clause it never received (yes, obvious in hindsight). Token spend also fell because the context carried fewer duplicate passages. Not really the problem we thought we were fixing, but I'll take it.

The idea is surprisingly simple:

You package your Agent Skills + MCP servers into one portable folder, and any compatible agent client can load it.

What I like about the approach:

→ It doesn't reinvent Skills or MCP. It just gives them a common packaging layer.

→ One plugin can bundle capabilities that belong together instead of making users install everything separately.

→ Failures are isolated. If one MCP server breaks, a valid Skill in the same plugin can still load.

→ The core stays small. Client-specific features can live in namespaced extensions instead of bloating the standard.

The important caveat: v1 is packaging, not a security model. No permissions, sandboxing, secrets management, registry, or trust model yet.

So if you already maintain Skills or MCP servers, this is probably worth looking at.

I dug into the spec, the folder structure, client support, and what v1 intentionally leaves out in a full breakdown.

Link in the comments 👇

Curious: are you already packaging Skills/MCP, or still managing them separately?

Hey everyone,

I built a multi-agent LangGraph workflow that takes a new starter (or leaver) form and generates a complete IT pack:

- Role-based checklist (25 steps for onboarding / 17 security-ordered steps for offboarding)

- Jinja2-generated PowerShell scripts

- LLM-drafted welcome email + Day-1 guide

- Human approval gate before finalisation

- Auditor agent that validates everything

Stack: LangGraph + FastAPI + Azure OpenAI + Pydantic v2 + Jinja2

Repo: https://github.com/DOWNEY7/employee-onboarding-orchestrator

Looking for feedback on:

- Architecture decisions

- Human-in-the-loop design

- Anything you’d improve for real company use

Stars and comments appreciated 🙏

It feels like nowadays, almost everything you might want to build with LangGraph is already being implemented — and arguably better — directly by the frontier AI providers.

OpenAI, Anthropic, Google, Microsoft, etc. are increasingly providing models with better tool use, reasoning, memory/context handling, agent loops, and orchestration capabilities out of the box.

So what is the real advantage of building your own agent architecture with LangGraph?

Is it mainly about control and customization — e.g. deterministic workflows, state management, human-in-the-loop, custom routing, retries, parallel execution, observability, and being model/provider agnostic?

Or are there use cases where LangGraph actually produces materially better agents than simply using the agent frameworks provided by the frontier model companies?

I’m particularly interested in hearing from people who have deployed LangGraph agents in production. What made you choose LangGraph instead of the native agent tooling from OpenAI/Anthropic/etc., and would you still make the same choice today?

Hey guys, I've been building a small experiment around AI agents. I’m just trying to see if developers or people that use ai for heavy workflow would find something like this useful?

The basic idea is:
You describe a task - the system figures out what specialist is needed - finds the best available agent - delegates the task - returns the best result.

So instead of you having to figure out which AI/tool/agent to use, the network handles the procurement for you and gives you the best match based on your task.
I've got a basic working prototype now and I'm looking for people to try it and tell me where the idea falls apart.

I'm particularly interested in tasks where you'd normally need to use multiple tools or hire someone.
I'm genuinely trying to build something useful, any feedback would be appreciated. I’ll drop a link in a few days if anyone would actually be interested in trying this out.

If you think this is genuinely crap and no one would use is, that’s great too.

Cheers everyone 😁

Handoffs work fine in dev but get messy once you are past three or four agents touching shared state. In small setups, you can get away with one agent passing a context object to the next, but that starts breaking down once agents run concurrently and touch the same resources. We have tried passing full context objects, using a shared memory store, and routing everything through a central orchestrator. Each has its own tradeoffs. The orchestrator approach feels stable so far, but it also feels like we are reinventing a workflow engine on top of LangChain.

Has anyone found an agent-to-agent communication pattern that holds up in production with real traffic? Is everyone building custom orchestration layers or has a standard approach emerged?

If you're building a document-processing pipeline today, does it actually make sense to send every stage through the same high-end multimodal model? My instinct is that a lot of document work doesn't need the most capable model.

For example:

• Clean PDFs / straightforward OCR: traditional OCR, direct text extraction, or a lightweight model may be enough.
• Parsing and simple extraction: a faster, lower-cost model such as Gemini Flash-class models may handle this well.
• Handwriting, poor scans, complex tables, or ambiguous fields: this may be where you route to a more capable multimodal/reasoning model.

The part I'm unsure about is whether the accuracy and cost advantage of model routing is actually worth the orchestration complexity in production.

Here’s how I’m thinking about the trade-offs:

• Accuracy: One model gives you more consistent behavior, but it may be overkill for simple documents and weaker on certain edge cases. Multi-model routing lets you optimize by document type or task, but poor routing decisions can hurt accuracy.
• Latency: One model means fewer routing steps and simpler execution. Multiple models can keep easy documents on faster models, but retries and escalations may add latency.
• Cost: One model is easier to predict, but expensive if a premium model handles everything. Routing can reduce cost significantly if most documents can stay on lightweight models.
• Privacy: One provider/model can simplify governance and data handling. Multiple providers add complexity, although routing could also keep sensitive documents on private or internally hosted models.
• Fallback behavior: With one model, a retry may simply reproduce the same failure. With routing, low-confidence outputs can escalate to another model or eventually to human review.
• Maintenance: One model is much easier to operate. Multi-model pipelines require more evals, routing logic, monitoring, version management, and regression testing.

I'm especially interested in the fallback strategy.

Would you use:

small model → larger model → different provider → human review

or simply:

one strong model → human review when confidence is low?

And what would you use as the routing signal: OCR confidence, image quality, handwriting detection, document type, extraction confidence, schema validation failure, or something else?

For anyone running document AI at meaningful volume: has multi-model routing actually reduced cost and improved accuracy, or does the added complexity outweigh the benefit?

ok so quick context if you haven't seen the other posts: I've been messing around with CacheVerifier, basically testing whether bolting a verifier onto semantic caching actually helps. right now it's dumb and binary, candidate answer either gets a thumbs up or thumbs down, no in-between.

there's this other paper, TweakLLM (arXiv:2507.23674), that does something I think is genuinely smarter: instead of rejecting a bad candidate and eating the full regen cost, it has a cheap LLM just... rewrite the candidate so it fits the new query. patch it instead of throwing it out. I want to add that as a comparison to my own setup and I've been stuck on it for a while, so figured I'd just ask here, since this sub has already bailed me out twice on this project (the axis-problem theory and the bucketing design both came from comment threads here, not from me).

here's where I'm stuck. everything I currently measure is trace-based against public benchmarks , "was this correct" comes entirely from the dataset's own labels, no actual LLM judge anywhere in the loop. works great when the answer is binary. falls apart completely once you're rewriting text, because now you've got a brand new string that isn't in any label anywhere. nothing to check it against.

things I've considered and don't love:

just throw an LLM judge at grading the rewrites. but now I'm introducing a whole new cost/noise source that literally nothing else in this project needed, and "let an LLM grade another LLM's output" is its own whole mess
when there happen to be multiple reference answers for the same query cluster, score the rewrite against one of them by similarity. except that's literally the "similarity ≠ correctness" problem this entire project exists to complain about. using it as my metric here feels like cheating on my own thesis
just skip fine scoring, measure something crude like "did rewriting recover some recall vs just rejecting," and not even try to put it on the same hit-rate/error-rate curve as everything else. doable but honestly a weaker result than I want
if anyone's had to evaluate a generate-a-rewrite step where there's no clean ground truth for the output, not classification, not ranking, an actual freeform string you have to judge somehow , genuinely curious how you dealt with it. or if you think I'm overcomplicating this and should just pick one of the above and move on.

repo's here if you want the full context on what's been tested so far: https://github.com/imxinchengyou/CacheVerifier

I haven’t seen RAG come up in agent architectures in over 6 months due to Agentic Search (letting the model use Bash/grep/glob/read), which seems to work pretty well. Wondering what others are experiencing. I’m sure there’s still a time and place for RAG, exposing semantic search as a tool… but where do we draw the line? When the corpus is too large to let the model comb through it progressively?

How are you actually deciding what context an agent should see at each step?
Not just “use RAG” or “increase the context window” — I mean things like task state, previous tool calls, memory, retrieved documents, conversation history, failed attempts, etc.
Do you have an actual context selection/pruning strategy, or are you mostly throwing everything into the prompt and relying on the model to figure it out?
Curious what people are doing in production, especially with long-running agents.

Hey everyone,

I’ve been building an open-source project called TraceMotive.

The basic idea is:

Given two AI agent executions, TraceMotive compares their observed behavior, finds the first supported divergence, and lets you jump into the evidence around it.

It runs locally.

The goal is not to claim root cause or automatically explain why something happened. One of the design principles I care about most is that if the structural evidence is ambiguous, TraceMotive should say that the result is uncertain instead of guessing.

I just released v0.5.0.

This release was mostly about making the project more adoptable rather than adding a huge new feature.

Some of the work in v0.5:

- packaged `tracemotive serve` and `tracemotive demo`

- structured JSON diff support

- Safe Later Observations for additional supported evidence

- improved first-time-user onboarding

- Python 3.10 / 3.12 CI

- frontend test/build CI

- dependency auditing and Dependabot

- threat-model/security documentation

- explicit compatibility, limits, and storage docs

- clean wheel/sdist installation dogfooding

- a 30-scenario evidence-conservative regression corpus

For that regression corpus, the current results still have:

- false-confident meaningful divergence: 0

- false-confident investigation starting point: 0

There are still intentional limitations.

For example:

- LangGraph is not currently supported.

- The validated OpenAI Agents SDK range is `>=0.17,<0.18`.

- TraceMotive does not claim RCA, causal inference, confidence scoring, reconvergence, or recovery detection.

A bit of context: I’m a high-school student, and I built the first version after roughly a week of programming experience, heavily using AI coding tools.

I know that’s an unusual way to start an OSS project, so I’ve been trying to compensate by being strict about tests, failure cases, compatibility claims, and not claiming more than the evidence supports.

At this point, the thing I need most isn’t another feature idea — it’s real users.

If you build AI agents and have a run you could try this on, I’d really appreciate feedback about:

- where installation/onboarding feels confusing

- whether the comparison is actually useful

- cases where TraceMotive becomes uncertain

- agent execution patterns the current model handles badly

Thanks to everyone who gave feedback on the earlier versions — several of those comments directly influenced v0.5.

I’m building a LangGraph application with a supervisor and several specialized agents:

• Booking Agent
• Payments Agent
• Recommendations Agent
• Support Agent

Currently, the supervisor classifies the user’s first message and stores the selected agent in checkpointed session state. Every later message in that chat is routed to the same agent.

This creates two problems:

1. The user may change topics during the same chat—for example, ask for recommendations and then make a booking.
2. One prompt may require multiple agents:

> “Recommend the best hotel for my trip, then book the top option.”

Here, the Recommendations Agent should run first and return structured results. The Booking Agent should then receive those results and continue the workflow. It may also pause for confirmation using a LangGraph interrupt.

Constraints

• Each agent has its own state and may have pending interrupts.
• State must not leak between agents.
• Dependent tasks must execute in order.
• Independent tasks may run in parallel.
• Permissions must be checked before each operation.
• A new message must not accidentally resume an unrelated interrupt.
• Agents currently run as subgraphs in one Python service.
• Agents must return both streamed UI output and structured data.

Questions

1. What LangGraph architecture would you recommend?
2. Should this use a router, supervisor, orchestrator-worker pattern, or subagents-as-tools?
3. Should agents use separate thread_id values, separate checkpoint_ns values, or both?
4. How should a new message be distinguished from a response intended for a specific interrupt?
5. What is the best way to pass structured results between agents?
6. Should the supervisor create a task DAG per turn, or dynamically call agents using ReAct?
7. Are Agent Cards, A2A, or an agent mesh useful if all agents run inside the same service?

I’m looking for reliable production patterns from people who have built persistent multi-agent LangGraph applications with human-in-the-loop workflows.

I started working on this problem because of a recurring frustration with AI coding agents: they were surprisingly capable inside a session, but much less reliable across sessions.

The obvious explanation was memory, so my first attempts were fairly conventional.

I tried project instruction files, persistent Markdown notes, embeddings, vector search, and eventually RAG over project documentation and source code.

They all helped.

None of them really solved the problem.

The interesting part was figuring out why.

Retrieval wasn't the same as understanding the project

My initial assumption was that if an agent could retrieve the most semantically relevant pieces of the project, it would have enough context to work correctly.

That turned out to be too simplistic.

Consider an architectural decision that changed over time:

Decision A
    ↓
implementation
    ↓
problem discovered
    ↓
Decision B supersedes A
    ↓
partial migration

A vector search can easily retrieve Decision A because it is semantically very close to the current task.

The problem is that Decision A may now be exactly the context you don't want the agent to follow.

So I started separating different kinds of project knowledge:

• source code
• documentation
• architectural decisions
• session history
• implementation outcomes
• changes
• dependencies
• agent activity

That led to a more difficult question:

How do you determine which project state is authoritative now?

Simply storing more memory made this worse rather than better.

More context can make the agent worse

My next mistake was assuming that increasing the amount of retrieved context would increase reliability.

It doesn't necessarily.

Large context windows make it tempting to send everything that might be relevant.

But relevance isn't binary.

A piece of information can be:

semantically relevant
but outdated

structurally relevant
but unrelated to the current task

historically relevant
but superseded

recent
but low importance

So the problem became less about retrieval and more about context selection.

I ended up treating context as a constrained resource.

Instead of asking:

>

the system needs to ask something closer to:

>

That required combining several signals rather than relying only on embedding similarity.

Code needed a different representation

Source code created another problem.

Chunking code and embedding the chunks works reasonably well for some questions, but poorly when the answer depends on relationships.

For example:

function A
   calls B
      imports C
         implements interface D

The relevant code might not be semantically similar to the user's query at all.

It is relevant because of its structural relationship to something that is.

So I added a local code graph built from AST analysis, with relationships such as:

IMPORTS
CALLS
REFERENCES
TYPE_USES

Retrieval could then combine semantic similarity with graph traversal.

That turned out to be particularly useful for impact analysis: starting from a symbol mentioned in the task and expanding only through bounded relationships instead of dumping large sections of the repository into the context window.

Then multiple agents made the problem harder

The next issue appeared when switching between coding agents.

I might spend a session with Claude Code, then continue the same work with Codex.

The second agent had access to the same repository, but not necessarily the reasoning and decisions produced during the first session.

This made me realize that attaching memory to an agent was probably the wrong abstraction.

The persistent state should belong to the project, not the model.

That changes the architecture.

Instead of:

Developer → Agent → Memory

I started experimenting with:

                 Claude Code
                      ↕
Developer ↔ Project Intelligence ↔ Codex
                      ↕
                    Cursor

The agents become replaceable clients of the same project state.

That also introduces concurrency problems.

If two agents are modifying related areas of the codebase, project memory alone isn't enough. The system needs some awareness of ongoing work, dependencies, and potentially conflicting changes.

The architecture that emerged

After several iterations, I ended up with roughly four different forms of project state:

Semantic layer
    documents + embeddings + retrieval

Historical layer
    decisions + memories + outcomes + session context

Structural layer
    AST-derived code graph

Coordination layer
    active work + changes + agent state

A context assembly step sits above them.

Its job isn't to expose everything.

Its job is to construct a bounded context package for the current task.

The coding agent itself remains external.

Communication happens through MCP, which means the project intelligence layer doesn't have to care whether the client is Claude Code, Codex, Cursor, or something else.

One unexpected result

The biggest change in my thinking was that persistent memory wasn't actually the main problem.

Memory is relatively easy to store.

The difficult problems are:

• deciding what deserves to become memory
• knowing when information has become stale
• determining when one decision supersedes another
• connecting semantic information to code structure
• selecting context under a token budget
• maintaining useful state across different agents
• preventing multiple agents from developing incompatible views of the project

In other words, the problem gradually stopped looking like "RAG for source code."

It started looking more like maintaining a small, continuously updated model of the project's state.

I eventually packaged these experiments into an open-source server called Snipara, but the project itself is less interesting to me than the architectural question behind it:

As coding agents become increasingly capable and interchangeable, should project knowledge live inside each agent's context, or should the project maintain its own persistent intelligence layer that agents query?

I'm increasingly convinced it's the latter, but there are still difficult questions around memory decay, conflicting decisions, graph expansion, and context selection that I don't think are completely solved.

You give an agent a few tools and point it at a task. The answer comes back right, the output looks clean, and it feels ready to ship. Then you scroll through the trace just to be sure, and the middle of the run is a mess.

This pattern is common in tool-using agents. A research agent does search, fetch, summarize. The final summary is correct, but the fetch step pulled the wrong URL and the search fired twice on the same query. The model reached a right answer anyway, ignoring the junk it pulled and leaning on what it already had. Change the input slightly and that same broken path returns a wrong answer, with no obvious reason why.

The problem is that grading only the final output lets it through. The output is correct, so nothing gets flagged. Every mistake in the middle stays invisible, even though the trace has all the evidence.

What catches it is scoring each tool call against what it was supposed to do, not just grading the final answer. A right answer built on a wrong step should not count as a pass.

How are you catching mid-chain tool-call failures? Grading the whole trajectory, checking each step, something else?