u/wassupabhishek

I recently ran an experiment comparing a 3-agent pipeline vs a 5-agent pipeline on the exact same workflow. For the first task, the 3-agent pipeline resulted in 86% task completion, and the 5-agent pipeline gave a 91% task completion rate.

This sounds great until I looked at the tradeoff. The 5 agent pipleline was ~40% slower and was twice as expensive to run. For this use case, the extra 5% completion rate wasn’t worth the latency + cost hit.

But then we tested the same architectures on a different task: research synthesis. And the results completely flipped. The 5-agent version consistently caught reasoning gaps and factual misses that the 3-agent setup let through. The additional reviewer/checker agents actually mattered there.

Big takeaway for me - there’s probably no universal answer to what the ideal number of agents is. Also, more agents don't always mean better outcomes.

It seems heavily dependent on the type of task, error tolerance, latency constraints, and where failures actually happen in the workflow

Curious how others here are deciding agent topology in production. Are you relying on any benchmarks, eval datasets, or production traffic experiments?

I was building infrastructure for AI agent experimentation recently and ended up doing 50+ deep conversations with engineering teams across startups and Series B companies about what actually breaks in production and why. A few things that surprised me:

• most agent failures are not model failures
• prompt changes are often tested way more casually than normal code changes
• almost nobody fully agrees on who owns agent reliability
• teams underestimate the operational cost of flaky agents until customers feel it

Happy to talk about how teams run controlled experiments on prompts/configs, common production failure patterns, evals, reliability ownership, rollout strategies, and the economics behind all this.

Ask me anything.

Frameworks like LangGraph, CrewAI, and AutoGen have gotten pretty good at orchestration and execution. But almost none of them really help me figure out how to safely test prompt changes in production.

The default workflow for most teams (including mine) still seems to be like this - tweak a prompt, deploy it, watch metrics and hope nothing weird happens. But the problem is that when behavior changes, it’s hard to isolate why. If it was a prompt update or a model/provider change, or multiple parameters changing at once.

My team seems to treat agent config changes more like code deploys with versioned configs, baseline evals, gradual rollouts, traffic splitting, rollback support, etc. But honestly, most people I talk to are still doing this with logs.

Curious what others are doing here. Are you adapting any feature flag tools for agents? Building internal tooling? Or running eval pipelines? Feels like this layer of the AI stack is still pretty immature.

I have noticed that everyone talks about prompt engineering as if it’s just tweaking prompts against some metrics/goals. But in reality most agent failures are impossible to debug because multiple things changed at once. You change the system prompt, model version, retrieval logic, or maybe the underlying data.

This is what has worked for me so far, and I want to validate if anyone in the community has a similar approach:

Build a baseline first, run the current setup for 1–2 weeks with proper logging before touching anything. Change just 1 variable at a time. Do percentage rollouts for example ~10% of production traffic to the new variant first. Let it run for at least 48 hours. Then wait for enough volume. A lot of teams conclude from just a few conversations. Usually need a few hundred interactions before results mean anything. Define rollback criteria clearly before rollout. What counts as failure should be decided before deployment.

The bigger issue is that most teams don’t actually have infra for systematic prompt evals or rollouts. A lot of LLMOps still end up being logging and manual reviews.

Curious what people here are actually using for this in production.

• Any existing feature flag tools?
• Custom infra?
• Langfuse / Helicone / Braintrust?
• Fully internal platforms?

A thing that surprised me while digging into agent reliability is that a model with 95% accuracy per step sounds excellent. But if your agent takes 10 steps to complete a task, the overall success rate drops to ~60%. And at 100 steps, it’s basically unusable (~0.6%). The failure compounds fast.

Then I came across a few numbers that made this feel less theoretical. Datadog tracked 8.4M AI model request failures in March 2026 and reported that ~5% of AI requests fail in production. A large chunk of these aren’t infra outages, but logic/quality failures that teams can’t properly debug. Similarly, McKinsey in its report said that while many enterprises are experimenting with agents, very few are actually scaling them successfully in production.

The more I look at this, the more it feels like an experimentation infrastructure problem, not a model capability problem. Most teams still test agents in playgrounds/staging and then hope production behaves similarly. But prompts, tools, memory, routing, temperature, context length, fallback logic, etc. all interact in weird ways under real traffic.

Web teams solved this years ago with A/B testing and controlled rollouts. Feels like agent teams need the same thing. Like experiment on live traffic, compare prompt/config variants, isolate regressions, and measure task success over time.

Curious if you agree to this or think there are better ways to solve these production issues.