u/21734234

Remember when everyone was running containers but nobody had a real answer for "how do you run a hundred of them in production"? Dockerfile existed, containers worked, but the operational layer hadn't caught up.

That's AI agents right now. You can build a capable agent in an afternoon. Deploying it reliably, connecting it to other agents, auditing what it does, rolling it back when it halts that's still chaos.

The Kubernetes parallel is weirdly accurate. K8s didn't make containers smarter. It just gave you:

• A way to define desired state
• Scheduling and orchestration
• Health checks and restarts
• Networking between containers
• Role-based access control

Multi-agent systems need basically the exact same list. Desired state for agent workflows, scheduling tasks across agents, health monitoring, secure communication between agents, access policies.

A few projects are starting to think this way like CoralOS uses Kubernetes for AI agents as their framing. There's also work happening in LangGraph, CrewAI etc. but those are more framework-level than infrastructure-level.

The devops/platform engineering skillset is going to matter a lot more in AI than people expect.

Sharing this because I spent way too long figuring it out and couldn't find a good write-up anywhere.

The setup: I needed a research agent, a drafting agent, a fact-checking agent, and a publishing agent. The problem: they were built in different tools at different times by different people on our team.

What doesn't work:

• Chaining them via raw API calls: brittle, no retry logic, context gets mangled
• Putting everything into one framework: not realistic when different teams own different agents
• Just using a message queue: you lose the agentic part, it becomes a dumb pipeline

What actually worked: Treating the coordination as its own concern entirely separate from the agents themselves. Each agent exposes an interface (MCP made this cleaner), and you have something above them deciding routing, fallbacks, and state.

This is basically what CoralOS is doing architecturally, the Coral Server sits above your agents and handles orchestration while agents stay independent. Whether you use that or build your own, the mental model is correct.

The hard lessons:

1. Agents need to be stateless if you want the orchestrator to handle state
2. You need observability at the inter-agent level, not just within each agent
3. Security between agents is not optional, agent-to-agent communication is an attack surface

Happy to go deeper on any of these if useful. This stuff is genuinely hard and under-documented.

Spent the last few months debugging production AI systems for a handful of mid-to-large orgs, and I keep seeing the same failure pattern that nobody really talks about in the benchmarking literature.

The model isn't the problem. The retrieval isn't even really the problem. The problem is document heterogeneity rot.

Here's what I mean. When you first stand up a RAG system, your corpus is relatively clean. You've chunked it, embedded it, indexed it. The retrieval scores look great in eval. Then six months pass.

Now you have:

• A 2023 policy doc that was superseded by a 2024 amendment that lives in a completely different folder
• Meeting transcripts that reference decisions that were later reversed via email (which is not indexed)
• Contracts with line-item exceptions that got negotiated verbally and exist only in someone's Outlook

Your retrieval system has no concept of document authority hierarchy. It treats a deprecated policy PDF the same as the current one because cosine similarity doesn't care about org chart logic or recency signals beyond naive metadata.

The fix isn't better chunking or a bigger embedding model. It's building provenance chains into your indexing architecture from the start so the system knows not just what a document says, but whether it's still true.

A few teams I've seen handle this well are essentially building a lightweight governance layer that sits between ingestion and retrieval tagging documents with confidence decay rates and authority signals rather than treating the corpus as a flat library.

It's more engineering overhead upfront. But it's the only thing that actually keeps production accuracy from drifting.