If you instrument an MCP server with OTel, you get a clean trace of what happened inside the server. But it's an orphan — it doesn't connect to the agent trace that called it. You end up with two disconnected trees and no way to answer "which agent turn caused this slow tool call."

The fix is W3C traceparent propagation through the MCP request, so the server span becomes a child of the agent span. Shipped that in opentel-mcp v0.11.0.

Also in this release:

• Per-model pricing overrides — the built-in table covers 19 models, but if you're on a negotiated rate or a model I haven't added, you can override it instead of getting wrong cost numbers
• Embedding model pricing, which was previously just missing

Still the only Node library I know of that catches CallToolResult.isError=true inside an HTTP 200 and marks the span as ERROR rather than success. That one silently ruins error-rate dashboards.

npm: https://www.npmjs.com/package/opentel-mcp
Docs and setup: https://opentel-mcp-site.pages.dev/

Happy to answer anything about the tracing model — the two-axis ToolOutcome × ObservationIntegrity contract in particular took a few iterations to get right.

Most LLM observability setups treat a tool call as successful if the transport succeeded. With MCP that's wrong: the protocol returns CallToolResult.isError=true inside an HTTP 200. Transport says fine, the tool actually failed, and your error-rate panel stays flat while the agent quietly retries or hallucinates around the failure.

I've been building opentel-mcp, an OpenTelemetry instrumentation layer for MCP servers, to close that gap. It's at ~1,500 npm downloads now. What it does:

Correct error semantics. isError=true gets marked status: ERROR with error.type: tool_error, so failed tool calls actually appear in error rate.

Two-axis outcome model. One axis is ToolOutcome (did the tool do its job), the other is ObservationIntegrity (can you trust what you observed). These get conflated constantly — a schema-drifted response isn't the same failure as a timeout, and collapsing them makes dashboards lie.

Cost and token attribution across 19 models with per-model overrides for negotiated rates, plus embedding pricing.

W3C trace context propagation, so the server span becomes a child of the agent span instead of an orphaned tree.

One finding worth flagging for anyone on stateless HTTP transport: cross-call detection — anything that counts patterns across calls, like agent thrash detection — cannot work when every request gets a fresh instance. I hit this instrumenting my own agent and filed it as a library gap rather than pretending it worked. It's a structural limit of stateless transport, not a bug you can patch around.

npm: https://www.npmjs.com/package/opentel-mcp

Docs: https://opentel-mcp-site.pages.dev/

Curious what others here are doing for tool-call-level observability, especially if you've solved the stateless correlation problem in a way I haven't thought of.

Shipped opentel-mcp v0.11.0. Two things in it, and the first one is a problem I suspect a lot of people have without knowing.

Every cost-attribution tool I've seen — mine included until today —hardcodes a pricing table and applies it uniformly. If you're on an AWS EDP, a committed-use discount, Bedrock provisioned throughput, or negotiated Azure rates, that number is wrong. Not slightly — you're paying a rate the tool has no idea exists, and it reports list price with full confidence.

v0.11.0 adds per-model pricing overrides merged over the defaults. It alsodoes two things I think matter more than the override itself:

A lastVerified stamp on the default table, surfaced at init. Provider pricing moves. A bundled table silently ages into wrongness, and nothing tells you. Now an old table announces itself.

Explicit unpriced status for unknown models. Previously an unrecognised model produced no cost, which a dashboard renders as zero — indistinguishable from "this call was free." Now it's explicitly unpriced, so you can show unattributed spend instead of a confidently wrong total.

Also added embedding model pricing with a pricingKind discriminator. Embeddings are input-only. Modelling them as a chat model with a zero output rate produces the right number by accident and makes the code lie about what it's doing.

Second feature: W3C trace context propagation. traceparent and tracestate extracted from request.params._meta (SEP-414's convention), wired into tool span parenting. So if your agent framework propagates context, MCP tool spans now land inside the calling agent's trace instead of floating as orphaned roots.

Worth knowing: neither MCP SDK does this itself. v2 exports the meta key constants but nothing in the compiled runtime reads or writes them. Real propagation today comes from third-party instrumentation wrapping the official clients — OpenInference's MCP packages do it on both JS and Python. If you're relying on the SDK to propagate, it doesn't.

Zero new runtime dependencies for it — createTraceState() from u/opentelemetry/api rather than pulling in u/opentelemetry/core. Absent or malformed _meta is byte-identical to previous behaviour.

https://www.npmjs.com/package/opentel-mcp

Genuinely curious: if you're doing LLM cost attribution, how are you handling negotiated rates? Every approach I looked at assumes list price.

Posted here a few days ago about MCP tool errors returning HTTP 200 with isError: true. Shipped two releases since. Then found this while testing a deployment shape I hadn't covered.

All my in-memory tracking — retry loop detection, cost attribution, budgetguardrails, schema drift — lives inside a single instrumentMcpServer() call.

That's correct for stdio: one process, one server, state accumulates normally. Correct for stateful HTTP too, where one long-lived McpServer handles many sessions.

But stateless streamable HTTP constructs a fresh McpServer per POST andre-instruments each time. So every counter resets before it can reach any threshold. Four features, zero output, no warning, no log.

That's the standard pattern on Lambda, Cloud Run, Workers — anywhere serverless. Which is where a lot of MCP deployment is heading.

Been true since v0.4. Nobody reported it.

The awkward part: I'd already documented this exact root cause for one feature as an accepted limitation, and didn't notice it applied to three others. Including one I'd shipped hours earlier with a docblock claiming "process-lifetime" state.

Fix direction is a host-supplied instanceKey so trackers can be looked up from a bounded registry instead of constructed per call. Deliberately not a module-level singleton — that would merge unrelated services in a multi-tenant process, which is the same class of bug one level up. Design is written, shipping as v0.9.0.

The limitation is documented in the README now rather than discovered by whoever hits it next.

Also in v0.8.0:

- Tool schema drift detection: hashes each tool's inputSchema from tools/list, flags silent changes. "Why did every call start failing at 3am" is often "someone changed a schema and nothing announced it."

- Two-axis observation contract: separates tool outcome from observation integrity, so "nothing failed" and "nothing was observed" stop looking identical. Notable finding — a HEALTHY state turned out to be unreachable in every configuration, so it isn't in the type at all.

- Cost-aware sampling: not a library feature. Samplers decide at span start, cost is known at span end. So it's a marker attribute plus a documented Collector tail-sampling recipe.

https://www.npmjs.com/package/opentel-mcp

Curious whether anyone here is running MCP on stateless HTTP in production — if you are, I'd like to know what your tracking assumptions look like, because mine were wrong.

If you're running MCP servers in production with OTel, worth checking:

tool errors come back as HTTP 200 with `isError: true` in the result.

Standard instrumentation reads 200 and marks the span successful. You

get a green dashboard over consistently failing tools.

The downstream effect is the expensive part. The agent gets the error

text as a normal result, assumes it asked wrong, and retries. Same

failure, 4-6 times, full context resent each round — and the context

grows every round, so each retry costs more than the last. Everyone's

watching token spend right now, but this particular leak doesn't show

up as errors anywhere. It shows up as a slightly higher bill.

I built a Node library that catches this — inspects the result payload,

marks the span ERROR, and fingerprints the failure so the same root

cause groups across varying error messages.

Just shipped v0.6.1, which adds detection for the retry loop itself:

when an agent hits the same failure fingerprint repeatedly in a session,

you get one event with the loop length and the tokens/cost burned on it,

instead of six spans that each look fine. There's also an in-process

summary accessor if you want to see it without standing up a collector.

Caveat on the cost numbers: pricing is a static table you can override,

and providers change rates often enough that any bundled table drifts.

Treat the cost attribution as directional unless you're supplying your

own pricing. The token counts come from the provider's own usage fields,

so those are solid — it's the dollar conversion that ages.

https://www.npmjs.com/package/opentel-mcp

Node/TS only right now. If you're on Python, the same class of bug

existed in fastmcp itself (#4549, fixed via #4587) — worth checking

your version.

Curious if people running MCP in production have hit this, or if you're

catching tool failures some other way.

Hi everyone!

I've just released opentel-mcp v0.5.0, an OpenTelemetry instrumentation library for Model Context Protocol (MCP) servers.

One thing I found missing while building MCP applications was cost visibility. We already get traces for latency and errors, but we usually discover AI costs later from provider billing dashboards.

This release adds AI FinOps-style observability directly into MCP tool spans.

What's new in v0.5.0

• Token tracking
  • mcp.tool.tokens.input
  • mcp.tool.tokens.output
  • mcp.tool.tokens.total
• Cost tracking
  • mcp.tool.cost.usd
• Model attribution
  • mcp.tool.model
  • gen_ai.response.model (for compatibility with existing OTel GenAI dashboards)
• Budget guardrails
  • mcp.tool.cost.budget_exceeded
  • mcp.tool.cost.budget_scope
• Two new OpenTelemetry metrics
  • mcp.tool.tokens.total
  • mcp.tool.cost.total

It currently includes built-in pricing for 19 models across Anthropic, OpenAI, Google Gemini, AWS Nova, and DeepSeek.

The library also exports DEFAULT_PRICING, defaultExtractor, and calculateCost so pricing and usage extraction can be customized.

One design choice I intentionally made is that this library only observes. It never blocks requests or enforces budgets—those responsibilities belong in AI gateways such as LiteLLM or Portkey.

I'd really appreciate feedback from anyone building MCP servers or working on OpenTelemetry instrumentation.

npm: https://www.npmjs.com/package/opentel-mcp

GitHub: https://github.com/Thirumalaiboobathi/opentel-mcp

If you'd like to follow my work on MCP and OpenTelemetry:

LinkedIn: https://www.linkedin.com/in/thirumalaiboobathi-b-902a51233/

Happy to answer questions or discuss the implementation!

I've been building OpenTelemetry instrumentation for MCP (Model Context Protocol) servers, and I hit a failure-semantics problem that I think generalizes beyond MCP, so I'm writing it up.

The two failure modes

An MCP tool handler can fail two ways:

1. It throws. The SDK catches the exception and converts it into a JSON-RPC error response. The call failed at the protocol level.
2. It returns { isError: true }. The handler returns normally — a successful JSON-RPC response whose payload is marked as a failure:

​

return {
  isError: true,
  content: [{ type: 'text', text: 'No weather data for that city' }]
};

The second one is idiomatic MCP. It's how a tool tells the agent "that didn't work — adapt" without crashing the server or killing the conversation. For agent workflows it's the preferred failure mode.

The instrumentation trap

The obvious way to instrument a tool call:

try {
  const result = await handler(request);
  span.setStatus({ code: OK });        // it returned → success
  return result;
} catch (err) {
  span.setStatus({ code: ERROR });     // it threw → failure
  throw err;
}

Mode 1 lands in catch → recorded correctly. Mode 2 returns, lands in the success path → recorded as OK. Your dashboard reports 100% success on a tool that fails on most inputs. The more idiomatic the tool author's error handling, the more invisible their failures become.

The fix

Inspect the resolved value before setting status:

const result = await handler(request, extra);
if (result?.isError === true) {
  span.setAttribute('error.type', 'tool_error');
  span.setStatus({ code: SpanStatusCode.ERROR });
} else {
  span.setStatus({ code: SpanStatusCode.OK });
}
return result;   // unchanged — the RPC genuinely succeeded, so nothing is thrown

Two details that matter:

• error.type = "tool_error" isn't my invention — it's what the OTel MCP semantic conventions (currently Development stage, in the semantic-conventions-genai repo) specify for exactly this case.
• The result is returned unchanged and nothing is thrown. The JSON-RPC call succeeded; only the tool failed. Instrumentation that converts a polite failure into a crash is changing application behavior, which instrumentation must never do.

In a real trace the difference looks like this:

tools/call fetch_weather ................. 605ms   ERROR
    error.type = tool_error

versus the naive version, where that same span reads OK.

The general lesson

This isn't really an MCP problem. Any protocol where application-level failures ride on transport-level successes has this trap — GraphQL (errors array on a 200), gRPC rich error models, half the REST APIs that return 200 {"status": "failed"}. If your instrumentation only watches for throws, your error rate is a lie wherever the ecosystem's idiomatic failure mode is a clean return.

FastMCP (Python) handles this natively. Among the Node MCP instrumentation libraries I could find, none documented handling it, which is why I ended up writing my own — it's on npm as opentel-mcp if you want to see the full implementation (spec-compliant attributes, stderr export to avoid corrupting stdio transports, ADRs for the design decisions). But the isError trap is the part worth knowing even if you never touch my library.

Happy to answer questions on the implementation.