u/Confident-Truck-7186

Everyone tests the happy path on their MCP server. I did too, and it hid the problems that actually bite in a real client.

So I wrote a harness that drives the server over stdio the way Claude Code does, and deliberately broke things. Setup: 45 tools, protocol 2024-11-05.

The four cases worth asserting on, in rough order of how much pain they cause:

1. Missing credentials entirely. Should fail immediately with a message naming the variable. If it starts fine and only dies inside a tool call, the model will try to work around it, and you end up debugging the model instead of the server.

2. Bad credentials. Should surface a readable error. A raw stack trace here is worse than useless, because the model cannot tell an auth problem from a transport problem.

3. Unknown tool name. Should be rejected cleanly. Models hallucinate tool names more often than people expect, especially with 40+ tools listed.

4. Invalid params. Should come back as an error the model can act on. This is the one that decides whether the model self-corrects or gives up.

The other assertion I would not skip: on tools/list, check every tool has a non-empty description and a valid inputSchema. Boring, and it caught a real problem for me. Separately, if any tool is long-running and returns a job id rather than a result, make sure the description says so. Mine did not, in 43 of 45 tools.

Implementation notes if you build one: read stdout line by line on a background thread, parse each line as JSON, match responses by request id rather than assuming order, and give the poll loop a hard timeout so a hung server fails your test instead of hanging it.

Took an afternoon. I would not ship an MCP server without it now.

Disclosure: I build an SEO API for agents, and this was our own MCP server. Nothing to buy here, the failure-mode list is the point.

I smoke-tested my own MCP server the way Claude Code actually drives it — spawn over stdio, initialize, tools/list, then a real tools/call — and found a problem that I think is easy to hit if you have any long-running work.

Setup: 45 tools, protocol 2024-11-05, stdio transport.

The call itself worked fine. What came back was this:

{
  "jobId": "071a3254-...",
  "status": "pending",
  "poll_url": "/api/v1/jobs/071a3254-...",
  "retry_after_seconds": 2
}

So the model does not get an answer. It gets a job ticket, and it has to decide on its own to call a separate status tool to find the result. In my test Claude worked it out and polled once, and the whole thing finished in about 18 seconds.

Then I grepped my own tool descriptions:

tools mentioning poll/job/async: 2 of 45

The description for the tool I called was:

"Run pre-publish SEO QA for metadata, indexability, canonical, headings, schema, links, and media."

Nothing about the call being async. Nothing about a follow-up call.

Why I think this matters: the model reads tool descriptions before it decides what to do, and reads the response after. A capable model can infer the polling loop from status: pending and retry_after_seconds. A weaker one, or one already mid-task with competing instructions, will reasonably report "I've queued a job for you" and stop. The user gets a job ID instead of an answer, and nothing errored, so nothing looks broken.

The fix is boring and text-only. Put the contract in the description:

"... Returns a job ticket. Call the job status tool with the returned job_id until status is completed."

Two other things that fell out of the same test, in case they are useful:

• Failure modes were the easy part to get right. Missing API key exits immediately with a clear "environment variable is required" message, bad key surfaces a readable error, unknown tool is rejected, invalid params come back as an error the model can act on.
• My serverInfo was reporting a stale hardcoded version that no longer matched package.json. Worth deriving that from the package rather than typing it twice, since clients surface it.

If you are building an MCP server where work takes more than a couple of seconds, I would check two things: does your tool description state the async contract, and does your response carry enough for a model to work out the next call on its own. Mine had the second and not the first.

Disclosure: I build an SEO API for agents and this was our own MCP server. Not pitching it here — the async thing is the actual point, and I got it wrong in 43 of 45 places.