u/Paoli99

Disclosure:I work at MCPJam, so that’s where I’m coming from.

While building evals, I’ve been thinking about how often MCP vs CLI discussions lump a few different problems together.

An agent picking the wrong tool, struggling to pass data between tools, and filling up its context window can all look like “MCP isn’t working.” But fixing one won’t necessarily fix the others.

Passing data between tools: Can the agent take the output from tool A and use it in tool B? If the output is prose, it may have to extract the values first. Structured output can help, though the next tool still needs to accept that data.

Context getting crowded: Tool definitions and results take up space, even when every tool is clearly described. Our evals don’t measure context size, so I don’t have results to share on that.

Picking the right tool: This is what we’ve been focusing on with evals.

Take a notes server with two tools:

• find_notes filters by tag.
• search_notes searches the text of your notes.

“Find my notes tagged travel” should use find_notes. But an eval that only checks whether it called find_notes would also pass if it called search_notes too, for no reason.

So we check both: did it call the expected tool, and did it avoid the other one? Then flip the test with “Which notes mention passport?” and run both cases several times to see how consistent the choices are.

That tests tool selection. It doesn’t tell us whether the agent can reliably chain several calls together, or whether code orchestration would work better. Same with checking a tool’s arguments or output: that tells you one step worked, not that the whole workflow did.

When you’ve had trouble with MCP, what went wrong? Wrong tool, trouble using the results, too much context, or something else?

I work at MCPJam, and we’re preparing our own submissions to Claude’s connector directory and ChatGPT’s app directory. I’d like to hear from developers who’ve already gone through review or are still waiting.

I’m interested in both the timeline and what happened during the review itself. How long did it take to get an initial response? Was that an approval, a request for changes, or just confirmation that your submission was being reviewed?

A few things I’d appreciate hearing about:

• Which directory did you submit to, and roughly when?
• If you’re still waiting, have you received any updates since submitting?
• If you were asked for changes or rejected, what feedback did you get? Was it specific enough to know what to fix?
• If you resubmitted, how long did the next round take?

Also, looking back, was there anything you wish you’d tested or documented before submitting? I’m especially curious about things that worked in your own testing but came up during review.

Experiences with either directory are appreciated!