u/chancemehmu

Been thinking about this a lot lately. Agents are getting scary good at the mechanical stuff - searching, calling APIs, writing code, executing multi-step plans. But they still face two problems that no amount of scaling fixes:

1. They hit decision points where the "right answer" is a judgment call, not a logic problem. Is this email tone too aggressive? Which of these three landing page headlines actually lands? Does this UI feel sketchy to a normal person? Models have priors on this stuff but their priors are an average of the internet, not your actual users.
2. You can't eval them on anything subjective without burning a week recruiting people, building a survey, paying a panel, etc. So most teams just don't, and ship on vibes.

I built an MCP server that solves both. Agent hits a fork in the road, calls the tool with a question + audience (e.g. "US women 25-34" or "developers who've used Cursor"), and gets back actual human responses in seconds. Not synthetic. Not Mturk graveyard. Real people replying within seconds.

Example from last week - someone wired it into a Claude Code agent generating marketing copy variants. Instead of picking the "best" one itself, the agent fires off 4 versions to 200 people in the target segment, gets back preference data, and only then commits.

Same primitive works for eval generation. Want a 500-person benchmark on whether your agent's outputs feel trustworthy? One tool call.

Anyway - curious if anyone else is doing the human-in-the-loop thing for agents, and how?

Most stuff I've seen is either slow HITL or pure LLM judge (cheap but circular).

I built a tool where you type a question, it goes out to real consumers (demo-targeted), and an AI layer synthesizes the responses. Marginal cost is ~5¢/response, so n=1000 runs about $50 and completes within a few hours.

The economics are cheap enough that I think it unlocks survey work that currently doesn't happen — stuff that's too small to justify a real study but too important to just guess on.

What I'm curious is If you were me, who's the first customer — agencies doing pre-tests, marketing teams testing ad creatives, founders doing pre-launch validation, somewhere else?

Hey,

I've building consumer apps for a while (recently scaled one to 150M+ users) and now deep in the weeds experimenting with a new ad-format.

We are seeing higher almost 50% higher CPMs than traditional mediation approaches with interstitial or rewarded ads, and recently launched self serve SDKs for both android and iOS.

If you are an android developer with an app that has at least 100k+ downloads, feel free to message me