u/asadlambdatest

Almost every server I add gives the agent another way to do something. Read the file system, hit an API, query a DB, control a browser, send a message. Which is genuinely great, the agent can touch more of the world now.

What almost none of them give it is a way to check whether the thing it just did actually worked. The agent calls the tool, gets a success response, moves on. But "the tool returned 200" and "the outcome the user wanted actually happened" are different claims, and nothing in the loop closes that gap.

Concrete version: agent edits a web app, says done. It had filesystem tools, maybe a shell, maybe even a browser tool it used to poke around once. None of that tells it whether the login flow it just changed still works. No feedback signal, so it asserts success and stops.

I've started thinking the missing primitive isn't another action tool, it's a verification tool. Something the agent calls after it acts that hands back real ground truth it can't talk its way past. Run the actual flow, check the actual result, return a pass or fail the model didn't author itself.

Is anyone here building this side of it? Feels like the whole ecosystem is optimized for "give the agent more capabilities" and nobody's on "give the agent a way to find out it was wrong." Where's the verification layer in your setup, if it exists at all?

I spent a long time treating flakiness as a timing problem. Added waits. Added retries. Bumped timeouts. Tests still went red on CI and green on my machine, the usual.

When I finally looked at what was failing, almost none of it was timing. It was selectors. A class name changed, a wrapper div moved, someone restructured a component, and a test that asserted nothing about that change broke anyway. The test wasn't testing the feature, it was testing the DOM shape, and the DOM shape changes constantly.

Two things actually moved the needle for me:

The boring one: stop selecting on structure. Lean on roles and accessible names and user-visible text instead of CSS paths and nth-child. Playwright's getByRole and getByText already push you this way. Most of my flake came from the handful of places I'd ignored that and reached for a brittle selector because it was faster to write.

The less boring one: for the flows that kept breaking, I started describing the intent ("click the submit button in the checkout form") and letting that resolve at runtime, with a fallback when the obvious locator moves, then exporting the result back to a normal Playwright script once it was stable. It killed a category of flake, but I want to be honest about the cost: it's less deterministic, it's slower to run, and for anything critical I still pin an explicit locator and a hard assertion. It's a tool for the churny parts, not a replacement for the suite.

Anyway, curious what everyone else's actual number one is. Not the textbook answer, the thing that genuinely cut your flake rate. Trying to build a real list.

Something that took me way too long to admit: when my coding agent finishes a task and says it's done, it usually has no idea whether the thing works. It wrote the code, it read the code back, it decided the code looks right. It never opened a browser. It never clicked the button it just built.

So I'd get "done," go check myself, and the login flow would be broken in a way that was obvious the second a real page loaded.

For a while my fix was telling the agent to write Playwright tests. That helped, but it just moved the problem. Now the agent is writing test code it also can't run and confirm, and I'm reading both. Half the time the test passed because the selector was wrong, not because the feature worked.

What's actually closed the gap for me is giving the agent a way to drive a real browser and report back in plain language. Not "generate a test file," but "go to the staging URL, log in with these creds, tell me if you land on the dashboard." Real Chrome, actual pass or fail, and that result goes back into the agent loop so it can fix and recheck instead of guessing.

Caveats, because this isn't magic. It's slow, you're literally booting a browser, so don't try to run a thousand of these in CI. It's flaky enough that I'd still keep a real e2e suite for anything critical. And it obviously can't help you if there's no browser involved, it's useless for pure backend stuff.

Mostly curious how other people here are handling this. Is anyone actually wiring browser-level verification back into their agent loop, or are you all just eyeballing the output like I was for months?