I built a harness to check if an AI agent's work actually works
When I ask an AI agent to "make the tests pass," I've noticed it'll often do exactly that, but not always in the way I intended. It might weaken an assertion, stub a response, or skip the failing case. It follows the instruction, but not necessarily the goal.
To experiment with this, I built a small harness that keeps the e2e tests outside the agent's control. The agent can only change the application code. After every change, the harness reruns the original tests, captures any failing service logs, and sends them back to the agent until the actual test passes.
I'm not sure if this solves a problem other people actually have, or if better prompting is enough. It might also just be something that's only useful for my workflow.
If you've worked with coding agents, I'd really appreciate your thoughts. Is this something you'd use? Or is there a simpler way to solve the same problem that I'm missing?
