Agent marked "done & tested", CI was green, PR merged. broke in staging anyway.
my agent recently rewrote the pagination logic in a data-sync job and reported "✅ done and tested." CI passed because the test fixture only had one page of data. next morning in staging, the sync died after 100 records—the cursor never advanced.
to make it worse, i had the same agent review its own PR. it approved it with flying colors.
having the model that wrote the code review its own PR is useless. it’s just grading its own hallucination with the exact same blind spots. asking an agent to "double-check your work" just gets you a super confident "yep, looks good!"
i’ve had to strictly decouple generation from verification. now, whatever writes the code isn't allowed to validate it.
i started by running fresh, separate review sessions manually, but it was tedious and easy to forget. eventually, i started using minmax to automatically route the validation step to a completely isolated model instance one with no prior conversation context or inherent bias from the generation prompt. its only job is to play devil's advocate, inspect the code, and challenge assumptions.
it's not 100% bulletproof for open-ended architecture, but it completely eliminated those dumb, self-rubber-stamped logic bugs overnight.
curious how others here are handling generation vs. verification? anyone found a rock-solid workflow to force the verifier to actually execute tests and inspect runtime outputs rather than just scanning static code?