We built an agent that scores its own output before it opens a PR — the architecture, and the three things that broke
Disclosure up front: I work on this. It's a commercial product (KeplerCrew, by AiChargeLabs). Happy to talk architecture either way, and I'd rather get torn apart here than in a sales call six months from now.
The problem we kept running into with agentic coding wasn't generation quality. That was fine. It was that nothing in the loop could tell us whether the output was actually correct before a human looked at it. So every change still queued behind a reviewer, and the reviewer was now reading more code than before. Net throughput barely moved. Faster typing, same gates.
What we ended up building is five stages, with sixteen phases distributed across them:
- Understand — reads the repo, its conventions, and the task intent
- Plan — decomposes the work into an ordered, safely sequenced plan
- Execute — writes the code and the tests against that plan
- Validate — scores the result against acceptance criteria; failures loop back into a fix cycle instead of surfacing
- Deliver — the verified diff lands as a pull request
Stage 4 is the part I think actually matters. Criteria get scored at every gate rather than once at the end, and a failed gate re-enters the pipeline instead of being handed to a human as "here's my attempt, good luck." The goal isn't to remove the reviewer — it's that the reviewer shouldn't be the one finding the bugs.
Three things that were harder than we expected:
Safely sequencing the plan. Naive decomposition produces steps that are individually valid and collectively broken — each one passes, the composition doesn't. Most of our planning work went into ordering and dependency detection rather than into the decomposition itself.
Cost predictability. Open-ended agent loops are financially unbounded by default. A task that retries its way to correctness can cost ten times what a similar task cost yesterday, which makes the whole thing impossible to budget. Capping spend per task without capping quality took more tuning than anything else we did.
Running with no egress. A lot of our buyers are regulated and their code cannot leave their network, so we support self-hosted and fully air-gapped deployment. Good for those deals, painful for every part of the system that quietly assumed it could make an API call.
The open question I'd actually like opinions on: how much of the review burden do you think can move to automated scoring before you'd stop trusting it? We've landed on "a human still approves the PR, but shouldn't be the first line of defence." I'm not certain that's the right line, and I'd rather hear where you'd draw it.
Happy to go deeper on any of the stages, the scoring model, or the air-gapped setup.