My agent kept losing track of itself between sessions, so I rebuilt the harness instead of switching models
Spent most of this year assuming a better model would fix the reliability problems I was seeing. Wrong assumption. My agent would repeat a step it already finished, or start a task fresh with no memory of being halfway through it the session before. Swapping models changed nothing because the problem was never in the model.
What mattered was three things sitting underneath it: something tracking what it had already done, something loading context before it took its first action, and something checking its output before letting it move to the next step. Once I split those out as separate pieces instead of letting the agent reason about all of it in one context window, the flakiness dropped a lot. The checking part mattered most. Letting the same context that generated an answer also grade it means a confident wrong answer sails through every time.
The part I'm still working through is versioning that logic. I had three slightly different copies of a state tracker across three repos, and fixing a bug in one meant remembering to go fix it in the other two by hand. Tried a private npm package first, which works but adds a publish step I kept forgetting to run. Currently testing a setup where the harness pieces live in a shared scope and get pulled into each project as versioned components, so a fix in one place propagates without me manually syncing files. Feels closer to how I'd want infra treated, but I've only been running it a couple weeks, so I don't have a verdict on whether it holds up at scale.
What's everyone else doing here? Are you packaging harness logic as a real dependency, copy-pasting, or is copy pasting between repos still the norm for most people?