Sat in enough estimation meetings to notice a pattern that has nothing to do with optimism or bad math. The question "how long will this take" gets asked as if it's one thing, and answered as if it's one number, when it's actually collapsing at least three separate questions that have different answers and different levels of certainty.

How long will the code itself take to write, assuming nothing surprising happens. How long will it take to actually figure out the right approach, which is a research problem, not a coding problem, and has no relationship to typing speed. And how long will it take to deal with whatever wasn't visible until the work started, the dependency nobody flagged, the edge case that only shows up once you're inside the problem.

Most estimates answer the first question and quietly present it as an answer to all three. That's not dishonesty, it's usually the only one that feels answerable at the time the question gets asked, since the second and third are genuinely unknown until you're partway into the work. But presenting an answer to "how long to type this" as an answer to "how long until this ships" is where almost every blown estimate actually comes from, not from the coding part running long, from the other two questions never having been asked separately in the first place.

What's helped, imperfectly: giving each of the three a rough separate number instead of one blended figure, even if two of them are honestly "I don't know, could be a day, could be a week." Doesn't make the uncertainty go away, but it stops the uncertainty from getting silently absorbed into a single confident-sounding number that nobody actually believes once things start slipping.

Curious how other teams handle this, whether anyone's found a format for surfacing "I actually don't know" as its own category instead of it getting rounded into the estimate anyway because a stakeholder wanted one number.

Noticed a pattern across a few different agent setups I've built or debugged: the failures that cost the most time aren't crashes or errors, they're agents that keep working, keep calling tools, keep producing plausible-looking output, while making zero actual progress. A retry loop that never escalates. A research agent that re-fetches the same source with slightly reworded queries because the earlier fetch didn't satisfy the objective, but nothing told it to recognize that and try a different approach instead of a different phrasing of the same approach.

The common thread isn't a bad model or a bad tool. It's that most agent setups define what the agent can do, but not what counts as "this isn't working, stop and escalate." A human running the same task recognizes stuckness almost automatically, three failed attempts at the same thing reads as a signal to change strategy. An agent has no equivalent signal unless something explicitly gives it one. Left alone, it just keeps sampling from the same distribution of "reasonable next action" and produces a slightly different variation each time, which looks like progress in the trace even when it isn't.

This seems like the actual gap between "agent with tools" and "agent that's reliable in production." Tool access solves capability. It does nothing for knowing when the current approach has stopped being productive. That has to be its own explicit check, something closer to a circuit breaker than a prompt instruction, comparing the current state against the last N states and forcing a strategy change or a handoff to a human once repetition crosses some threshold, rather than trusting the model to notice on its own.

Curious how people here are actually implementing that in practice: hard iteration caps with forced escalation, a separate model call that periodically judges whether the last few steps made real progress, or something else entirely? Feels like this gets skipped in a lot of agent architectures until it causes a production incident.

Noticed this after blaming a model for a string of bad outputs on a task I'd been running for weeks. Same model, same general request, quality visibly declining. Went back and compared my actual messages over that period instead of assuming model drift.

Turned out I'd been getting lazier, not the model. Early requests spelled out constraints explicitly. Later ones assumed the model would infer them from earlier context, patterns established messages ago that I stopped restating because saying them again felt redundant. Except redundant to me isn't the same as redundant to whatever's actually shaping the next response. The constraints I stopped stating were exactly the ones that stopped showing up in the output.

Uncomfortable thing to notice about your own habits, since "the model is inconsistent" is a much more satisfying explanation than "I got sloppier once the first few responses were good and I relaxed." Curious if others have caught this in themselves, mistaking your own growing laziness for the model's declining quality.

Technical debt used to mean something vague and slightly accusatory in retros, "we have a lot of debt in that service," with nobody able to say specifically what, when it was taken on, or whether it was a deliberate tradeoff or just something that happened. Half the debt conversations turned into archaeology, trying to reconstruct why a shortcut existed months after whoever took it had moved on or forgotten.

Started keeping a dead simple log, one line per deliberate shortcut, right when it happens, not retroactively:

2026-06-03 — Skipped input validation on the bulk-import endpoint.
Reason: internal tool only, low traffic, ship date mattered more.
Revisit if: exposed externally, or import volume grows past ~500/day.

That's it. Date, what got skipped, why, and the condition that should trigger revisiting it. Doesn't need to be longer than that to be useful.

What changed wasn't the amount of debt, that stayed roughly the same. What changed was that "should we deal with this now" stopped being a vague argument about how bad things felt and became a check against a condition someone had already written down. The bulk-import endpoint got flagged for revisit six months later, not because someone remembered the tradeoff, but because import volume actually crossed the number in the log, and the log made the trigger checkable instead of a gut call.

The bigger shift was in how shortcuts got taken in the first place. Writing the "revisit if" line at the moment of the decision forces you to actually think about what would make the shortcut wrong later, instead of just knowing vaguely that it's not ideal. A surprising number of shortcuts turned out to not have a clean revisit condition at all, which was itself useful information, if you can't articulate when this would become a problem, that's worth noticing before shipping it, not after.

Curious how other teams track this instead of letting it live as institutional memory that erodes the moment someone leaves. Anyone doing something similar, or is verbal/retro-based tracking still the norm most places?

Small habit change that had a bigger effect than expected. Before asking for a fix or a feature addition, started asking the model to first explain what a piece of code was actually doing, in its own words, no changes, just a plain description of the current behavior.

Expected this to be a formality, mostly useful for onboarding someone else onto unfamiliar code. Turned out to be useful on my own code, code I'd written and thought I understood completely. The model's explanation occasionally didn't match what I thought the function did, and in a couple of cases, the model was right and I was wrong about my own logic. One function I'd assumed handled a specific edge case actually didn't, the explanation described behavior that only looked correct because the edge case in question had just never come up yet.

What's interesting is why this works better than just re-reading the code myself. Explaining requires committing to one specific interpretation instead of holding a vague, flexible sense of "yeah, this looks about right" in your head. A model forced to state plainly what a function does can't hedge the way a quick visual skim lets you hedge. Either its stated interpretation matches your intent or it doesn't, and the mismatch is obvious immediately instead of staying buried in an assumption you never examined closely.

Doesn't replace actual testing, obviously, an explanation being wrong doesn't guarantee the code is broken and an explanation being right doesn't guarantee it isn't. But it's turned into a cheap first-pass check I run before touching anything, especially on code I wrote a while ago and am about to modify without having fully reloaded the context in my head first.

Anyone else use "explain this before I ask you to change it" as a deliberate step, or is jumping straight to the fix/feature request the more common workflow? Curious if this is specific to older or unfamiliar code, or if people find it useful even on stuff they wrote last week.

Genuine question, not a claim dressed up as one. Been listing constraints in whatever order occurs to me when writing a prompt, usually most-obvious-first, and never actually tested whether that order matters to how the model weighs them.

Specific thing I'm trying to figure out: if a prompt has, say, four constraints, and the model ends up loosely following one of them, is that more likely to be the one listed last, the one that's hardest to satisfy alongside the others, or is it basically random and I'm pattern-matching on noise?

Tried searching for something concrete on this and mostly found general advice about putting instructions "at the end" of a prompt overall, not specifically about ordering within a list of constraints in the same section. Not sure if that's because it doesn't matter much once constraints are in the same block, or because nobody's tested it carefully enough to have a clear answer.

Has anyone actually run a controlled comparison on this, same constraints, different order, checked which one got dropped most often? Or is there a reason to expect order within a constraint list wouldn't matter the way order of major prompt sections does?

Wanted to stop assuming and actually look. Took an eval prompt from a real pipeline, temperature locked at 0, same model, same input, ran it fifty times back to back. Not a rigorous study, just curious how much drift was actually there versus how much I'd been imagining.

Most runs clustered tightly, as expected. A meaningful minority didn't, different enough in structure or emphasis that if I'd only seen two of them side by side, I would have called one of them wrong. Went through the outliers individually instead of averaging them away.

What I expected to find: random noise, maybe some inherent sampling variance even at temperature 0, nothing systematic. What I actually found: almost every outlier varied along one of three predictable axes. How much the response hedged versus asserted. Whether it prioritized brevity or completeness when the two traded off. What it assumed about who'd be reading the output.

None of those three things were specified anywhere in the prompt. Not omitted by accident exactly, just never occurred to anyone that they needed to be, because in a single run they don't visibly matter, the model just picks one and moves on. It's only across many runs that the gap becomes obvious, since each run resolves the same missing spec independently and inconsistently.

Reframed how I read eval failures after this. A flagged mismatch used to send me straight to "which of these is correct." Now the first question is which of the three axes moved, and whether that axis was ever actually pinned down in the prompt, or genuinely left for the model to decide.

Curious if others running repeated-sample evals have found a similarly small set of recurring axes explaining most of their variance, or if it's noisier and more scattered than what I saw here.

Ran into this constantly early on and treated it as a bug every time: same prompt, same model, two runs, genuinely different answers, not wildly different, but different enough to matter. Instinct was always to figure out which one was correct and patch the prompt to force that answer consistently.

Stopped doing that after noticing how often the two outputs weren't actually contradicting each other, they were answering slightly different implicit questions the prompt had left open. A summarization task with no stated audience produces a technical summary one run and a plain-language one the next, both valid readings of "summarize this," neither wrong, the prompt just never disambiguated which reading it wanted.

That reframes the debugging question. Instead of "which output is correct," the more useful question turned out to be "what latent choice did the model make differently between these two runs, and was that choice ever actually specified." Usually it wasn't. Audience, priority between competing goals, level of confidence to express, format assumptions, all places where an unspecified prompt hands the model a decision it then makes inconsistently across runs, not because the model is unreliable, but because there genuinely wasn't a single correct answer given what it was told.

Practical thing that's helped: when two outputs disagree, before rewriting the prompt, diff them for what dimension they actually vary on, not just that they vary. Half the time that dimension turns out to be something nobody had pinned down in the first place, and specifying it directly resolves the "inconsistency" faster than any amount of prompt rewording aimed at forcing one specific output.

Curious whether others doing evaluation or output comparison at any scale have built this distinction into their process, treating variance as a signal pointing at an underspecified dimension rather than defaulting to "the model got it wrong this time."

Payment webhook handler. Intermittent 500s.

I pasted the error into an AI coding assistant, got a fix, tried it, still broke.

Pasted the new error, got another fix, tried that, still broke.

Did this maybe four times before realizing what I'd actually turned into: not someone debugging anymore, just someone pasting error messages into a chat window and hoping the next response would be the one that stuck.

The interesting part was that none of the suggestions were obviously stupid.

The first was a retry around a database write. Reasonable response to "database write failed."

Except the actual problem was duplicate webhook delivery upstream hitting a handler that wasn't idempotent. Two workers were occasionally processing the same event.

The retry addressed the symptom I'd shown the model, not the mechanism producing it.

I then tried the obvious solution: give it more context.

That made things worse.

I pasted more surrounding code, but the context I added was already biased by my own suspicion. I'd started thinking the caching layer was involved, so I gave the model more caching-related code.

It reasoned confidently about the wrong subsystem.

That's when I realized I'd been mixing up two completely different tasks:

Generating a fix and validating a fix.

Generating asks:

>"Does this make the error go away?"

Validating asks:

>"Does this address the mechanism that caused the failure, and what does it change that I didn't explicitly ask for?"

Almost every one of those first fixes could have passed the first question.

None had passed the second.

What finally broke the loop was changing the process:

• define what's actually failing before asking the AI to diagnose it
• separate facts from hypotheses
• ask for competing explanations before asking for fix code
• understand the failure mechanism first
• validate the proposed change against the original failure
• add a regression test that reproduces the actual bug

The biggest lesson for me wasn't "AI is bad at debugging."

It was that a plausible fix is dangerously easy to mistake for a diagnosis.

Curious if other people doing AI-assisted debugging have run into this: a fix technically resolves the error you showed the model, but leaves the underlying problem untouched (or introduces a different one).

How do you validate AI-generated fixes before they reach production?

Had an AI feature inside the product that kept producing inconsistent output, sharp and usable some days, generic filler that needed a manual rewrite on others. First instinct was "the model isn't good enough for this," so I upgraded. Then upgraded again a few weeks later when the problem didn't actually go away, just got slightly less frequent.

Only fixed it once I stopped comparing models and started comparing my own requests to the model side by side. The pattern had nothing to do with which provider I was on. It tracked almost exactly with how much I'd actually specified versus assumed would carry over between calls, whether the prompt behind that feature had a defined role, a stated objective, explicit constraints on what not to output, and a fixed format, or whether it was closer to a quick instruction someone (usually me) had typed once and never touched again.

The part that actually cost money: two model migrations, each with its own testing and rollout overhead, to fix a problem that was sitting in application code the whole time, in a prompt nobody was treating as something worth reviewing or version-controlling.

What changed after: any prompt behind a user-facing AI feature now gets reviewed like an API contract, not edited inline the moment a bug report comes in. Sounds obvious written down. Wasn't obvious while I was mid-support-ticket assuming the fix had to be a bigger model.

If you're running AI features in a SaaS product and output quality feels inconsistent, worth checking whether it's actually a model problem before paying to switch, in my case it wasn't even close.

Used to treat inconsistent output as a wording problem first, rephrase, add an example, try again. Started checking a fixed list before touching the wording at all, and most of the time the actual issue was on that list, not the phrasing.

Role, stated narrowly enough to actually constrain behavior, not "helpful assistant." Objective, since a request optimizing implicitly for speed versus thoroughness versus teaching produces different output from the same surface task, and leaving it unstated means the model picks one inconsistently between sessions. Constraints, which usually do more work than positive instructions, telling it what not to do eliminates categories of bad output that no amount of "please do X" reliably prevents. Output format stated as a rule, not inferred from a single example. And tone, especially whenever the output is going straight to someone else instead of staying private.

Whenever output feels randomly worse than it was yesterday, checking which of these five quietly went missing usually explains it faster than any amount of rewording the actual request. Rewording without checking this list first tends to just produce a different flavor of the same underlying gap.

Curious what's on other people's pre-flight list before assuming a prompt itself is broken. Feels like most "prompt engineering" advice focuses on wording tricks, when the actual failure is usually one of these being silently absent rather than badly phrased.

The rollback mechanism itself took maybe two days. Detect a regression against a baseline metric, trigger, revert to the last known-good version, done. Felt like the real engineering work of the sprint.

What ate the rest of the week was everything the rollback assumed already existed and didn't. "Last known-good version" turned out to be a genuinely fuzzy concept once I went looking for it. Good by what metric, measured over what window, and was that version even still compatible with the current feature schema, since two upstream changes had landed since it was last serving traffic. The rollback code was fine. The thing it was rolling back to was the part nobody had kept honestly documented.

Ended up spending more time building a lightweight registry that tracked, per deployed version, exactly which metric windows it had cleared and what schema it assumed, than I did on the actual revert logic. Felt like scope creep in the moment. In hindsight it was the actual project, and the rollback trigger was the easy 20% that happened to be visible from the ticket description.

Not sure if this is a me problem or just a common shape these projects take. Curious whether other teams building rollback or deployment safety nets found the same thing, that the mechanism was the easy part and the actual work was in defining what "safe to roll back to" even meant in a system that keeps changing underneath it.

Small naming change that had a bigger effect than expected. Kept noticing that when people on the team talked about "the prompt," they treated it like a one-off message, something you'd tweak in the moment and not really think about again. When the exact same content got referred to as "the spec" instead, people treated it completely differently, worth reviewing, worth version-controlling, worth having someone other than the original author look at before it shipped.

Nothing about the actual artifact changed. Same role definition, same constraints, same output format requirements. Just the word attached to it shifted what category of thing people mentally filed it under. "Prompt" reads as disposable. "Spec" reads as something you maintain.

That distinction seems to matter more than it should, mechanically speaking, but it tracks with how the same team already treats other artifacts. Nobody reviews a Slack message like a PR. Everybody reviews an API contract like one. The system prompt sits closer to the contract end of that spectrum in terms of actual impact on behavior, but it kept getting treated with Slack-message level of rigor because of what it was called and where it lived, usually a raw string buried in application code, not somewhere that invited scrutiny.

Started keeping specs in their own reviewable files after that, separate from the code that calls them, with the same PR process as anything else. Didn't change the model. Changed whether a second person ever looked at the thing actually driving behavior before it shipped.

Curious if others have run into this, where the informal framing of "just a prompt" quietly lowered the bar for how carefully a team treated something that was functionally deciding a lot of downstream behavior. Or is this specific to teams still early in treating LLM behavior as something that needs the same rigor as other production logic?

Intermittent failure on a checkout flow, worked fine in staging, failed maybe one in twenty times in production, no pattern I could pin down for the first two days. Went down every interesting rabbit hole first, race condition in the payment callback, some kind of caching inconsistency between regions, a timing issue with a webhook. Spent real time on each of those, ruled them all out one by one.

Turned out to be a trailing space in an environment variable that only existed in one of three production instances, added months earlier by someone copy-pasting a value from a Slack message instead of typing it. Compared exact string, failed silently on that one instance, worked everywhere else, hence the "one in twenty" pattern, since requests got load balanced across instances and only hit the broken one some of the time.

What got me afterward wasn't the bug itself, it was how long I avoided checking something that boring. Kept reasoning my way toward more sophisticated explanations because a whitespace typo felt like too small an answer for three days of investigation. There's a weird bias where the amount of time already spent searching makes you expect the answer to match the effort, like the bug owes you something more interesting for the trouble.

Started deliberately checking the dumbest possible explanation first now, no matter how much time has already gone into a search, specifically because the sunk cost makes it feel unjustified to check something trivial, which is exactly the moment it's worth doing anyway.

Anyone else notice that bias, where the longer you've been debugging, the less likely you are to check the boring explanation, even though the odds it's something boring don't actually go down the longer it takes?

Took an embarrassingly long time to name what was actually going wrong. Kept getting review comments back from the model that felt technically fine and were completely useless in practice, "consider adding error handling" on code that already handled errors, approvals on things that shouldn't have been approved. Assumed the model just wasn't good enough yet.

The actual issue had nothing to do with model capability. "Review this code" isn't a testable request, it doesn't specify what's being checked against what standard, so there's no way to fail it. A model asked a vague question gives back a plausible answer, and plausible isn't the same bar as correct.

What eventually fixed it was treating the whole thing less like a single request and more like a pipeline with actual gates. Context established before anything gets evaluated, what the system does, what depends on it, what constraints actually matter, so the review isn't operating blind. Scope declared explicitly, security pass, performance pass, architecture pass, run separately instead of blended into one unfocused check. And critically, a validation step against an actual checklist instead of a gut read, does this match known failure patterns, is this claim testable, does the fix introduce new risk, because "looks right" is exactly the kind of soft judgment that let a race condition through undetected in my case until it took down something in production three days later.

The step I underestimated most going in: explicitly asking the model to argue against its own findings before accepting them. Models are noticeably better at finding holes in a claim when told to look for holes than at self-flagging blind spots by default.

Curious whether anyone building agents that do code review specifically has run into the same "vague request in, plausible-but-wrong output out" pattern, and if a staged/gated approach like this holds up once the agent is fully autonomous instead of a human reading each pass.