Opus 5: watch out for contradicting instructions
▲ 3 r/Claudeopus+1 crossposts

Opus 5: watch out for contradicting instructions

TL;DR: if two instructions in your CLAUDE.md contradict, Opus 5 picks whichever one sits lower in the file (or more accurately: whichever is more recent) and drops the other.

longer version:

opus 5 wont flag a contradiction or split the difference. It picks one rule, commits to it, and you find out which rule was followed by reading the actual diff. In Opus 5 position decides which one it keeps: the rule lower in the file (read most recently) governs, the earlier one is ignored, so the rule you put first is sitting in the weakest spot from this perspective.

Now the fun(?) part: older models used to waver* between two fighting instructions, and that was a tell that something was off. Opus 5 however works a bit differently, it will confidently resolve the contradiction (mostly wrongly) and commit way harder to it, so the telltale goes away. The built in model level judge is just confidently wrong (well, debatable, it's not the model is wrong, the instructions are contradicting).

these rarely look like "do X" right above "dont do X". Usually present as:

  • same subject under two headings (Testing vs CI)
  • different words for the same thing (never use mocks vs wrap calls in a fake)
  • a general rule and an exception written like a new topic
  • root CLAUDE.md vs a nested src/CLAUDE.md

you can catch it more easily by reading from top to bottom. Grab two rules that touch the same thing (tests, deps, what ships without review) and ask: can both be true at once? if not, opus 5 will chose for you and you're not gonna like it.

*waver: two contradicting instructions leave the distribution over next actions relatively flat, with neither strongly peaked, so which instruction gets sampled swings with temperature and tiny context shifts. It is sampling variance: a flat distribution resolves to a different winner each run.

Disclaimer: I'm working on deterministic instruction diagnostics and evals, the observed behaviors are coming from controlled experiments, the frequent "hiding" places are coming from a large corpus analysis (source: https://www.reddit.com/r/ClaudeCode/comments/1srm2vv/we_analyzed_12356_repos_with_claudemd_files/ )

u/cleverhoods — 6 days ago
▲ 146 r/AskVibecoders+2 crossposts

Small demonstration of how instructions decay as the context window fills

This weekend I put together a small demo page to show how instructions "decay" as the context window fills up and how well formatted instructions can "fight" against it. Some instructions should be supported with hooks (for example "Never commit secrets", some only missing the correct formatting.

I always wanted to find a good presentation for this, so... here it is, I hope you lot find it useful.

Full write-up link with the reasoning behind it is in the comments (I cannot share link in the body without tripping off Reddit filters)

u/cleverhoods — 1 month ago
▲ 2 r/LangChain+1 crossposts

How do you actually test a self-improving agent?

The closest thing to an answer I've seen is SkillOpt (Yang et al., 2026): you let it rewrite its own skill docs, and you reject most of the rewrites. It keeps a self-authored edit only when a held-out score improves - not the score the edit was tuned against - and most edits don't clear that bar.

I hit the same wall in plain code, no agent involved. I have an architecture test that's been green on every commit for months; it checks my pure core imports nothing it shouldn't.

Last week I went to see whether the refactor it guards actually made changes cheaper, the way it was supposed to. It can't tell me. The test knows the import graph follows the rule. Whether following the rule made a single change cheaper is something it never looks at.

So I started writing a one-line prediction for the changes that make a measurable claim - what should move, where it's measured, when - and letting a script grade it at the due date. I figured most would come back confirmed. Most come back inconclusive or refuted; confirmed is the minority.

Same lesson as SkillOpt: the thing that grades a change has to be separate from the thing that shaped it, or confirmed just means you set the bar on what you were already moving.

Full writeup with the 2×2 and the spec-driven/hypothesis-driven framing and the SkillOpt research links in comments.

reddit.com
u/cleverhoods — 2 months ago

Remember when they searched for rockstar developers?

… until someone made the “Rockstar” language, most likely out of pure spite

u/cleverhoods — 2 months ago