u/No_Skill_8393

Agents made running experiments cheap. They also made fooling yourself cheap.

Point one at a prompt tweak and it'll happily decide what "better" means after it sees the results, test five cases and call it a trend, then score its own output a 9/10. That isn't a finding. It's a vibe with a percentage stapled on. It's also why nobody believes the numbers in half the "I improved X" posts here.

crucible is a plugin that makes the agent do the study properly, by refusing to let it skip the three things that make one hold up:

Pre-register. The hypothesis and the bar for success get hashed and frozen before any data exists. Try to move the goalposts later and the tool catches it.

Power. It computes how many cases you need for the effect you care about. Run fewer and it won't say confirmatory, it says exploratory.

Blind judge. A separate agent scores the outputs without knowing which one is your new version, and it has to match human-labeled examples first or its vote doesn't count. No grading your own homework.

Clear all three and you get an effect size, a confidence interval, and a receipt. Miss one and it tells you which, in plain words.

I read the field before building. Every "AI scientist" and eval tool I found only advises; the model can nod and ignore it. Sakana's got audited at 57% invented numbers while reviewing its own work. Google's co-scientist scores itself on a metric its own paper says isn't ground truth. crucible's gates are code, so the agent can't opt out.

Free, MIT, Claude Code and Codex today, OpenCode next.

/plugin marketplace add nagisanzenin/crucible
/plugin install crucible@crucible

Then: "run a study with crucible on whether B beats A." It drives the rest. Codex build and setup are in the repo.

github.com/nagisanzenin/crucible

Point it at something you're sure works and see if it survives. I'd love for you to try to break it. That's the point.

Agents made running experiments cheap. They also made fooling yourself cheap.

Point one at a prompt tweak and it'll happily decide what "better" means after it sees the results, test five cases and call it a trend, then score its own output a 9/10. That isn't a finding. It's a vibe with a percentage stapled on. It's also why nobody believes the numbers in half the "I improved X" posts here.

crucible is a plugin that makes the agent do the study properly, by refusing to let it skip the three things that make one hold up:

Pre-register. The hypothesis and the bar for success get hashed and frozen before any data exists. Try to move the goalposts later and the tool catches it.

Power. It computes how many cases you need for the effect you care about. Run fewer and it won't say confirmatory, it says exploratory.

Blind judge. A separate agent scores the outputs without knowing which one is your new version, and it has to match human-labeled examples first or its vote doesn't count. No grading your own homework.

Clear all three and you get an effect size, a confidence interval, and a receipt. Miss one and it tells you which, in plain words.

I read the field before building. Every "AI scientist" and eval tool I found only advises; the model can nod and ignore it. Sakana's got audited at 57% invented numbers while reviewing its own work. Google's co-scientist scores itself on a metric its own paper says isn't ground truth. crucible's gates are code, so the agent can't opt out.

Free, MIT, Claude Code and Codex today, OpenCode next.

/plugin marketplace add nagisanzenin/crucible
/plugin install crucible@crucible

Then: "run a study with crucible on whether B beats A." It drives the rest. Codex build and setup are in the repo.

github.com/nagisanzenin/crucible

Point it at something you're sure works and see if it survives. I'd love for you to try to break it. That's the point.

The plugin changes what Claude says to you, not how it works. That distinction is the entire point, so I will start there.

Claude Code is verbose by default. Preamble, a restatement of the request you just typed, "Great question", and a closing summary that repeats the summary from the middle. I read walls of text badly and they wear me out. This has annoyed me since months, so I wrote something that makes it answer first and drop the filler. It is called less.

The naive version is one line: tell the model to be concise. That version is worse. It starts cutting caveats, shortening the actual work, and skipping the one file path you needed. Concise and lobotomised are easy to confuse if you optimise the wrong layer.

So less does not touch the work. Reasoning, tool calls, and other plugins run at full depth. It governs the user-facing prose only, and it is instructed in writing to keep the load-bearing facts: numbers, paths, risks, anything you cannot undo. Minimal, not missing. When there is a decision it gives you a, b, c, and you answer with one letter.

Mechanically it is boring, which I mean as the compliment. A UserPromptSubmit hook injects a short protocol into the context each turn. It only adds, it replaces nothing, so it cannot fight your other plugins. Three modes: on is the default, zen is near silent, off is plain Claude. You switch with /less.

The whole plugin is a hook script and a markdown file. I spent longer on the markdown file than on the code, which tells you where the difficulty actually sat.

One honest limitation. I cannot measure "less tiring", it is subjective and I have no benchmark, so treat it as a preference and not a result. It is also a nudge repeated each turn, not a hard constraint, so across a very long session the model drifts back to chatty and you remind it.

MIT and free.
github.com/nagisanzenin/less
claude plugin marketplace add nagisanzenin/less
claude plugin install less@less

Title

Founder disclosure: I built this. It's on v0.1.1 because v0.1.0 lasted about two hours before my own test run caught three bugs.

Humanizer tools are good at what they do, which is deleting AI tells. The problem is everyone deleting the same tells ends up sounding like the same person, and that clean careful nothing-voice is quietly becoming its own tell.

idiolect goes the other way: 62 complete fictional people instead of one scrubbed nobody. There's an HVAC owner in Ohio who drops the apostrophe in "its" about once per post, never twice. His wife Diane runs the phones till 6. A Glasgow art student writes everything lowercase, capital letters feel like announcements to her. My favorite is the Houston banh mi truck owner. Each profile is a biography that generates opinions, plus systematic flaws with actual rules. And three example posts that each had to score under 10 on the linter before shipping.

I lost a chunk of today to my own conform checker failing five of my sample posts in a row for being too rhythmically even. It was right all five times, which was worse.

The linter's the other half: a dependency-free python CLI that flags 235 tells with line numbers, from era-tagged vocabulary (delve is a 2023 tell, the 2026 residue is different words) to the metronomic sentence rhythm models can't stop producing. One product brief through five voices scanned 0, 0, 0, 0 and 9 out of 100, against 84 for the chatbot control. The 9 was the art student, dinged by the burstiness metric because her clipped little lines run too even. Clipped is her whole personality. I let it stand.

The limit that actually worries me is decay, word-level tells rot as models update, so the lexicon ships as era-tagged data with a local extension file rather than something I pretend is finished. Polite testimonial slop gets past any regex, mine included, that one belongs to the blind auditor agent. There's no posting automation in the box: it writes drafts, you post them.

Anyway. Free, MIT: github.com/nagisanzenin/idiolect

Install: add "skills": { "urls": ["github:nagisanzenin/idiolect"] } to opencode.json, then bash scripts/install-opencode.sh for the three agents. Skills are model-invoked here, no slash commands to memorize, you just ask for the thing. Claude Code and Codex work too.

btw, this post was written by idiolect and graded by its own blind auditor. draft one got rejected with the note "a post about metronomic rhythm has metronomic rhythm." this is draft four, and the best grade the judge will give me is "mixed," because I built it to round against the writer and it doesn't care that the writer is me. so: if you thought a real human wrote this, the plugin worked. if you clocked it, the auditor already agrees with you, which means the other half works too.

Founder disclosure: I built this. It's on v0.1.1 because v0.1.0 lasted about two hours before my own test run caught three bugs.

The humanizer skill is good at what it does, which is deleting AI tells. The problem is everyone deleting the same tells ends up sounding like the same person, and that clean careful nothing-voice is quietly becoming its own tell.

idiolect goes the other way: 62 complete fictional people instead of one scrubbed nobody. There's an HVAC owner in Ohio who drops the apostrophe in "its" about once per post, never twice. His wife Diane runs the phones till 6. A Glasgow art student writes everything lowercase, capital letters feel like announcements to her. My favorite is the Houston banh mi truck owner. Each profile is a biography that generates opinions, plus systematic flaws with actual rules. And three example posts that each had to score under 10 on the linter before shipping.

I lost a chunk of today to my own conform checker failing five of my sample posts in a row for being too rhythmically even. It was right all five times, which was worse.

The linter's the other half: a dependency-free python CLI that flags 235 tells with line numbers, from era-tagged vocabulary (delve is a 2023 tell, the 2026 residue is different words) to the metronomic sentence rhythm models can't stop producing. One product brief through five voices scanned 0, 0, 0, 0 and 9 out of 100, against 84 for the chatbot control. The 9 was the art student, dinged by the burstiness metric because her clipped little lines run too even. Clipped is her whole personality. I let it stand.

The limit that actually worries me is decay, word-level tells rot as models update, so the lexicon ships as era-tagged data with a local extension file rather than something I pretend is finished. Polite testimonial slop gets past any regex, mine included, that one belongs to the blind auditor agent. There's no posting automation in the box: it writes drafts, you post them.

Anyway. Free, MIT: github.com/nagisanzenin/idiolect

Install: /plugin marketplace add nagisanzenin/idiolect then /plugin install idiolect@idiolect. Codex and opencode too, one config line each.

btw, this post was written by idiolect and graded by its own blind auditor. draft one got rejected with the note "a post about metronomic rhythm has metronomic rhythm." this is draft four, and the best grade the judge will give me is "mixed," because I built it to round against the writer and it doesn't care that the writer is me. so: if you thought a real human wrote this, the plugin worked. if you clocked it, the auditor already agrees with you, which means the other half works too.

A few months back I released production-grade, a free plugin that turns Claude Code into a 14-agent pipeline: PM, architect, backend, frontend, QA, security, the whole crew. It got some love here, then I went quiet because I was using it myself and kept hitting the wall everyone hits with agents.

The agent finishes, prints a lovely summary, claims every test passes. You open the app and half the buttons are decorative. My pipeline was one pass: each agent does its job, hands the work down, done. Real engineering is not one pass. It's loops. Write, run, fail, fix, run again.

v5.5 is one idea applied everywhere. An agent is not done when it says it's done. It's done when a check it cannot argue with says so. I'm calling it the Loop Engine.

Concretely:

The pipeline generates a fast check script for your project (typecheck + lint, under 15 seconds), and a hook runs it after every single file edit. Claude breaks something, the error lands back in its face immediately. Not at the end. Every edit.

Coding agents are banned from touching tests. QA writes failing tests first and owns the test folder. If a coder adds .skip or loosens an assertion to get green, that gets flagged as a critical finding. No more grading their own homework.

"12 test files written" stopped being an acceptable QA report. Suites have to actually run: executed, passing, failing.

Before the final gate, a separate agent boots your real app and drives it like a user. Every button clicked, every form submitted, every link followed. A button that renders but does nothing is a critical bug. This feature exists because I was tired of being my own QA department at 1am.

Loops stop on evidence, not vibes. Each one tracks a number (failing tests, open findings) and stops when it hits zero or stops improving, then shows you the trend, like 7 to 3 to 3, and asks. No retrying the same fix five times.

And when Claude hits something weird with no existing check, it can build its own loop, with one rule: first create a check it can run, like a failing repro script. No check, no loop.

Still free, MIT. Gates, receipts, and worktree isolation for parallel agents are unchanged.

Install:
/plugin marketplace add nagisanzenin/claude-code-plugins
/plugin install production-grade@nagisanzenin

Repo: github.com/nagisanzenin/claude-code-production-grade-plugin

Theory writeup lives in docs/LOOPS.md if you want the reasoning.

Honest caveat: this shipped this week. The edit-hook enforcement is real code, the rest is protocol the agents follow, and I want to see where they drift in the wild. If an agent cheats the rules or a loop converges badly on your project, I want the transcript.

A few months back I released production-grade, a free plugin that turns Claude Code into a 14-agent pipeline: PM, architect, backend, frontend, QA, security, the whole crew. It got some love here, then I went quiet because I was using it myself and kept hitting the wall everyone hits with agents.

The agent finishes, prints a lovely summary, claims every test passes. You open the app and half the buttons are decorative. My pipeline was one pass: each agent does its job, hands the work down, done. Real engineering is not one pass. It's loops. Write, run, fail, fix, run again.

v5.5 is one idea applied everywhere. An agent is not done when it says it's done. It's done when a check it cannot argue with says so. I'm calling it the Loop Engine.

Concretely:

The pipeline generates a fast check script for your project (typecheck + lint, under 15 seconds), and a hook runs it after every single file edit. Claude breaks something, the error lands back in its face immediately. Not at the end. Every edit.

Coding agents are banned from touching tests. QA writes failing tests first and owns the test folder. If a coder adds .skip or loosens an assertion to get green, that gets flagged as a critical finding. No more grading their own homework.

"12 test files written" stopped being an acceptable QA report. Suites have to actually run: executed, passing, failing.

Before the final gate, a separate agent boots your real app and drives it like a user. Every button clicked, every form submitted, every link followed. A button that renders but does nothing is a critical bug. This feature exists because I was tired of being my own QA department at 1am.

Loops stop on evidence, not vibes. Each one tracks a number (failing tests, open findings) and stops when it hits zero or stops improving, then shows you the trend, like 7 to 3 to 3, and asks. No retrying the same fix five times.

And when Claude hits something weird with no existing check, it can build its own loop, with one rule: first create a check it can run, like a failing repro script. No check, no loop.

Still free, MIT. Gates, receipts, and worktree isolation for parallel agents are unchanged.

Install:
/plugin marketplace add nagisanzenin/claude-code-plugins
/plugin install production-grade@nagisanzenin

Repo: github.com/nagisanzenin/claude-code-production-grade-plugin
Theory writeup lives in docs/LOOPS.md if you want the reasoning.

Honest caveat: this shipped this week. The edit-hook enforcement is real code, the rest is protocol the agents follow, and I want to see where they drift in the wild. If an agent cheats the rules or a loop converges badly on your project, I want the transcript.

Most teams ship prompts the same way they used to ship CSS in 2008. Tweak, eyeball a few outputs, push to prod, wait for users to complain, repeat. Prompts are production code. They deserve the same testing infrastructure your Python does.
 
That's why I built PromptLabs.
 
How the loop works, in five steps:  

1. You provide the input. Either an intent ("classify customer support emails as billing, technical, account, or other") or an existing production prompt plus the failure modes you've been seeing.
2. EvalGen writes your test suite. It picks 5 to 8 categories of inputs that will exercise the prompt (happy path, edge cases, adversarial), fires one parallel LLM call per category, and dedupes the result. So you get real coverage, not 50 reworded copies of the same easy case. The same call also writes the scoring rubric. Then it splits the test set into train and holdout. The holdout never leaks into optimization.
3. Runner executes the prompt across every target model in parallel. Choosing between Sonnet 4.6, GPT-5, and Gemini 3? All three run at once on the same eval set. Results in minutes, cost per eval plotted on the same chart.
4. Judge scores every output, criterion by criterion. LLM-as-judge with reasoning attached, so you can see exactly why a score is what it is.
5. Optimizer proposes a diff, not a regeneration. It looks at where the prompt failed, then returns specific line edits (insert this clause after line 3, delete this sentence, reword this paragraph). You read it like a pull request. The new version is scored on the holdout set. The loop checks for convergence or overfitting, and either accepts the result or loops back to step 3 with the new prompt.

 
The accepted prompt is served over HTTP. Your production code fetches the latest version at request time, so you can iterate without redeploying.
 
Three things that make this different from tools you've probably tried:
 
The eval set is real, not theater. Stratified by category with parallel generation and dedup, so you get coverage of edge cases instead of fifty rewordings of the happy path. Most tools either skip eval generation entirely, or give you one LLM call that quietly produces 40 near-duplicates.
 
Train and holdout stay separate, and the loop enforces it. The trajectory chart shows the gap widening the moment you start overfitting, and the loop halts itself when it does. The "best version" pick uses a lower confidence bound so a lucky high-variance run can't game the leaderboard. Most "optimizer" tools you've seen don't even have a holdout set.
 
The Optimizer evolves your prompt, it doesn't replace it. A diff is reviewable. You can accept some edits and reject others. The domain knowledge you spent six months baking into your prompt isn't thrown out every iteration. DSPy-style frameworks regenerate; this one refines.
 
If you've been gluing promptfoo + dspy + langfuse together to do what should be one workflow, this is one tool that does the whole thing. If you're treating prompts like config strings instead of like the production code they are, you're leaving accuracy on the table and inviting silent regressions you wont see until they hurt.
 
MIT, local, your keys.

Most teams ship prompts the same way they used to ship CSS in 2008. Tweak, eyeball a few outputs, push to prod, wait for users to complain, repeat. Prompts are production code. They deserve the same testing infrastructure your Python does.
 
That's why I built PromptLabs.
 
How the loop works, in five steps:  

1. You provide the input. Either an intent ("classify customer support emails as billing, technical, account, or other") or an existing production prompt plus the failure modes you've been seeing.
2. EvalGen writes your test suite. It picks 5 to 8 categories of inputs that will exercise the prompt (happy path, edge cases, adversarial), fires one parallel LLM call per category, and dedupes the result. So you get real coverage, not 50 reworded copies of the same easy case. The same call also writes the scoring rubric. Then it splits the test set into train and holdout. The holdout never leaks into optimization.
3. Runner executes the prompt across every target model in parallel. Choosing between Sonnet 4.6, GPT-5, and Gemini 3? All three run at once on the same eval set. Results in minutes, cost per eval plotted on the same chart.
4. Judge scores every output, criterion by criterion. LLM-as-judge with reasoning attached, so you can see exactly why a score is what it is.
5. Optimizer proposes a diff, not a regeneration. It looks at where the prompt failed, then returns specific line edits (insert this clause after line 3, delete this sentence, reword this paragraph). You read it like a pull request. The new version is scored on the holdout set. The loop checks for convergence or overfitting, and either accepts the result or loops back to step 3 with the new prompt.

 
The accepted prompt is served over HTTP. Your production code fetches the latest version at request time, so you can iterate without redeploying.
 
Three things that make this different from tools you've probably tried:
 
The eval set is real, not theater. Stratified by category with parallel generation and dedup, so you get coverage of edge cases instead of fifty rewordings of the happy path. Most tools either skip eval generation entirely, or give you one LLM call that quietly produces 40 near-duplicates.
 
Train and holdout stay separate, and the loop enforces it. The trajectory chart shows the gap widening the moment you start overfitting, and the loop halts itself when it does. The "best version" pick uses a lower confidence bound so a lucky high-variance run can't game the leaderboard. Most "optimizer" tools you've seen don't even have a holdout set.
 
The Optimizer evolves your prompt, it doesn't replace it. A diff is reviewable. You can accept some edits and reject others. The domain knowledge you spent six months baking into your prompt isn't thrown out every iteration. DSPy-style frameworks regenerate; this one refines.
 
If you've been gluing promptfoo + dspy + langfuse together to do what should be one workflow, this is one tool that does the whole thing. If you're treating prompts like config strings instead of like the production code they are, you're leaving accuracy on the table and inviting silent regressions you wont see until they hurt.
 
MIT, local, your keys.
 
https://github.com/temm1e-labs/promptlabs

Most teams ship prompts the same way they used to ship CSS in 2008. Tweak, eyeball a few outputs, push to prod, wait for users to complain, repeat. Prompts are production code. They deserve the same testing infrastructure your Python does.
 
That's why I built PromptLabs.
 
How the loop works, in five steps:  

1. You provide the input. Either an intent ("classify customer support emails as billing, technical, account, or other") or an existing production prompt plus the failure modes you've been seeing.
2. EvalGen writes your test suite. It picks 5 to 8 categories of inputs that will exercise the prompt (happy path, edge cases, adversarial), fires one parallel LLM call per category, and dedupes the result. So you get real coverage, not 50 reworded copies of the same easy case. The same call also writes the scoring rubric. Then it splits the test set into train and holdout. The holdout never leaks into optimization.
3. Runner executes the prompt across every target model in parallel. Choosing between Sonnet 4.6, GPT-5, and Gemini 3? All three run at once on the same eval set. Results in minutes, cost per eval plotted on the same chart.
4. Judge scores every output, criterion by criterion. LLM-as-judge with reasoning attached, so you can see exactly why a score is what it is.
5. Optimizer proposes a diff, not a regeneration. It looks at where the prompt failed, then returns specific line edits (insert this clause after line 3, delete this sentence, reword this paragraph). You read it like a pull request. The new version is scored on the holdout set. The loop checks for convergence or overfitting, and either accepts the result or loops back to step 3 with the new prompt.

 
The accepted prompt is served over HTTP. Your production code fetches the latest version at request time, so you can iterate without redeploying.
 
Three things that make this different from tools you've probably tried:
 
The eval set is real, not theater. Stratified by category with parallel generation and dedup, so you get coverage of edge cases instead of fifty rewordings of the happy path. Most tools either skip eval generation entirely, or give you one LLM call that quietly produces 40 near-duplicates.
 
Train and holdout stay separate, and the loop enforces it. The trajectory chart shows the gap widening the moment you start overfitting, and the loop halts itself when it does. The "best version" pick uses a lower confidence bound so a lucky high-variance run can't game the leaderboard. Most "optimizer" tools you've seen don't even have a holdout set.
 
The Optimizer evolves your prompt, it doesn't replace it. A diff is reviewable. You can accept some edits and reject others. The domain knowledge you spent six months baking into your prompt isn't thrown out every iteration. DSPy-style frameworks regenerate; this one refines.
 
If you've been gluing promptfoo + dspy + langfuse together to do what should be one workflow, this is one tool that does the whole thing. If you're treating prompts like config strings instead of like the production code they are, you're leaving accuracy on the table and inviting silent regressions you wont see until they hurt.
 
MIT, local, your keys.
 
https://github.com/temm1e-labs/promptlabs

Most teams ship prompts the same way they used to ship CSS in 2008. Tweak, eyeball a few outputs, push to prod, wait for users to complain, repeat. Prompts are production code. They deserve the same testing infrastructure your Python does.
 
That's why I built PromptLabs.
 
How the loop works, in five steps:  

1. You provide the input. Either an intent ("classify customer support emails as billing, technical, account, or other") or an existing production prompt plus the failure modes you've been seeing.
2. EvalGen writes your test suite. It picks 5 to 8 categories of inputs that will exercise the prompt (happy path, edge cases, adversarial), fires one parallel LLM call per category, and dedupes the result. So you get real coverage, not 50 reworded copies of the same easy case. The same call also writes the scoring rubric. Then it splits the test set into train and holdout. The holdout never leaks into optimization.
3. Runner executes the prompt across every target model in parallel. Choosing between Sonnet 4.6, GPT-5, and Gemini 3? All three run at once on the same eval set. Results in minutes, cost per eval plotted on the same chart.
4. Judge scores every output, criterion by criterion. LLM-as-judge with reasoning attached, so you can see exactly why a score is what it is.
5. Optimizer proposes a diff, not a regeneration. It looks at where the prompt failed, then returns specific line edits (insert this clause after line 3, delete this sentence, reword this paragraph). You read it like a pull request. The new version is scored on the holdout set. The loop checks for convergence or overfitting, and either accepts the result or loops back to step 3 with the new prompt.

 
The accepted prompt is served over HTTP. Your production code fetches the latest version at request time, so you can iterate without redeploying.
 
Three things that make this different from tools you've probably tried:
 
The eval set is real, not theater. Stratified by category with parallel generation and dedup, so you get coverage of edge cases instead of fifty rewordings of the happy path. Most tools either skip eval generation entirely, or give you one LLM call that quietly produces 40 near-duplicates.
 
Train and holdout stay separate, and the loop enforces it. The trajectory chart shows the gap widening the moment you start overfitting, and the loop halts itself when it does. The "best version" pick uses a lower confidence bound so a lucky high-variance run can't game the leaderboard. Most "optimizer" tools you've seen don't even have a holdout set.
 
The Optimizer evolves your prompt, it doesn't replace it. A diff is reviewable. You can accept some edits and reject others. The domain knowledge you spent six months baking into your prompt isn't thrown out every iteration. DSPy-style frameworks regenerate; this one refines.
 
If you've been gluing promptfoo + dspy + langfuse together to do what should be one workflow, this is one tool that does the whole thing. If you're treating prompts like config strings instead of like the production code they are, you're leaving accuracy on the table and inviting silent regressions you wont see until they hurt.
 
MIT, local, your keys.
 
https://github.com/temm1e-labs/promptlabs