Do you use agents to write the LLM prompts in your code? What’s your setup?
I’m building a coding-agent workflow that designs, writes and tests LLM calls (classifiers, extractors, tool-using agents) in a real codebase.
I’m curious how you write and test the system prompts inside those calls, the ones that live in the code, not the ones you type into a chat.
My setup
- A spec records each LLM call’s input size and call volume, and I pick the model size/cost tier from that.
- One agent writes the system prompt from a schema, a small per-model profile (which structured-output features the target model supports) and the spec.
- A second agent checks the prompt against a fixed rule list (model, sampling, untrusted input). It catches real problems.
The problem: it loops, coining it loop hell. In one run I accepted a fix, the writer reworded the prompt, the checker found a new mismatch, and that went four rounds before I stopped it. Loop hell. The same rule was written in several places (schema, prompt, spec), so fixing one made the others disagree, and a model change means redoing the prompt too.
What I’d like to know
- Do you use an agent or a skill to write the prompts in your code, or do you write them yourself?
- If you use a checker, is it a second agent, a rule list, or evals?
- What ends the loop for you? What tells you a prompt is done, and who decides?
- How do you measure whether an agent-written prompt works: golden test cases, an LLM judge, production metrics, user feedback?
- Where does each rule live (schema, prompt, spec, tests), and how do you stop them drifting apart?
If you use a named skill or plugin for this, I’d like to hear which one and how it’s going. Thx