I added one member to my AI team whose only job is to watch me, not the code — and its goal is to go silent
English isn't my first language. I wrote this in Korean and translated it with AI, so please excuse any awkward phrasing. A short Korean summary is at the bottom.
TL;DR: When I started building my own AI tools as a non-developer, the real danger wasn't being wrong — it was not knowing I was wrong. The AI often knew my blind spots but never said anything unless I asked. So I gave my AI team a "mentor" that watches my habits instead of the output, speaks only when it matters, and "graduates" me from each lesson. Success means it eventually has nothing left to say.
The last three posts were heavy on system design. This one is lighter: a small mechanism that I think is the most unusual part of how I use AI.
1. The problem: not knowing what I don't know
I'm not a developer. When I first started building my own skills (reusable AI workflows) with Claude Code, I wasn't good at working with AI yet. What scared me most wasn't making mistakes. It was making mistakes without noticing.
The Johari window puts it neatly:
- Open: I know it, the AI knows it
- Blind spot: I don't know it, but the AI does ← this one
- Hidden: I know it, the AI doesn't
- Unknown: neither of us knows it
Hardcoding an API key. Trusting "Done!" without running the code myself. Reading "deploy succeeded" as "my new code is live." The AI already knows these are risky. But it doesn't say anything unless you ask. It just does what it's told.
I wanted the blind-spot window to open without me having to ask.
2. The mentor note
My setup has several AI roles: planner, developer, auditor, and so on. All of them look at the work. The mentor is different. It looks at me. It doesn't write, edit, or grade any code or documents.
It shows up at the end of a response, like this:
🎓 Mentor note
- Did you actually run it and see the result yourself?
Every rule comes from that original reason:
- Only when relevant. If it shows up every time, it becomes nagging. Nagging gets ignored, and the blind spot closes again.
- 1–3 lines, phrased as questions. A question makes me check for myself; an order doesn't.
- Safety → verification → understanding. The most expensive blind spots come first.
- Graduation. Once I do the right thing on my own three times in a row, it stops mentioning it. That blind spot has moved into the open window.
- Success = silence. It was designed as training wheels from day one.
It's also cheap. It isn't a separate agent, just a few lines added to the main response, so it costs about 50–150 tokens per reply. A separate sub-agent would have cost 2,000–5,000.
3. Real mentor notes I've received
- "A 'deploy succeeded' message is not the same as your new code being live." I checked: the build I was about to ship was 11 days old. Nothing would have changed.
- "Don't just adopt my recommendations on these 6 conflicts. Decide them yourself." The AI telling me not to over-trust the AI.
- "One of your assumptions is factually wrong." It pointed straight at a gap in my own knowledge.
- "This URL is public with no login, so having zero real personal data in the database is your only safeguard."
4. The mirror had blind spots too
Here's the fun part. The mentor broke twice, and both times I noticed before the system did.
Failure 1: the mentor fell asleep. Through an entire design phase, not a single mentor note appeared. When I said "the mentor doesn't seem to be active," the AI admitted it had broken the rule and corrected itself.
Failure 2: the mentor had no memory. Graduation only works if the mentor remembers my habits. After three months I asked, "Where are mentor notes actually recorded?" It turned out the storage had never existed. The design said "shared memory across projects," but the real memory was isolated per project. For three months the mentor had been meeting me for the first time, every single session.
The mirror I built to show me my blind spots had blind spots of its own, and a human had to point them out.
5. Where it stands now
The mentor now has a real file to remember things in. But zero habits have graduated so far. I decided not to graduate anything retroactively. There's no honest way to tell, after the fact, which habits were real and which were luck.
Graduation candidate #1: "Verify it yourself, even when the AI says it's done." That used to be my biggest weakness. Recent records show it's improved a lot.
6. Want to try it?
A few lines in your AI config file (CLAUDE.md, if you use Claude Code) are enough to start:
- Watch me, not the code (verification, security, habits)
- Only when relevant, 1–3 lines, as questions
- If I do it right 3 times in a row on my own, I graduate. Don't repeat it.
- Keep the memory in a real file (check that the storage actually exists!)
That last line took me three months to learn.
Do you ever ask your AI what you might be missing?
🇰🇷 한국어 요약 처음 AI로 스킬을 만들 때 가장 위험했던 건 '모르는 걸 모르는 것'이었습니다. 그래서 코드가 아니라 저를 보는 '멘토' 역할을 만들었습니다. 필요할 때만 1~3줄 질문으로 맹점을 짚고, 같은 습관을 3번 스스로 해내면 '졸업'시킵니다. 목표는 멘토가 할 말이 없어지는 것입니다. 정작 멘토가 잠들고 기억을 잃은 건 제가 먼저 알아챘습니다. 한국어 댓글도 좋습니다!