Building a proxy that blocks what an AI agent does, not what it says — sanity check?
The thing that scares me about agents isn't the model saying something dumb, it's it doing something. One hallucinated rm -rf and there's no undo.
So I'm building a proxy that sits between your app and the LLM (anything OpenAI-compatible). Change one line, your base_url, and every tool call has to pass a policy first. You write rules in YAML deny rm -rf, force dry_run on deploys. When it blocks a call it tells the model why, and the model usually rethinks instead of erroring.
Core works, and I've got an eval running in CI to keep myself honest:
attack catch rate: 40/44 (90.9%)
false positives: 1/20 (5.0%)
The 4 misses (base64'd secrets, non-English injection) are left in the tests on purpose so the number doesn't lie.
It's early and MIT. Demo runs with no API key: github.com/MuhammadFarazAftab/toolwarden
Two things I actually want to know: is the YAML rule format a pain to write, and what attacks am I obviously not thinking about?