r/AIInnovationInsights

Built an open-source MCP-aware proxy: every tools/call, resources/*, prompts/*
goes through policy + budget + audit, and a per-identity anomaly detector can
auto-block a client whose behavior spikes — no rule written, no human in loop.

Catches abrupt deviation, not low-and-slow (baseline adapts to slow ramps —
documented with tests). Three policy backends (YAML/OPA/Cedar), one Go binary.

Repo: https://github.com/kabirnarang39/wardline — feedback on the threat model
wanted.

You write a Claude Skill. It works. Six months later Anthropic ships a new model, and nobody tested your skill against it first because until now, nothing did that automatically. It might stop triggering, ignore instructions it used to follow, or start blowing past a token budget you never noticed. You find out by accident, not by CI.

skillci (https://github.com/kabirnarang39/skillci) is a Go CLI that runs your skill's eval suite across a matrix of Claude models and fails your build only when something that used to pass stops passing. Just shipped v0.4.1, so here's the actual feature set, no fluff:

- Self-growing eval loop. An uncovered failure doesn't just go red it writes the failing case as a permanent test file. Run skillci accept and it's covered forever. Catch once, covered forever.

- Git-native bisect. skillci bisect binary-searches your skill's actual commit history with a real git worktree to find which commit broke a case including correctly handling merge commits (falls back to a linear scan and finds the real culprit, not the merge commit itself).

- LLM-as-judge, done properly. Chain-of-thought reasoning before every verdict, response caching so you're not re-paying for the same grading twice, optional multi-sample self-consistency voting not a bare pass/fail prompt.

- Adversarial red-team assertions. redteam: runs named attack plugins prompt injection, instruction leakage, jailbreak override, harmful content elicitation against your skill. A successful attack becomes a permanent regression test automatically, same as a model regression does.

- Deterministic + LLM-assisted fuzzing. Free mutation testing (synonym swaps, negation, reordering) to check your skill still triggers correctly on reworded prompts, plus optional model-generated realistic paraphrases, cached so you pay for generation once, ever.

- Flake-aware retries. flake_retries reruns and majority-votes instead of trusting one noisy sample because even temp=0 isn't fully deterministic on a real API.

- Cost and latency budgets. Fail CI on runaway token count, latency, or estimated dollar cost — not just wrong output.

- OWASP Agentic Skills Top 10-mapped security lint, zero API calls, runs in milliseconds — catches things like a skill piping curl | sh, an unpinned : latest dependency, or a YAML anchor-bomb in frontmatter, before you ever spend a model call on it.

- Live VS Code diagnostics as you type, and an optional self-hosted dashboard for a per-skill "still passes on this week's model" badge.

No other tool in this space combines all of these most do one slice (a security linter, or a generic eval runner, or an LLM-judge wrapper). Where I'll say competitors are genuinely ahead: promptfoo ships 50+ red-team plugin variants against skillci's 4, with years of production hardening skillci's few-day-old redteam assertions haven't had yet. That's a real gap, not false modesty.

Apache-2.0, go install or Homebrew/Scoop, GitHub Action included. If you're maintaining Claude Skills and this solves a problem you actually have, I'd love the feedback — and if you can find a way to break it, even better.

github.com/kabirnarang39/skillci