I built an LLM eval gate that can't silently pass
https://github.com/albertofettucini/faithgate
Most LLM eval setups I've seen have a failure mode ops people will recognize: the happy path is green, and every unhappy path is also green. Judge API dies, no scores, nothing to compare, pass. That's an availability metric wearing a quality-gate costume.
I built faithgate around the opposite default. It's a faithfulness regression gate (suite of cases, score per prompt/model version, diff vs baseline, nonzero exit on regression) where every ambiguous state fails closed. Zero matched cases: fail. Unscored run: fail. Every score an abstention: fail. Abstentions are a distinct state in storage, never coerced to 0.0, and there's a --max-abstained policy flag for when you actually want tolerance.
Reproducibility bits: every run writes a manifest with judge id, model, kind, ragas and runner versions, and the suite hash. If the judge changed between baseline and head, comparing the scores is meaningless, so the gate exits 3 unless you explicitly pass --allow-judge-change. A corrupted manifest also fails closed. Duplicate case keys resolve pessimistically (baseline keeps max, head keeps min) so dupes can't quietly lower the bar.
My favorite part lives in CI. Next to the normal green gate there's a proves-detection job that runs the gate against a deliberately regressed suite and inverts the exit code. If the gate ever loses the ability to catch a known-bad change, dependency bump, refactor, whatever, the pipeline itself goes red. Tests for the test.
Judge honesty: default is Claude via your own key (RAGAS underneath). The keyless offline mode is published as untrustworthy, 68% balanced on a 40-example hand-labeled set, catches 9/20 unfaithful, with a unit test asserting the weakness.
Storage is one SQLite file with WAL, no server. Python 3.9 to 3.13, MIT. Known limitation: case identity is content-based, rewording a question mints a new case.
