How are you all actually evaluating LLM/agent systems in prod? LLM-as-judge feels shaky
​
So i run evals for a multi-agent system at work and right now my main approach is LLM-as-a-judge against a gold set, plus some semantic similarity scoring. And honestly... it works until it doesn't.
The judge is inconsistent. Same output, slightly different prompt phrasing, different verdict. It's biased toward longer answers, it rationalizes things the gold set clearly says are wrong, and calibrating it feels like im just stacking prompt rules on top of prompt rules hoping the false positives go down. Which they do, partially, but I don't fully trust the number at the end.
What I'm trying to figure out:
- do you treat LLM-as-judge as a real signal or just a smoke test before human review
- how do you handle judge drift when you swap the underlying model
- for agent systems specifically, are you scoring final output or the whole trajectory? feels like scoring just the end misses a lot
- anyone actually getting value out of semantic similarity or is it mostly noise
Not looking for a vendor pitch, genuinely want to know what's working for people running this stuff day to day. Feels like everyone has a different homegrown setup and nobody's sure theirs is good.