How to structure scoring live traffic
We've had offline evals as part of our CI for a while now, but last month we got hit with something that none of our CI runs flagged. Our production inputs had drifted and users were asking things our test set just didn't cover, and output quality on that section of things had degraded for weeks. So unfortunately, our existing evals gave me a false sense of safety because they can only ever test what I thought to put in them.
So now I'm trying to figure out actually sampling and scoring real production responses, not just CI runs against a fixed dataset.
Main things I'm rubber ducking:
Sampling. Are people scoring all live traffic or some percentage?
Alerting. I want to know when quality drops on live traffic, but I’m not trying creat another annoying alert channel. And then if/when it goes off who/what owns the response?