IMO CX team needs to be part of the eval loop
Our org has been struggling with silo’d departments over the past year and wanted to share something that’s been working well for us.
We’re a small(ish) startup and have some “flagship” AI features that have been live for a while now with around 3 people on our CX team. Since launch there has always been a weird gap between the engineers and the CX team handling tickets. A customer would get a weird response, complain, and the agent would deal with it in a ticket. However, none of that info would make its way back to the people improving the features. I mean, sometimes it would, obviously the teams talked, but there wasn’t any systematic process in place.
This ended up being really frustrating and finally we built a lightweight version of a human review queue. When a support agent runs into a bad AI response, they flag it and slap a quick label on it (wrong info, weird tone, didn't follow instructions, etc.). That flagged + labeled example then feeds straight into our eval dataset (currently using Braintrust for our eval platform). Now the agents' labels turn into actual test cases we run against future changes.
We’ve now got the extra benefits of:
A. The support team genuinely likes it. NGL just kinda assumed they’d be annoyed haha.
B. The dataset got way more realistic.
C. It quietly bridged the technical/non-technical divide.
Still early and the labeling taxonomy needs work (agents disagree on categories more than I'd like). But overall it's turned our support team from the last to know into the first line of quality signal.