How do you find the user intents nobody thought to put on the dashboard
We have millions of interactions and a weekly ritual where someone samples traces, adds tags and updates a ticket taxonomy. The known intents were pretty easy to graph but we had no good way of seeing the ones we didn't name yet. Recently, cancellation questions started growing alongside a frustrated sentiment cluster, while overall helpfulness and resolution scores barely even moved
I am trying to get beyond manual tagging without pretending embedding-based clustering is an oracle. Clusters need representative traces, stable labels, and enough context to distinguish a product issue from a wording change. Sampling rate matters too. A rare but expensive failure can disappear in random sampling, while one noisy customer can look like a distribution shift. Then topic drift arrives and last month’s clean taxonomy becomes another maintenance job.
Braintrust Topics is one option we are evaluating because it gives us a way to surface and track these clusters without deciding all the categories upfront. Persisting those labels for SQL queries is a big part of why we're looking at it. That sounds useful, especially if we can turn a growing cluster into an eval slice. I am still skeptical about how much human review is needed to keep labels meaningful.
I basically want an alert when a new intent or failure pattern grows materially, a handful of traces that explain it and a path to test the eventual fix. I don’t want another dashboard that says engagement changed.
How are teams detecting unknown intents and topic drift at production scale and what review loop keeps the clusters from becoming nonsense?