Gave a talk on AI observability in prod — the demo-vs-production gap is bigger than most teams admit. Sharing what actually mattere
I build and ship AI products for a living, and I just gave a talk in the DR about the thing nobody puts in the demo: what happens to your LLM app after it's live and real users start doing weird stuff to it.
The pattern I keep seeing: a feature demos perfectly, ships, and then quietly degrades for weeks before anyone notices — because there's no instrumentation to catch it. The model didn't "break," it just started doing something slightly wrong 8% of the time and nobody was watching that 8%.
The three things I argued you actually need:
Evals you run continuously, not once. Most teams treat evals like a pre-launch checkbox. The useful version is a regression suite that runs against real traffic samples so you catch drift before users report it.
LLM-as-judge, but with a sanity check. It scales review way past what a human team can do, but it's not free — you have to validate the judge against human labels periodically or you're just trusting a black box to grade another black box.
A real failure-case library. Every prod incident becomes a permanent test case. This is the boring part that actually compounds.
Curious how others here handle this — specifically: do you trust LLM-as-judge in your pipeline, or have you been burned by it? And what's your actual tooling stack? (I lean on [your tools — e.g. Langfuse/Portkey/Sentry], but I'm always looking for what's working for people.)