Gave a talk on AI observability in prod — the demo-vs-production gap is bigger than most teams admit
I build and ship AI products for a living, and I just gave a talk on the thing nobody puts in the demo: what happens to your LLM app after it's live and real users start doing weird stuff to it.
The pattern I keep seeing: a feature demos perfectly, ships, and then quietly degrades for weeks before anyone notices — because there's no instrumentation to catch it. The model didn't "break," it just started doing something slightly wrong some percentage of the time and nobody was watching that slice.
The three things I argued you actually need:
Evals you run continuously, not once. Most teams treat evals like a pre-launch checkbox. The useful version is a regression suite that runs against real traffic samples so you catch drift before users report it.
LLM-as-judge, but with a sanity check. It scales review way past what a human team can do, but it's not free — you have to validate the judge against human labels periodically, or you're just trusting one black box to grade another.
A real failure-case library. Every prod incident becomes a permanent test case. This is the boring part that actually compounds.
Curious how others here handle this — specifically: do you trust LLM-as-judge in your pipeline, or have you been burned by it? My stack leans on Langfuse for tracing, Portkey as the gateway, and Sentry for the app layer, but I'm always looking for what's working for people.