u/Plus-Lawflbaness1576

I made this mistake recently. An agent generated a fix for a bug in our checkout flow, the diff looked reasonable, tests passed, so it shipped with a normal PR review and nothing more.

I assumed passing tests meant the function behaved the same as before under real traffic. It didn't. It handled a currency rounding edge case differently, and it took almost two days of scattered complaints before anyone connected it back to that deploy.

If I did it again, I'd want something checking the function's actual runtime behavior before treating a green test suite as enough. What's the mistake that took you longest to recognize as a pattern?

I used to log everything, every request, every response body, full stack context on every error, and it made debugging faster but the storage bill got embarrassing within a month. Then I over-corrected and sampled aggressively, and now half my incidents involve me wishing I'd captured the one request that actually mattered.

The pattern I've landed on is logging light by default and only capturing full forensic detail, request params, DB queries, execution path, when something actually looks anomalous. It's a reasonable middle ground but it assumes your anomaly detection is good enough to trigger at the right moment, which isn't always true. How are other people balancing storage cost against actually having the context you need when something breaks?

Ten plus years in and the hardest bugs I hit are never the ones that crash loudly. It's the request that touches four services, degrades quietly, and by the time someone notices the SLO dashboard, the actual trigger happened three hops upstream and twenty minutes earlier. Distributed tracing helps you see the path, but it rarely explains why a specific function made the decision it made under that specific load.

Lately my team has been leaning harder on capturing actual execution context at the function boundary rather than just span timing, since that's usually where the real story is hiding. Curious how others are structuring this. Are you mostly still doing log correlation by hand across services, or has anyone found a workflow that actually shortens the investigation instead of just making the evidence prettier?