Accidentally charged 31 customers twice but every single invoice looked correct
We bill on usage, nightly job sums the day's events and pushes them to stripe as usage records and it's been running fine for over a year.
Last Tuesday morning support got 4 tickets, all about getting double bills. I opened the invoices and they looked... fine? usage numbers are plausible, nothing weird
But it took us 2 days to see it, the aggregation job timed out at 2am, our scheduler retried it and the retry pushed the same usage again. stripe didnt know it's a duplicate, it just saw more usage and billed them twice
every invoice was internally consistent but fundamentally wrong
This fiasco ended with 31 customers, $6,200 in refunds and one enterprise account that now wants a monthly usage export to cross verify now
The fix was boring, an idempotency key on every event at the source and not at sync time and a check that flags any customer whose usage jumps 80%+ vs last period before the invoice finalizes.
the issue is that nothing was alerted, no errors or failed jobs
Does anyone else get burned by retries in billing? How are you de-duping usage?