u/SorryAttorney7

▲ 4 r/CIO

The OpenAI outage in March cost us four hours of claims processing and no one owned the failover

We run claims intake through an agent that reads documents and routes them. It has been live since January and it handles about 3000 files a week.

In March the provider went down mid morning. Our agent did not fail over because there was no fallback configured and nobody had decided who owned that. The team assumed the platform group had it and the platform group assumed the vendor did. Four hours of manual intake and two people working late to clear the backlog. The postmortem found we had budgeted for a second provider in the original business case and quietly dropped it during procurement.

What we changed is boring. There is a named owner for provider redundancy and a quarterly test where we cut the primary and watch what happens. The first test failed too. How are you handling provider outages when the workflow is customer facing?

reddit.com
u/SorryAttorney7 — 5 days ago