AI wrong outputs with business impact
Hello Reddit,
I'm looking into how organizations handle incidents where an AI system acted, or told a customer something, and the wrong output had direct business consequences: money, contractual or legal commitments, regulated decisions, customer-facing promises. I got interested after the Air Canada chatbot ruling, where the company had to pay the customer 812$ in damages and tribunal fees.
Do you have AI in production today in a process like that, where a wrong output could cost real money or create a liability? If so, is there an incident process specific to it, or is it handled as a regular IT/security incident?
I'm especially interested in lessons learned: what turned out to be different from a normal incident, and what you wish had been in place beforehand. If you'd rather not share on the list, feel free to message me directly and I won't attribute anything. Happy to share a summary of what I hear.