If service reliability conflicts with cost, what’s your strategy?
Before containers and Kubernetes, I deployed services onto virtual machines. We had a fixed number. When the peak arrived at night, whoever was on call tried to spread the load by hand. Buying more hardware took time, and some unreliability was acceptable.
Then AWS and Cloudflare made it easier to throw money at capacity and let some infrastructure scale. You have to design the system with scaling in mind. And a bill.
Now tokens add cost.
If you own both the service and the bill, what’s your strategy? How do you cap, what cap acceptable? How to gracefully degrade? And how rules applies across the organization?