u/VehicleNeither3208

i've been wiring a couple of ai agents into our stack and i'm stuck on how to keep them from doing something dumb once they touch real prod systems.

rn the backend exposes actions: create order, update subscription, send refunds, write crm notes, send emails through our transactional provider. the agents only talk to our backend, which then calls the real systems, so all tool usage goes through one internal api layer. in dev and stage we run everything with a dry-run flag, so we see what the agent would do without side effects. that part feels fine for testing.

prod is where it gets risky. "just add approvals" turns every action into a ticket, which kills most of the value of automated agents. but giving agents free access to anything that moves money or can fan out (refunds, bulk emails, deletes) seems like asking for trouble. a per-day refund limit sounds nice until the thing auto-refunds 200 customers because it misread a policy or a tool response.

stuff i keep hearing about: least privilege per agent and per tool (separate identities, scoped roles/creds), shadow mode on real traffic with no execution, hard rate limits and quotas on risky actions, and a separate review step where another model or a policy engine checks proposed actions against business rules before they go out. put together it feels like we're rebuilding a workflow and access-control engine just to babysit the agent.

for people who already have autonomous agents on real prod systems: what did you do to prevent unintended actions, what runs fully automated vs forced approvals, and what control sounded good on paper but was useless in practice?

Start with the smallest base that runs your workload  distroless or scratch for compiled binaries, minimal OS images for everything else. Multi-stage builds are non-negotiable: build dependencies should never appear in the final image. Every unnecessary layer is a future CVE to triage.

Beyond the build: automate rebuilds on base image updates, scan every image before it hits a registry, and enforce a maximum image age policy so stale images don't persist in production. Lean and secure aren't in tension, they reinforce each other. What's the one thing that's made the biggest difference for your team in keeping images clean?

Start with the smallest base that runs your workload  distroless or scratch for compiled binaries, minimal OS images for everything else. Multi-stage builds are non-negotiable: build dependencies should never appear in the final image. Every unnecessary layer is a future CVE to triage.

Beyond the build: automate rebuilds on base image updates, scan every image before it hits a registry, and enforce a maximum image age policy so stale images don't persist in production. Lean and secure aren't in tension, they reinforce each other. What's the one thing that's made the biggest difference for your team in keeping images clean?