I let an LLM propose K8s remedation actions,but not execute them - built a deterministic safety veto instead
Been dealing with the classic on-call problem: alerts fire, root cause is often something simple (pod stuck, resource starved, node cordoned wrong), but by the time a human looks at it, 10-15 minutes have passed.
Obvious next step is "let an agent fix it automatically." Obvious next problem: LLMs are not deterministic, and I did not want an agent hallucinating a kubectl drain on the wrong node at 3am.
So I split the system into two halves:
Agent layer -- a FinOps Agent and an Availability Guardian that watch cluster state and propose actions: scale, rollback, cordon, drain.
Safety Engine -- a fully deterministic layer, no LLM involved, that validates every proposed action against live Kubernetes API state before anything executes. If the action doesn't pass hard-coded safety checks, it's vetoed regardless of how confident the agent is.
The design bet is: use the LLM for judgment/reasoning about what might be wrong, never trust it for the final go/no-go on an action that touches prod.
It's built as an MCP server (TypeScript/Node), came out of a hackathon, and I'm now working through what it'd take to make it production-grade multi-tenancy, a configurable policy engine instead of hardcoded safety rules, pluggable LLM backends, Prometheus/Grafana/ArgoCD integrations.
Repo: github.com/viswa0028/agentBuilders-helmsman
Genuinely curious how people running real prod clusters would think about the safety boundary here where would you not trust this, and what would you want the policy engine to check before you'd let it touch your cluster?