Would an AI that investigates production incidents before an engineer gets involved actually be useful?
I've been thinking about a problem I keep seeing in infra/DevOps work.
A production alert fires and someone has to manually jump between logs, Grafana/Datadog, Kubernetes, deployment history, GitHub, cloud metrics, etc. just to figure out what actually happened.
I'm wondering if a useful chunk of that investigation could be automated.
The idea would be something that, when an alert or deployment failure happens, automatically:
- investigates logs, metrics and traces
- checks recent deployments/commits
- correlates changes with when the issue started
- checks Kubernetes/cloud infrastructure
- identifies the likely root cause and supporting evidence
- suggests a remediation
- if a code/config change is required, generates a PR
- waits for an engineer to review/approve it
- after deployment, verifies whether the issue was actually resolved
For example:
Instead of an engineer starting from scratch, the system might come back with:
I'm not really thinking about fully autonomous production changes initially. More like an AI first-line SRE that does the investigation and prepares the fix, while engineers stay in control.
I'm trying to understand whether this solves a meaningful enough problem or whether existing observability/SRE tooling already gets teams most of the way there.
For people running production systems:
- How much engineering time do you actually spend investigating alerts/deployment failures?
- What's the most repetitive part of incident investigation today?
- Would you trust a system like this to read your infra/logs if it couldn't make changes without approval?
- Would automatic root-cause analysis be valuable by itself, or would it need to generate the actual fix/PR?
- What integrations would be essential? Datadog? Grafana? Sentry? Kubernetes? AWS? GitHub?
- What would make you immediately say "there's no way I'm giving this access to production"?
I'd especially like to hear from founders, SREs, DevOps/platform engineers and engineers at smaller teams where dedicated SRE coverage is limited.
Not selling anything here I'm trying to understand where the actual pain is before building too much.