u/jj_at_rootly

We haven't received an RCA for yesterday yet. GitHub mentioned in the resolution note that it had committed to producing one. The only public information regarding the cause is an update at 19:13 UTC in which they stated that they had partially disabled authentication token retries and had noticed an improvement; this corresponds with Copilot auth being the last service to recover, about four hours after the other seven services had been mitigated.

Well worth reading the resolutions for the rest of this month. The account of the Aug 6 Actions incident is particularly specific compared to most public writeups. When a routine deployment was made to an internal Actions service, an existing weakness in terms of capacity and concurrency came to light, the pods were replaced and the remaining capacity was saturated as a result of which the services crashed and a cascade of failures spread across the clusters. There was then a second phase in which a latent bug caused runners to be assigned jobs that were no longer valid and so they ended up retrying those invalid jobs rather than picking up real work. At peak times 71% of the workflow runs failed due to infrastructure errors.

Although the pull request of 24 July is the better piece of writing, a backfill cancellation came across what they refer to as a misundertood Vitess codepath, as a result of which the backing table was dropped and a dangling vschema reference was left behind. That is what blameless should mean, identifying a gap in the tooling rather than blaming a person.

The line that I had not anticipated appeared in the Aug 6 Pages writeup, where it was stated that the incident had not been fully picked up by their availability metrics and that they are altering the way Pages availability is measured so that it will be reflected in the future. Approximately 128,000 deployments failed to be processed during a period which their own SLI had not recorded.

There are seventeen incidents listed on the status page from August 1st to August 18th; each of them will have its own root cause analysis, none of which will account for the seventeen. That's the real analysis needed imo.

Most of the vocabulary in this field is borrowed and most of it is wrong in one of two directions. Some words take on weight they haven't earned. Firefighter borrows danger from people who carry strangers out of burning buildings. War room borrows the discipline of an actual command structure and almost never brings the discipline, just the drama. Postmortem borrows death, which is why the field had to bolt "blameless" onto the front of it, we run retrospectives.

"AI SRE" fails in the opposite direction. It shrinks the job to a single task.

Calling a tool an AI SRE tells the market that being an SRE is mostly root-cause analysis and incident triage. That is a real slice of the work and a small one. SRE is reliability design, SLOs and error budgets, observability, capacity planning, toil reduction, and the human work of making systems and teams more dependable. A tool that correlates alerts and proposes a probable cause is useful. It is not an SRE.

The cost of the name is not in the room with the engineers who know better. It is in the budget review, where a leader reads "an AI SRE that handles on-call" and hears permission to under-invest in the practice. The headcount, the reliability design, the work that does not fit on a product page.

So why do we still use it? Because the search traffic is part of how a company our size gets found at all. We keep the word and we constrain it. The tool augments the people doing the work, it does not replace them.

We need to replace the term with something else... something that describes exactly what the product is doing. RCA, triage and investigation, etc... I know it has caught on at this point but for everyone building root cause analysis internally what are you calling it?

Brent Chapman, who is writing a book on incident management, sent me an email recently that I keep thinking about.

His argument is that when AI writes your retrospective, the document looks fine but the learning is gone. Not because the document is bad. Because the document was never the point. The learning happens in the process of writing it, not reading it.

He breaks it down into three layers. Readers absorb what gets published. Individual writers discover things mid-sentence they didn't know they knew, like starting to write "the deploy caused the outage" and realizing as you trace it that the deploy only surfaced a problem already waiting to happen. And the group of writers learns from reconciling what each of them separately remembered, catching gaps, correcting misremembered moments, surfacing disagreements that turn into the most useful part of the whole review.

When AI writes the document, none of those layers work. Readers get the AI's synthesis with no human pressure-testing behind it. Nobody stops mid-sentence to discover anything. No disagreement surfaces in the comments because there are no comments. You get a polished artifact and an empty experience.

His framing that really landed for me was that you could throw the retrospective document away after writing it and still get most of the value. The real value leaves the room in the heads of the people who were there.

Where he thinks AI legitimately helps is when collating raw material from Slack, surfacing timeline gaps, cross-referencing past incidents. Mechanical support that gives writers a clean starting point. Not substitution for the thinking itself.

It's worth reading in full.

Full disclosure, I'm the co-founder and CEO of Rootly. We build retrospective tooling so I have a direct stake in this question. Brent's argument is one we wrestle with internally and ultimately think he gets right.