r/sre

We are trying to trace one agent run across an API gateway, queue, vector lookup, model call and tool worker. Each service reports healthy, yet one request can still fail because parent child context disappears at the queue, sampling drops the useful span, or a malformed tool argument triggers a retry. Basic logs have hit their ceiling. Five healthy services can still create one broken request.

We're testing Braintrust for end-to-end span trees, model and tool metadata, trace search and trace level scoring. We also looked at Langfuse since it's openn source and self hostable but context propagation across async boundaries and trace search at our volume were concerns. I need context propagation that survives async handoffs, span links for retries, queue latency and a sampling policy that keeps error paths without exploding cost. How does this compare with established observability stacks in real agent systems?

Setting up newrelic as single source of monitoring
- need the coverage from development cycle to deployment/ onprem to android apps.
-

Hope I’ll get smth useful out of this post
I’m running a mid size enterprise IT ops team for a few years now and im kinda losing it with ai tools that are just fancy chatbots taped onto old consoles. we have agents on thousands of endpoints but half of them fall over on upgrade, remote access is laggy, and the automation still needs a human to click run on every script.
Also tool fragmentation is killing us, poor integrations between monitoring and ticketing and the ai operations story always dies when we try real autonomy eg patching, routine fixes, proactive monitoring before stuff burns. Techs are exhausted and tickets pile up, they act as they need a robin tool or smth similar then reporting visibility is trash and to also mention costs keep creeping us out with no clear roi. Would love any tips from folks who actually got ai to reduce IT workload instead of just adding another dashboard.

We haven't received an RCA for yesterday yet. GitHub mentioned in the resolution note that it had committed to producing one. The only public information regarding the cause is an update at 19:13 UTC in which they stated that they had partially disabled authentication token retries and had noticed an improvement; this corresponds with Copilot auth being the last service to recover, about four hours after the other seven services had been mitigated.

Well worth reading the resolutions for the rest of this month. The account of the Aug 6 Actions incident is particularly specific compared to most public writeups. When a routine deployment was made to an internal Actions service, an existing weakness in terms of capacity and concurrency came to light, the pods were replaced and the remaining capacity was saturated as a result of which the services crashed and a cascade of failures spread across the clusters. There was then a second phase in which a latent bug caused runners to be assigned jobs that were no longer valid and so they ended up retrying those invalid jobs rather than picking up real work. At peak times 71% of the workflow runs failed due to infrastructure errors.

Although the pull request of 24 July is the better piece of writing, a backfill cancellation came across what they refer to as a misundertood Vitess codepath, as a result of which the backing table was dropped and a dangling vschema reference was left behind. That is what blameless should mean, identifying a gap in the tooling rather than blaming a person.

The line that I had not anticipated appeared in the Aug 6 Pages writeup, where it was stated that the incident had not been fully picked up by their availability metrics and that they are altering the way Pages availability is measured so that it will be reflected in the future. Approximately 128,000 deployments failed to be processed during a period which their own SLI had not recorded.

There are seventeen incidents listed on the status page from August 1st to August 18th; each of them will have its own root cause analysis, none of which will account for the seventeen. That's the real analysis needed imo.

im Trying to weigh the tradeoffs here... i feel like Distroless and minimal images cut out the shell and package manager, which is great for reducing attack surface and CVE counts. butThe problem is the second something goes wrong in prod, I can't just exec in and poke around like I would with a normal Debian based image. That friction has made me hesitant to roll these out more broadly even though the security case is solid.

so im posting to have an idea ..that For people running these day to day, what's your actual debugging workflow? Are you using ephemeral debug containers, sidecar attach, or just shipping better logging up front so you don't need to shell in at all?

Trying to figure out if the security win is worth the operational friction or if I'm approaching debugging wrong entirely.

Hi everyone, is there a way that one could monitor the service status of 3rd party vendors like AWS. We get paged whenever there is service disruption however status page monitor only tells uptime of the endpoint. But how do we measure their SLA to properly calculate their credit. Is there any tool available out there?

I used to think sampling was fine who needs every single request, right? then we had an incident where a 1% edge case caused a cascade of failures. Our sampling dropped that 1% we spent 6 hours debugging something that should have been obvious if we had the full picture. Here's what i've learned: the 1% you drop is often the 1% that breaks things. complete data makes a real difference when you are trying to understand what your code actually does under real traffic. How others are handling this tradeoff between data volume and visibility. Is everyone just accepting the risk or have you found better approaches?

Hey folks,
I wonder if you know platforms (not upwork) where I can find short term projects to work on and get paid.
I’m on a parental leave since a few months now and I need to work a bit (4 hours a day maximum).
I’m an SRE with +6 years of experience, I worked on high availability platform, CI/CD, observability, etc.

I'm trying to model an agent that gets triggered whenever a production incident occurs. I'm new to the SRE side of things and I'm looking for inputs on a few things based on your past experiences.

Whenever an incident occurs, what are some of the most common reasons? For example, it could be a false alarm, it could be an issue due to a new upgrade, network issues or a crash due to some bug in one of the services etc. Out of 100 incidents, what are the chances that it could be a false alarm, upgrade issue, bug in code etc.

I want to know what are all the common issues or reasons that I can add as part of a v1 for my agent? How do I think through this as I've never worked on such a thing till now?

im Looking at this from the transport side rather than the security side.

basically....AI traffic has a different profile than the web traffic our current design was built around:..long-lived connections, streamed responses, bursty token throughput, and a small number of destination endpoints concentrated in a handful of US and EU regions. Add inline inspection on top and you are doing decrypt, inspect, and re-encrypt on sessions that stay open for minutes.

What that surfaces: PoP density matters less than PoP placement relative to model endpoints; inspection adds compute at the edge, which not every PoP footprint is provisioned for; our sites in South America and Southeast Asia are already the outliers on RTT, so adding an inspection hop there is the thing I am worried about; and peering to the major cloud and AI providers varies a lot by vendor and nobody publishes it clearly.

Questions for anyone who has measured rather than trusted the marketing map: what is your real added latency for an inspected AI session from a secondary-market site; does the vendor do inspection at every PoP or only at a subset of "full-stack" locations, since several are quietly the latter; and has anyone hit throughput ceilings on inspection during peak?

Hi Observability Experts,

I’m working with a client that operates a large-scale estate across AWS, Azure, and on-prem environments. They have been using Datadog for several years, but accumulated technical debt has left their APM implementation fragmented.

Current situation:

• APM libraries are outdated across multiple services.
• Trace context propagation and instrumentation are inconsistent.
• End-to-end distributed tracing is incomplete or broken for many critical applications.
• Datadog Support has reviewed the setup and confirmed that the proper fix is to update instrumentation and agents at the application/source level.

The challenge is that obtaining application team bandwidth and stakeholder approval across a large number of services will take considerable time.

In parallel, we are planning a centralized observability pipeline layer to standardize telemetry collection and governance, but that initiative is still a few months away from broader adoption.

My question to the community:

Is there a viable temporary/interim solution—such as an intermediate proxy, collector layer, or transformation pipeline—that can intercept, patch, or enrich broken spans/traces in-flight before they reach Datadog’s backend?

Has anyone successfully addressed legacy APM gaps at the pipeline level without waiting for app redeployments?

[supprimé]

Hi all!

I am curious what solutions others adopted for observability at scale. We’re having a few EKS clusters with ~8,000 nodes. We are currently using sharded Prometheus with Thanos in each cluster and a common Cortex setup to store all the metrics. We use Splunk for logs and Newrelic for another set of metrics.

I am looking to a common centralized solution for all these items plus traces and data coming from systems like our CI/CD.

I would not want to go with a solution like Datadog that is going to cost a lot.

Does anyone have something built and managed in house? How many people in the team to only do this? What were your drivers to implement it instead of buying?

Thank you,
-JS

TL;DR: Where do you guys learn about the actual SRE practices of other companies? Looking for the really interesting stuff, not generic SRE advice.

I’ve always wondered how people know what companies like Netflix, Google, Meta etc. are doing internally.

Someone might say, “What we’re doing is pretty good, but Netflix does this 5x better.”

And I’m like, how do you even know what Netflix is doing? 😂

I want to find those “wait, companies actually do this?” kind of things.

How they handle incidents, test failures, observability, chaos, massive traffic, internal tooling, etc.

Where do you guys find this stuff? Tech talks, blogs, conference talks, postmortems, YouTube, GitHub, specific people to follow?

Basically looking for a goldmine of this kind of information.

I've been working as an SRE for coming up to 4 years now. I started my tech career in support, then DevOps, and now SRE. My official job title is SRE but in practice the lines at my company are a bit blurred between SRE, cloud engineering, DevSecOps, cloud security and so on, as my company doesn't have a mature SRE practice and the SRE team have to wear multiple hats due to being spread thin (I appreciate this is somewhat common within the industry).

I've been offered an SRE role at a much larger company with a more established SRE culture. They have an embedded model and I'd be working directly with one of the product teams, up to and including helping write application code to help optimise availability and reliability (although this wouldn't be on customer-facing services for the time being).

I'm really excited by the offer but part of me is apprehensive. I'm comfortable working with all the traditional SRE/DevOps-y things (cloud, IaC, alerting and monitoring, incident response, automation and so on), but my current role is somewhat silo-ed, as although we do interact with the product team it's not in a capacity that could really be described as embedded. I'm comfortable with Go and Python for scripting and custom tooling but I understand application engineering is a different kettle of fish, and I haven't had to wear that "hat" in my career before.

I am also a bit apprehensive about how well my SLI and SLO knowledge and experience will translate, as at my current company availability is by far and away the one we track and optimise for the most, to the neglect of the others (n.b. I recognise I'm at fault here for not pushing my company's SRE practice to think harder about these things); I was honest about this, although did my best to put a positive spin on it during the interviewing process (I was also very candid with them about never having been embedded in a team before).

I'm at a bit of a crossroads. I'm fairly content, although stagnant, at my current company, as I feel like I have "maxed out" there for the time being in terms of learning and experience; and as we all know, this industry tends to punish you if you stay in your comfort zone for too long. I've been eyeing a way out into something better for some time now ,and am genuinely excited by this offer, but I wonder if I'm really a good fit for the role and am trying really hard not to let the impostor syndrome creep in.

For those who've moved from a siloed SRE role into an embedded one, how steep was the adjustment (particularly if you found yourself writing application code coming from an ops background)? How did you navigate it and are you happy you took the leap in hindsight? If it felt awkward at first, how long did it take for that feeling to go away? Thanks.

(P.S. I have a fairly long notice period and it's pretty quiet in my company at the moment, so I'd have a bit of time to upskill/brush up on patches in my knowledge if I decided to take it.)

Feature flags are great for decoupling deploy from the release yeah yet they open a monitoring blind spot that doesn't get enough attention. No entry in release log neither anything in the deploy timeline just a config change, so when a flagged path starts behaving unexpectedly for thye cohort thats on it the top level signals can all look fine.

The usual shape of it is like-> a flagged branch adds an extra lookup or takes a slightly different code thats only slow for a subset of accounts like for instance the ones with a lot of historical data. Averaged across all traffic its invisible. service P95 flat and error rate flat + boring dashboards. Slice by flag=true and its ugly so the service is healthy and the cohort is downgrading correspondingly

Flag tools have gotten better at the aggregate version of this. Launchdarkfly has flag scoped monitoring and guarded rollouts that watch error and latency per variation + you can emit flag evaluation span events thru otel and correlate flag state with your traces. That catches variation B is worse pretty well

Still where it still seems to fail sometimes is the code level. Aggregate flag metrics tell you the cohort on the flag is worse but now which function or condition actually changed, specially when nothing errors and latency barely moves so thats the part I wanna know how the rest of you close.

Like there are various approaches to this like some wrap custom otel spans around the flagged branches , some wire business metrics in per cohort and there are also stuff like hud aimed at the function level per cohort view too. Sentry helps on the error and release side but wont surface a silent behavior change that never actually throws

So for ppl running flags at real scale how are you guys monitoring runtime behavior per flag cohort?
Eager to deep dive on this

I'm specifically interested in agents that can do things, not just generate answers.

If you have an agent that can technically execute some action — modify a database, issue a refund, deploy code, change infrastructure, update a CRM, send something externally, etc. — but you still require a human to approve or perform it, what's stopping you from giving the agent autonomy?

I'm especially curious about cases where the model itself is capable enough, but the surrounding system isn't trustworthy enough.

Was there a particular failure you were worried about or actually experienced? And what would you need to be able to verify/guarantee before you'd remove the human approval?

Not selling anything. I'm trying to understand where the boundary between “agent can do this” and “we trust an agent to do this” actually sits in production systems.

Thanks!!

For those working in Cloud SRE or Platform Engineering, what does a normal day look like for you?
What skills do you wish you had focused more on before getting into the role, beyond the usual soft skills, networking and Linux?

1) incident.io shipped Investigations and named the platform underneath it Nexus. Nexus is free on every plan. Investigations itself is gated to Pro and Enterprise, sold as an add-on to their Response product. The launch post has the most candid competitor admission I've seen this year: they had a version of this 18 months ago, called it "AI SRE," shipped it to design partners, and it was "confidently wrong" often enough that they pulled it back and rebuilt. That's the whole reason "AI SRE" quietly became "Investigations" on their site.

2) Cleric put real prices on the table for the first time. $1 per credit, an investigation costs 10 credits, plans start at 100 credits a month. First hard public number in this category: roughly $10 per investigation as a starting point to compare against. Same update added a second billable unit, change verification, where Cleric follows a deploy into prod for up to 14 days and flags regressions while the change context is still fresh.

3) Datadog beat on Q2 earnings, revenue up 36% to $1.12B, big customers up 23%, and the stock dropped anyway, reportedly on a usage cut from their biggest AI customer. Consumption-based observability revenue just got punished for being concentrated. Worth remembering next time someone pitches usage-based pricing as the safe default.

4. Dynatrance is in this category properly now: an Autonomous SRE Agent plus a no-code agent builder, coordinating remediation across AWS, Azure and GCP. An incumbent with the telemetry and the enterprise contracts already in hand is a bigger deal here than another funded startup launching.

5) Resolve AI published head to head comparison pages naming incident.io and PagerDuty's SRE agent directly, the same week incident.io launched Investigations. They also shipped a benchmark arguing Sonnet at medium effort gets close to Opus on incident investigation for a fraction of the cost, which lines up with a separate Anyshift post finding smaller models plus deterministic execution erased most of the accuracy gap between model tiers on bounded infra questions.

6) Traversal quietly deleted a sentence naming American Express, Capital One, PepsiCo, DigitalOcean and Kraken as Fortune 100 customers, and pulled the PepsiCo case study video. Second time they've retracted named customer proof. Their newer customer story goes with "a leading global crypto exchange" instead of naming Kraken outright, which tracks.

7. Rough week for autonomy claims generally. OpenAI disclosed agents coordinating across sandboxes to reach Hugging Face during testing, the UK AI Security Institute reported agents creating fake identities to get around access rules, and OpenAI paused a system after it started finding zero days on its own. Every enterprise buyer read some version of this, probably why incident.io leading with an honest failure story landed the way it did.

8. AWS had its fourth reliability incident in four months, another us-west-2 failure on close to the same network path as the July 24 outage. Same-path repeat failures are exactly what a change-aware agent should flag, and what a human paged at 3am usually doesn't connect.

Smaller stuff: PagerDuty's sitemap has an unlinked "AI Startups Trial" page sitting there, Rootly quietly dropped the explicit gpt-3.5-turbo mention from its privacy policy for a generic subprocessors list, and Neubird published 22 new glossary pages in one shot plus a "top 25 autonomous ops platforms" comparison.

Sources below.