How do you decide what to investigate next during a production latency incident
Hey hi everyone,
I am just trying to understand how engineers/SREs who dealt with real production latency incidents investigate it
Lets say you have the following
- Logs
- Recent deployment information
- Application health
- Database metrics
- External dependency health/metrics
- Infrastructure metrics
You just encountered the incident, you dont know the root cause. You are uncertain about the truth.
From here how do real engineers go about reasoning to find the root cause
- Do you follow a standard sequence of investigative steps
- How do you determine what investigative step to take next under uncertainty to narrow down the possibilities for the root cause
- Have u ever encountered with incident where initial information was misleading, how did you navigate from there
- Is there any situation where you have lot of information but struggled to form a proper hypothesis
- Have you tried any AI investigative tools that help you in achieving this
I just wanted to understand how do real engineers reason through the uncertainty to find the root cause. What are the biggest pain points