u/Elegant_Quantity_583

Shipping an agent that diagnoses why a CI run failed instead of leaving that to the human. Flow: hypothesize possible causes → search for evidence with tools → update probability per hypothesis → act based on confidence (recommend a specific fix if very confident, keep digging if medium confidence and it's worth it, hand off to a human if getting it wrong is expensive).

Hypotheses tracked right now (can overlap):

1. H_flaky — flaky test/system
2. H_fault_revealing — real bug
3. H_dependency_fault
4. H_environment_fault
5. H_config_error
6. H_shared_root_cause — one cause behind multiple failures

If you've built anything in this diagnostic-agent space: what's missing from this list, and does anything on it seem like it wouldn't actually be actionable once the agent is confident about it?

I'm building an agent (on LangGraph) that diagnoses failing CI runs instead of just re-running the pipeline and hoping. The loop is: hypothesize → pull evidence via tools → update belief state → act.

I'm representing the "why did this fail" question as a set of non-mutually-exclusive hidden states, each with a probability that gets updated as the agent gathers evidence (log parsing, git diff, past run history, etc.):

• H_flaky — test/system behaves nondeterministically
• H_fault_revealing — failure is a real bug/regression
• H_dependency_fault — a dependency broke something
• H_environment_fault — execution environment is the cause
• H_config_error — CI/build/runtime config is wrong
• H_shared_root_cause — multiple failures trace to one underlying cause

At the end, the agent acts based on confidence: very high → propose a specific fix, medium → gather more evidence if the value of information justifies it, low-confidence-but-high-cost-of-being-wrong → escalate to a human.

Since a lot of you have built LangGraph agents with real state schemas for multi-step diagnosis/investigation flows: what am I missing in this hidden-state list? Anything here not actually actionable once inferred

Hi, if a CI run fails, there can be multiple explanations, so over the past few days I've been researching what the agent needs to have that helps me diagnose the failing Continuous Integration run.

The agent first makes assumptions -----> searches for evidence -----> updates probability of each assumption ------->

At the end, the agent takes an action like

if very high confidence, then:
Ask the user to change the exact thing or to something specific.
If medium, then:
hold, ask for more search evidence if the value of information is greater,
If the low cost of being wrong is too high, then:
simply escalate to human

I have mapped out some hidden states. Here 'they are; they are not mutually exclusive, as a failing CI can be because of many reasons.

1. H_flaky → Basically, the test/system itself can behave nondeterministically
2. H_fault_revealing → The failure is actually revealing a real bug/regression
3. H_dependency_fault → something is wrong with a dependency
4. H_environment_fault → something in the execution environment is causing the failure
5. H_config_error → some CI/build/runtime configuration is wrong
6. H_shared_root_cause → Multiple failures may actually be coming from the same underlying cause

Each hypothesis has a probability that gets updated with evidence.

Can you spot any weaknesses in here ?
What hidden states did I not include?
Are these hidden states actionable?

I'd your honest opinion..

 I'm trying to build my own architecture, so, the agent I'm building sets its beliefs. As it sees a ci fail and searches for the evidence that it's allowed to use that clears out a lot of beliefs,this is how it reduces its uncertainty. Not just an API call where I dump every piece of information.

Let's think of it like you're tying to train someone to do your work

Somewhat similar to how humans work.

But I'm not an expert in this field. Probably I learned about Git just a few months back

So, I'm wondering what those initial beliefs should be. And what is the info that should be accessible to the agent if it asks? Excluding credentials, API keys,

And at last it needs to give a diagnosis, so at what point do you feel like I very sure that this is a problem? Let's fix this.

And I think we're comparing the benefit of evidence vs. the benefit of acting now. And I want to ask you, is that benefit only about time, or are there other costs too?

And if we give a wrong diagnosis and the user acts on it, then what do you think we measure cost in?

And I think always, there should be one belief that asks, "Is this test flaky?" I think we should check that first. But how do you guys check that in practice? Can you please give me a list of ways you do it?

So in summary, I'm asking ,
What information should the agent be able to ascess
What should be beliefs,
And when should it stop?
What are the cost's involved with acting and finding evidence? If you have any concrete number i'd love that
What should it search over or do to catch flaky.

Hi, I'm trying to build an agent for CI fail diagnosis.

Let's say you are collecting evidence; at what point in time do you realize that instead of collecting more info, I can act because I'm pretty much sure?

How can I think in terms of the cost of acting vs. the cost of gaining more evidence? Is it only about time, or are there any other costs involved?

I'm trying understand in how practitioners think why their CI run fails

For that I am curious to know about how practitioners think when a CI run fails.

When a CI build fails and there are multiple potential causes, what’s the very first thing you inspect? Also, what’s your immediate mental checklist or set of suspicions when a pipeline turns red?
how do you usually determine whether it’s a genuine code issue versus just a flaky failure?
And at last how do you determine that I'm pretty much sure what's happening and there's not much value of additional searching.
Would love to hear how you all approach CI triage!

I'm trying to build an agent that finds the reason why CI run failed.

For that I am curious to know about how practitioners think when a CI run fails.

When a CI build fails and there are multiple potential causes, what’s the very first thing you inspect? Also, what’s your immediate mental checklist or set of suspicions when a pipeline turns red?
how do you usually determine whether it’s a genuine code issue versus just a flaky failure?
And at last how do you determine that I'm pretty much sure what's happening and there's not much value of additional searching.
Would love to hear how you all approach CI triage!

I'm trying to build an agent that finds the reason why CI run failed

Like, what are the things going on in your head and

Let's imagine. Your brain has something like a belief system that sort of makes you suspect certain things more.

I want to know how practitioners think, like what doubts they are holding the moment they see a CI file, and how they determine what to look for to find the real cause and if it's a real code failure or some flaky test

And finally, At what point do they determine that they're pretty much sure about the cause?

After, I learned linear regression, I thought let's do a project.I started with the data and suddenly, I am prompting with chatgpt, if give it a plan and ask to break it, now it look's like nothing works, How should I do this task so that i won't get stuck in optimization and what's the right way to do data clearning an feature engineering .