u/No-Cheetah-4745

I'm new to building AI agents and working on a CI-review agent: it takes a failing CI run and tries to find the root cause.

I want to start with a baseline version, then iterate on it — but I need a way to measure whether a new version is actually outperforming the old one.

Right now the agent maintains several hypotheses about the failure, assigns each a probability, and updates those probabilities as it gathers more information. Depending on its confidence, it either outputs a summary of the likely root cause, or escalates to a human developer if uncertainty is too high.

Two questions:

1. How do you evaluate an agent like this, where the output isn't a single "correct" answer but a probability distribution over hypotheses?
2. How do you get a labeled dataset of CI failures with known root causes, so I can score the agent's probability estimates against ground truth instead of just eyeballing whether the output "feels right"?

I'm new to building AI agents and working on a CI-review agent: it takes a failing CI run and tries to find the root cause.

I want to start with a baseline version, then iterate on it — but I need a way to measure whether a new version is actually outperforming the old one.

Right now the agent maintains several hypotheses about the failure, assigns each a probability, and updates those probabilities as it gathers more information. Depending on its confidence, it either outputs a summary of the likely root cause, or escalates to a human developer if uncertainty is too high.

Two questions:

1. How do you evaluate an agent like this, where the output isn't a single "correct" answer but a probability distribution over hypotheses?
2. How do you get a labeled dataset of CI failures with known root causes, so I can score the agent's probability estimates against ground truth instead of just eyeballing whether the output "feels right"?

I'm new to building AI agents and working on a CI-review agent: it takes a failing CI run and tries to find the root cause.

I want to start with a baseline version, then iterate on it — but I need a way to measure whether a new version is actually outperforming the old one.

Right now the agent maintains several hypotheses about the failure, assigns each a probability, and updates those probabilities as it gathers more information. Depending on its confidence, it either outputs a summary of the likely root cause, or escalates to a human developer if uncertainty is too high.

Two questions:

1. How do you evaluate an agent like this, where the output isn't a single "correct" answer but a probability distribution over hypotheses?
2. How do you get a labeled dataset of CI failures with known root causes, so I can score the agent's probability estimates against ground truth instead of just eyeballing whether the output "feels right"?

I'm new to building AI agents and working on a CI-review agent: it takes a failing CI run and tries to find the root cause.

I want to start with a baseline version, then iterate on it — but I need a way to measure whether a new version is actually outperforming the old one.

Right now the agent maintains several hypotheses about the failure, assigns each a probability, and updates those probabilities as it gathers more information. Depending on its confidence, it either outputs a summary of the likely root cause, or escalates to a human developer if uncertainty is too high.

Two questions:

1. How do you evaluate an agent like this, where the output isn't a single "correct" answer but a probability distribution over hypotheses?
2. How do you get a labeled dataset of CI failures with known root causes, so I can score the agent's probability estimates against ground truth instead of just eyeballing whether the output "feels right"?

Hey everyone,

I’m building an AI agent designed to diagnose failing CI/CD builds. Instead of using a simple one-shot chain, I’m structuring it as a stateful agent (using LangGraph) that manages dynamic hypothesis updating.

The agent maintains a state array of possible root causes, assigns probability scores to each hypothesis, and updates those probabilities as it invokes tools to parse build logs, git diffs, and context files.

• High Confidence: It routes to an output node that provides a concise root-cause summary and fix recommendation.
• High Uncertainty: It routes to a human-in-the-loop (HITL) node for developer escalation.

As I build out the baseline state graph, I need advice on two fronts:

1. Evaluation in LangSmith: How do you effectively benchmark an agent whose trajectory involves continuous state-based probability updates? Beyond final-output "LLM-as-a-judge", what custom evaluators or intermediate state checks are best for measuring single-step decision-making, calibration error, and escalation threshold reliability across agent iterations?
2. Ground-Truth Dataset Sourcing: I want to ground the agent's probability updates in real failure distributions rather than raw LLM estimates. Are there recommended ways to pull historical GitHub Actions/Travis CI logs at scale, or existing open-source benchmarks (e.g., BugSwarm or SWE-bench) suited for offline LangSmith datasets?

've been thinking about this after seeing a lot of people here talk about struggling to get internships/jobs even after finishing CS/IT degrees.

From what I've seen in discussions here, there seem to be 3 camps:

• Degree + skills = safest route
• Degree doesn't matter much once you have solid projects/experience
• Degree mainly matters because HR/recruiters use it as a filter

The interesting part is that I've seen people on both sides claiming real experience. Some say they've never been asked about their degree after getting into the industry, while others say companies won't even consider you without a bachelor's.

So for people actually working in Nepal's tech industry:

If you had to start over today, would you still spend 4 years getting a CS/IT degree?

Or would you do something like:

+2 → self-learn → build projects → internship → job → degree later (if needed)

And for people who hire: does the degree actually influence your decision, or is it just an HR requirement before the technical team even sees the CV?

Genuinely curious about people's experiences rather than the usual "skills > degree" / "degree is necessary" one-liners.

I'm new to building AI agents and working on a CI-review agent: it takes a failing CI run and tries to find the root cause.

I want to start with a baseline version, then iterate on it — but I need a way to measure whether a new version is actually outperforming the old one.

Right now the agent maintains several hypotheses about the failure, assigns each a probability, and updates those probabilities as it gathers more information. Depending on its confidence, it either outputs a summary of the likely root cause, or escalates to a human developer if uncertainty is too high.

Two questions:

1. How do you evaluate an agent like this, where the output isn't a single "correct" answer but a probability distribution over hypotheses?
2. How do you get a labeled dataset of CI failures with known root causes, so I can score the agent's probability estimates against ground truth instead of just eyeballing whether the output "feels right"?