What I've built to keep agents alive on long-horizon tasks, and why it's still not enough. Looking for concrete suggestions
What I've built to keep agents alive on long-horizon tasks, and why it's still not enough. Looking for concrete suggestions
I run a fleet of LLM agents on long jobs that last hours to days. The sample project: a pipeline that transcribes and summarizes a few hundred videos from several channels into a searchable knowledge base. It runs in batches, hits rate limits, gets interrupted, and has to resume without redoing or double-sending anything.
Agents kept dying quietly. Nothing crashed loudly, they just stopped, and I found out hours later. Here is what I built so far.
- Heartbeats on one shared board. One row per job: last check, next expected check, deadline, current step, checkpoint link. Everyone writes the same six fields, so a single sheet shows the whole fleet.
- Deadlines at 2-3x the check interval. A missed beat is a flag, not an instant verdict.
- Bounded retries. Every retry loop has a cap and a total deadline. Standing watches leave the total deadline blank so they never "expire"; only finite jobs get one.
- A parent audit every 30 minutes. A separate agent reads the board and flags overdue rows. If more than half the jobs are flagged at once, it sends one grouped alert, because that is usually an outage or quota problem, not 10 dead agents.
- "Unknown" is never "dead". If the status check itself is throttled or the service is down, the result is unknown and nobody gets relaunched.
- Intent/done records for side effects. Before and after anything irreversible (send, commit, publish) the agent writes an operation key, so a relaunched agent can tell whether it already happened.
- Epoch fencing for relaunches. A relaunched agent gets a higher epoch and the old one stops itself if it wakes up and sees a newer epoch.
- Drive-first checkpoints and milestone commits. Progress goes to durable storage after each batch, with a pointer in the board row.
- A watchdog wake that checks the audit itself is still running.
Why it's still not enough:
- Credential delivery. Agents can't safely get a git token. Our vault can only fill browser inputs and never hands a value to a shell, so commits go through a slow browser route, small text files only.
- Workspace resets wipe local state. Anything not checkpointed externally is gone, and the board only helps if the agent got to write to it.
- No external dead-man switch. If the audit agent and the board go down together, nothing notices. Everything lives in the same ecosystem.
- Step and context limits. Agents hit them mid-task and stop without a clean handoff.
- Sheets quota throttling. With a dozen agents beating hourly plus audits, read quota runs out, and I've had to treat throttling as "unknown" everywhere.
What I'd like suggestions on:
- A cheap external dead-man switch that doesn't depend on the same platform (cron ping service? something else?).
- A better place than a spreadsheet for heartbeats at this scale.
- How you hand secrets to agents so they can commit without exposing the value in chat or logs.
- Patterns for clean handoff when an agent is about to hit its step limit.
- Anything you'd cut. I suspect some of this is over-built.
Happy to share more detail on any piece.