I've been using coding agents heavily, and one architecture problem kept bothering me:

The same system that modifies the repository is usually also the system deciding whether the task is finished.

It writes the code, runs some commands, summarizes what happened and says “done”.

I wanted that final decision to live outside the agent's own reasoning loop.

So I built plan-auditor, an open-source verification supervisor for coding agents.

It isn't one big evaluator. I split it into 15 layers with different responsibilities and, more importantly, different levels of authority.

Here is what those layers actually are.

L0 — Event detection

A small deterministic layer detects things such as completion claims, retries, changes to verification criteria and some security-relevant patterns.

It can trigger verification.

It cannot declare PASS.

L1 — Requirements

The task is represented as structured requirements with priority, acceptance criteria, dependencies, ambiguity and verification strategy.

There is also a host-owned request contract so the plan inside the workspace isn't automatically allowed to redefine what the user originally requested.

L2 — Workspace model

The verifier independently reads the real repository state:

• Git branch and HEAD
• dirty and untracked files
• file inventory
• detected language
• available tools

So the agent's description of the workspace isn't treated as ground truth.

L3 — Policy engine

Deterministic policies evaluate plan integrity, verification state and security-related conditions.

Control paths are also constrained to the workspace rather than blindly trusting arbitrary filesystem paths.

L4 — Goal state

The supervisor keeps its own explicit representation of task/goal state instead of deriving completion from the agent's latest message.

L5 — Plan verifier

Before implementation can be considered complete, the plan itself is checked.

This includes:

• dependency graph validity
• topological ordering
• requirement coverage
• declared and required outputs
• dependency-to-output bindings
• behavioral verification

A step that only checks file_exists or a regex isn't considered strongly verified.

Behavioral checks such as an actual command, test or executable verification are required.

L6 — Lifecycle

The task moves through an explicit state machine rather than jumping directly from “working” to “done”.

The lifecycle includes states for analysis, plan review, sealing, implementation, verification, final audit, retries, escalation, recovery, failure and UNKNOWN.

L7 — Authority hierarchy

Not every layer is allowed to overrule every other layer.

Safety, plan integrity and deterministic verification have higher authority than semantic AI judgment.

If a deterministic test fails, an AI reviewer saying “looks correct” cannot turn that result into PASS.

L8 — Contract sealing

Once a verification plan is approved, its contract is sealed.

The seal covers the task, requirements, required tools, steps, dependencies, outputs, requirement coverage, verification checks and verifier environment.

The contract can be strengthened later.

It cannot be silently weakened.

Adding another deterministic check is fine.

Removing an existing check, requirement, dependency or approved output scope is not.

L9 — Watchdog

The workspace is observed for file creation, deletion, modification and Git changes.

The final verification path also fingerprints the workspace so it doesn't certify a target that changed while it was being audited.

L10 — Deterministic audit core

This is where an agent's claim becomes actual evidence.

If the agent says:

>“all tests pass”

that statement itself has zero verification authority.

The checks are executed again in fresh subprocesses and their real results are used.

This is the deterministic path that can mark a step verified.

Repeated failures are also tracked instead of allowing unlimited retries until something happens to turn green.

L11 — Evidence integrity

Verification results are stored in a SHA-256-linked JSONL evidence chain.

Each record links to the previous one.

Rotated evidence archives are linked as well.

Optional external HMAC authentication can protect records and checkpoints when the key is kept outside the agent's reach.

L12 — Adversarial / semantic review

There is an optional AI-assisted review layer.

But I deliberately did not give it permission to create PASS.

It can find a possible problem and request stronger deterministic verification.

A serious semantic concern can hold the result at UNKNOWN until that concern is converted into something checkable.

L13 — Completion gate

Only this layer emits the final:

PASS / FAIL / UNKNOWN

It considers fresh deterministic evidence, unfinished steps, policy findings, contract integrity and adversarial findings.

No fresh deterministic proof means no PASS.

L14 — Multi-agent registry

For multiple coding agents working on the same workspace, the system tracks agent identity, task/plan assignment, heartbeat, current action, retries, file ownership and conflicts.

The registry itself is also persisted as a sequence/hash-linked log rather than existing only in memory.

There is one additional part I found useful: a deterministic formal compiler.

It takes already-approved structured requirements, dependencies, named outputs and coverage information and compiles them into a conservative STRIPS-style planning contract.

It can represent facts such as:

step-completed:3

output-available:2:<name>

requirement-satisfied:REQ-004

It deliberately doesn't read natural-language prose and invent arbitrary domain semantics.

The generated contract carries a fingerprint of the source plan and can be independently recompiled during verification.

So removing a requirement, dropping an output prerequisite or modifying the generated contract produces a detectable mismatch.

The deterministic verification path itself does not require an LLM.

One important limitation: this is not an OS sandbox.

If a deliberately malicious agent has the same OS credentials as the verifier, hashes and Python-level locks aren't a kernel security boundary.

For that threat model, the verifier needs to run under a separate OS identity, container or VM, with its integrity key inaccessible to the agent.

I'm also not claiming it is “X% better” than other agent-verification approaches because I don't have a controlled cross-tool benchmark that would justify a number like that.

The project is free, open source and MIT licensed.

The part I'm most interested in getting feedback on is the authority boundary: semantic AI review is allowed to find problems, but it is never allowed to manufacture PASS.

I've been using coding agents heavily, and one architecture problem kept bothering me:

The same system that modifies the repository is usually also the system deciding whether the task is finished.

It writes the code, runs some commands, summarizes what happened and says “done”.

I wanted that final decision to live outside the agent's own reasoning loop.

So I built plan-auditor, an open-source verification supervisor for coding agents.

It isn't one big evaluator. I split it into 15 layers with different responsibilities and, more importantly, different levels of authority.

Here is what those layers actually are.

L0 — Event detection

A small deterministic layer detects things such as completion claims, retries, changes to verification criteria and some security-relevant patterns.

It can trigger verification.

It cannot declare PASS.

L1 — Requirements

The task is represented as structured requirements with priority, acceptance criteria, dependencies, ambiguity and verification strategy.

There is also a host-owned request contract so the plan inside the workspace isn't automatically allowed to redefine what the user originally requested.

L2 — Workspace model

The verifier independently reads the real repository state:

• Git branch and HEAD
• dirty and untracked files
• file inventory
• detected language
• available tools

So the agent's description of the workspace isn't treated as ground truth.

L3 — Policy engine

Deterministic policies evaluate plan integrity, verification state and security-related conditions.

Control paths are also constrained to the workspace rather than blindly trusting arbitrary filesystem paths.

L4 — Goal state

The supervisor keeps its own explicit representation of task/goal state instead of deriving completion from the agent's latest message.

L5 — Plan verifier

Before implementation can be considered complete, the plan itself is checked.

This includes:

• dependency graph validity
• topological ordering
• requirement coverage
• declared and required outputs
• dependency-to-output bindings
• behavioral verification

A step that only checks file_exists or a regex isn't considered strongly verified.

Behavioral checks such as an actual command, test or executable verification are required.

L6 — Lifecycle

The task moves through an explicit state machine rather than jumping directly from “working” to “done”.

The lifecycle includes states for analysis, plan review, sealing, implementation, verification, final audit, retries, escalation, recovery, failure and UNKNOWN.

L7 — Authority hierarchy

Not every layer is allowed to overrule every other layer.

Safety, plan integrity and deterministic verification have higher authority than semantic AI judgment.

If a deterministic test fails, an AI reviewer saying “looks correct” cannot turn that result into PASS.

L8 — Contract sealing

Once a verification plan is approved, its contract is sealed.

The seal covers the task, requirements, required tools, steps, dependencies, outputs, requirement coverage, verification checks and verifier environment.

The contract can be strengthened later.

It cannot be silently weakened.

Adding another deterministic check is fine.

Removing an existing check, requirement, dependency or approved output scope is not.

L9 — Watchdog

The workspace is observed for file creation, deletion, modification and Git changes.

The final verification path also fingerprints the workspace so it doesn't certify a target that changed while it was being audited.

L10 — Deterministic audit core

This is where an agent's claim becomes actual evidence.

If the agent says:

>“all tests pass”

that statement itself has zero verification authority.

The checks are executed again in fresh subprocesses and their real results are used.

This is the deterministic path that can mark a step verified.

Repeated failures are also tracked instead of allowing unlimited retries until something happens to turn green.

L11 — Evidence integrity

Verification results are stored in a SHA-256-linked JSONL evidence chain.

Each record links to the previous one.

Rotated evidence archives are linked as well.

Optional external HMAC authentication can protect records and checkpoints when the key is kept outside the agent's reach.

L12 — Adversarial / semantic review

There is an optional AI-assisted review layer.

But I deliberately did not give it permission to create PASS.

It can find a possible problem and request stronger deterministic verification.

A serious semantic concern can hold the result at UNKNOWN until that concern is converted into something checkable.

L13 — Completion gate

Only this layer emits the final:

PASS / FAIL / UNKNOWN

It considers fresh deterministic evidence, unfinished steps, policy findings, contract integrity and adversarial findings.

No fresh deterministic proof means no PASS.

L14 — Multi-agent registry

For multiple coding agents working on the same workspace, the system tracks agent identity, task/plan assignment, heartbeat, current action, retries, file ownership and conflicts.

The registry itself is also persisted as a sequence/hash-linked log rather than existing only in memory.

There is one additional part I found useful: a deterministic formal compiler.

It takes already-approved structured requirements, dependencies, named outputs and coverage information and compiles them into a conservative STRIPS-style planning contract.

It can represent facts such as:

step-completed:3

output-available:2:<name>

requirement-satisfied:REQ-004

It deliberately doesn't read natural-language prose and invent arbitrary domain semantics.

The generated contract carries a fingerprint of the source plan and can be independently recompiled during verification.

So removing a requirement, dropping an output prerequisite or modifying the generated contract produces a detectable mismatch.

The deterministic verification path itself does not require an LLM.

One important limitation: this is not an OS sandbox.

If a deliberately malicious agent has the same OS credentials as the verifier, hashes and Python-level locks aren't a kernel security boundary.

For that threat model, the verifier needs to run under a separate OS identity, container or VM, with its integrity key inaccessible to the agent.

I'm also not claiming it is “X% better” than other agent-verification approaches because I don't have a controlled cross-tool benchmark that would justify a number like that.

The project is free, open source and MIT licensed.

The part I'm most interested in getting feedback on is the authority boundary: semantic AI review is allowed to find problems, but it is never allowed to manufacture PASS.

I’ve been working on a small open-source project called Plan Auditor.

The idea came from a problem I kept running into with coding agents: they can say a task is finished even when part of the original request was skipped, a test wasn’t really run, or an intermediate result was never verified.

So instead of trusting the agent’s final message, Plan Auditor checks the work separately.

It keeps the original requirements explicit, links them to plan steps, runs real checks, verifies dependencies and outputs, and only returns PASS when the current state has fresh evidence behind it.

I also added a formal-planning layer. For larger plans it can generate a grounded STRIPS model from the structured plan and verify that model separately. The generation itself is deterministic, so another LLM isn’t being trusted to decide what counts as a valid proof.

There’s also support for sealed plans, evidence chains, multiple active plans, and parallel agents working in the same repository.

The core verification path runs locally and doesn’t require an LLM or API.

It’s still not meant to be an OS sandbox. If a malicious process has the same OS permissions as the verifier, that needs actual process/container/VM isolation.

Repo:
https://github.com/Furox-Art/plan-auditor

I’d be interested in feedback, especially on the verification approach and formal-planning part.