Skip to main content
The verify configuration, verdict contract, and redo behavior are experimental and may change without migration. Discuss feedback in GitHub Discussions.

​
What is Verify?

Verify puts a quality gate between your agent’s final output and delivery. After the run finishes, a judge evaluates the output against your criteria. If it fails, the judge’s critique is injected back into the same session as a revision instruction and the agent redoes the output, with all its tool results and completed work still in context, up to maxRedos times. Only then does the output ship to channels, a delegating manager, or the CLI. The redo is a continuation, not a re-run: the agent sees everything it already did (including side effects like sent messages or filed issues), so it revises the deliverable instead of repeating actions.

​
Quick Start

Or with your own rubric:

​
Configuration Options

verify: true uses a generic task-fulfillment rubric. verify: "<string>" is shorthand for { criteria: "<string>" }.

​
Gate Placement (approval agents)

For agents with an approval gate, verify defaults to at: gate: the judge reviews each await_human request before it suspends to the human.
  • Fail → the run does not suspend. The critique returns as a rejection-with-comment tool result, the same protocol as a human rejection, so the agent revises its draft and requests approval again. The human never sees the failed draft.
  • Pass → the gate suspends to the human as normal.
  • After a human Comment → the next gate skips pre-review and goes directly back to that reviewer. Their requested revision is authoritative for that review cycle; an automated judge cannot reinterpret or override it. Machine pre-review feedback does not trigger this bypass.
  • Fail-open → after maxRedos pre-review rejections, or on any judge error, the request goes to the human regardless, with the unresolved critique in the session log. The judge protects your attention; it can never deadlock a run or replace your decision. A gate that reaches the human this way records a skipped verify marker, so the approval card and sessions show say “not judged” and why. gateReview: fresh instead opens a feedback-only human gate and keeps approval blocked.
When a gate offers several drafts (a slate keyed by options / changes[].optionId), the judge returns one verdict per candidate and lists every failing candidate in one pass. A candidate that passed keeps its pass on the next attempt as long as the reviewed content is byte-identical (the visible text, the exact command, and the media URLs): the gate carries it forward, tells the judge not to re-judge it, and skips the judge call entirely when nothing changed. The bounce sent back to the agent names the passing candidates so it leaves them alone. A rejection of the whole request holds even when every candidate passed. With a judge agent, the attempts on one gate share one judge session: the second attempt resumes the first attempt’s child session and sends only the revised request plus which candidates changed. The judge keeps its own reasoning about what it already read, and the provider serves the long prefix from cache, so extra redos cost a fraction of the first. The session is dropped whenever the gate suspends, so a human decision always gets a fresh judge, and a resume is declined when the judge history grows past a fixed size or the judge file changed. Otherwise, the judge receives the full approval payload (draft, exact on-approval content, target reference/excerpt, context) plus bounded prior human decisions from the session, so write criteria against what the reviewer would see. Because the check runs before anything posts, this placement is the quality gate; at: output is a post-run audit. Gate placement also works for delegated sub-agents: a leaf’s gate is judged in the leaf, before the request cascades to the root.
The payload is all the built-in judge sees. It runs without tools, so a criterion about a file on disk is really judged against the agent’s account of that file. A payload that paraphrases its own artifact gets rejected even when the artifact is correct, and a payload that flatters a broken artifact passes silently. When the criteria describe something on disk, use a judge agent that can open it.

​
Using a Judge Agent

When a one-line rubric isn’t enough, the evaluation needs its own files, tools, or a stronger model, point judge at another agent file:
The path resolves relative to the agent file that declares it. judge and criteria are mutually exclusive, and model is invalid alongside judge (the judge agent declares its own). The judge’s body is the rubric. Where the built-in judge gets your criteria, an agent judge is handed Apply the evaluation standard defined in your own instructions. in that slot, so the standard has to live in the judge agent itself. Everything else it needs, the task and the candidate output, arrives around it. The judge is otherwise a normal agent: its own frontmatter controls its model, reasoning effort, and tools. That is the point of using one. Grant it read access to the rubric files, exemplars, or the artifact under review, and it evaluates what is actually on disk instead of what the payload claims. It runs as an inspectable child session, so which files it opened and why it decided are visible in the session log, not just the one-line verdict. It records its decision by calling submit_verdict, a tool injected into every judge agent, exactly once:
The tool call is the verdict; a judge should not also write a JSON object in its text. (A trailing JSON object is still parsed as a fallback, which is how the built-in judge reports, but new judge agents should call the tool.) When pass is false, critique is required and must be actionable in one revision, it is injected verbatim into the redo turn. Everything else the judge writes stays in its own reasoning. Judges conventionally stay read-only (they evaluate, the parent revises), but this is not enforced, a judge that runs your test suite is a legitimate design. Judges cannot suspend on approval gates, so never give a judge agent approval: true; a suspended judge counts as a judge error. A nested verify: on a judge is ignored.

​
What Happens on Failure

  1. The verdict and critique are recorded in the session log (visible in the web UI).
  2. The critique is injected as a revision instruction: attempt count, criteria, the critique verbatim, and directives (revise, don’t argue with the review, completed side effects stand, tools allowed for new information).
  3. The agent’s revised output is judged again, up to maxRedos.
  4. If every attempt fails, the last output still ships, and the session is marked verification: failed, a needs-attention signal for you or a delegating manager, not a crash.
A judge that itself errors (model/auth/parse failure) never blocks the run: the output ships unverified with a visible error marker.

​
Cost and Behavior Notes

  • Each verification is one extra LLM call; each redo is a continuation of the existing session (prompt caching keeps it near generation-only cost). Redo steps count against maxSteps.
  • On a gate with a judge agent, the judge’s own session is resumed across attempts, so raising maxRedos on a slate agent is cheaper than the count suggests: later attempts re-read a cached prefix and re-judge only the candidates that changed.
  • The built-in judge defaults to the agent’s own model so authentication always works; set model to use a cheaper judge.
  • Verify requires session storage (the default), it is skipped when no session substrate is available, and skipped for runs that declared themselves incomplete via report_outcome.
  • If a redo hits an approval gate, the run suspends normally; verification resolves at the end of the resumed run.

​
Verify vs. Learning

Verify catches a bad output now; Learning makes the agent better next run. They compose: a verify critique that fixed the output this run is exactly the kind of rule worth promoting into learnings (automatic promotion is planned).

​
Fresh gate review

For editorial workflows that require a new reading of every approval request, opt in with gateReview: fresh:
Each judged gate gets a fresh judge session and all candidates are checked again. No previous candidate verdict is carried forward, so unchanged wording is rechecked when the source or evidence changes. A gate that directly follows a human Comment skips the judge, exactly as in the default mode: the revision returns straight to the reviewer who asked for it, with a skipped marker saying it was not judged. The judge filters what reaches the reviewer’s first look; it does not second-guess a reviewer who is already in the loop. Fresh mode never lets an agent’s own unreviewed replacement reach approval. When failed verdicts or reviewer errors consume the review budget, the current draft returns to the human in a feedback-only gate with the latest critique. Approval stays unavailable. The reviewer can comment with guidance to resume the same session, which returns the revision straight to them, or reject to stop the action. The current draft, prior human comments, and automated critique remain in the session rather than being discarded by a terminal failure. With maxRedos: 0, the initial request is still judged once. A failure then opens the same feedback-only gate. A reviewer error consumes one attempt. An approval still requires the human’s decision after the automated pass. Without this option, existing incremental review and fail-open behavior remain unchanged. This setting only affects gate placement, not final-output review.