Skip to main content
For instruction improvement, start with agentuse test result <agent> --session <id>: it supplies source evidence directly without depending on matching tool calls. See Testing agents. This page documents the advanced strict-replay compatibility command. Use replay when you are changing an agent’s instructions and want to hold its external inputs fixed:
The writer runs again. Its tools return recorded results instead of opening browsers, running commands, updating stores, or starting child agents. No mock model is needed. This is a fresh generation, not a continuation of the source session.

​
What stays fixed and what changes

  • Current: the local agent definition and configured model, explicit preloaded skills, applicable learnings, and explicitly granted read-only reference files. Replay reads learnings without recording usage.
  • Recorded: the original additional user prompt, tool schemas, external results and errors. Relative dates use the original session’s start date.
  • Withheld from the writer: the old assistant draft, its approval proposal, later revisions and human feedback. Those appear only in the comparison.
Explicit read-only file grants are snapshotted before generation so edits to writing briefs take effect. Directory and wildcard grants stay recorded because they may contain source data or drafts. The report lists refreshed files and hashes; do not treat a refreshed data file as frozen evidence. Inline agent instructions and preloaded skills are always current. Dynamic skill/tool requests use their recordings.

​
Where it stops

Replay captures the first await_human input before gate preflight, automated verification or human approval. It does not grant approval, run revisions, or publish. An agent that finishes without a gate yields its final output. An explicit incomplete report remains incomplete. A tool call without a matching recording stops the test with REPLAY_INPUT_MISSING. No live lookup or LLM fabrication fills the gap. The failed call and its exact inputs appear in the report. Matching uses the tool name and arguments. Object key order and the narration field intent do not matter; filesystem read paths are normalized. Other arguments, including shell commands, URLs, prompts and array order, must match. Repeated calls consume matching recordings in their original order. store_list permits a different field projection when filters are identical and the requested page is contained in the recorded page and every requested value or known absence is recorded. Metadata requests (such as id and title) use the metadata already present on each row. Unknown data fields still stop the replay. Projection provenance appears in the report. store_create writes to an isolated in-memory store with fresh replay- IDs. Those items support store_get, store_update, and store_delete, so new drafts can change without touching production. Queries after these writes are not yet supported; replay stops rather than returning stale recorded lists. Only recordings before the source’s first approval gate are available. For an ungated run, later user continuations are excluded. Mock and replay sessions cannot be used as sources. An actively running source must first finish or reach approval.

​
Read the result

The CLI prints the old and new proposal/output and a path to replay.json. With --json, it emits the structured comparison, refreshed-reference hashes, per-call recording provenance and any mismatch:
The new session is marked as a test (mock: true) and stores config.replaySourceSessionId. It does not modify or resume the original session. Replay sessions cannot be resumed as live runs; run another replay to try the next instruction change. Exit status is 0 for a captured proposal or completed output, and 1 for a mismatch, incomplete outcome or execution error. Success means the comparison was captured, not that the draft passed a quality review.

​
First-version limits

Replay supports recorded text/JSON tool results. Tool schemas are frozen, so this command measures instruction changes, not a migration to new tools. A changed query, a new research request, or a write/dry-run containing newly generated text may not have a matching recording. That is a visible mismatch, not permission to return unrelated old data. No shell command executes, including commands that are usually read-only or called “dry run.” This command does not support --scope, --approval, --mock-model, a remote agent URL, or an additional prompt. Use ordinary agentuse test for mocked flow testing. Judge regression against past human decisions is a separate operation; replay deliberately captures the writer before the judge intervenes.

​
Experimental fixed-input extraction

Full workflow replay still depends on recorded calls. For testing an agent’s output from fixed evidence, the prototype scripts/extract-fixed-inputs.ts asks a model to classify the source session’s pre-approval tool results. It works across tool names and output formats; it does not assume the output is writing. It does not grade the output or rerun the agent. The selector distinguishes source evidence, agent-produced material, feedback, reference instructions, irrelevant records, and uncertain provenance. It sees call arguments to detect generated material read back through another tool. Only tool-result text is selectable. Code validates complete classification, source IDs, exact excerpts, and exclusion of known mutation outputs. Classifier reasons stay in a separate audit, not the evidence pack. This is LLM-assisted selection, not a guarantee of semantic purity: arbitrary tools can return mixed evidence and generated content, and classification can be wrong. Inspect the audit before treating a pack as a regression fixture. The model receives broader historical tool data than the eventual input pack; use an endpoint authorized for that data. No tool executes during extraction. From the source checkout, build the standalone prototype for Node:
Then run from the original agent’s project so session storage resolves correctly:
The output is input-pack.json, selection-audit.json, source-records.json, and a separate baseline.json. The pack contains exact excerpts, hashes, and UTF-16 string offsets; old assistant output and classifier explanations are not copied into it. Reference instructions should be supplied fresh to the output-generation step. Large sessions currently fail an explicit size limit instead of silently truncating evidence. The public agentuse test result command now uses this validated selection primitive and caches evidence for reuse. The standalone script remains useful for inspecting extraction separately.

​
Experimental output loop

scripts/loop-x-output.ts reuses an existing fixed-input pack for a controlled X reply experiment. It performs fresh generation, conversation and evidence checks, then instruction revision only after failure. The canonical brief, source pack, model and evaluator criteria remain fixed. A passing draft stops the loop without forced polishing; three failed rounds exhaust the run. The reviser produces a separate test-guidance paragraph. The next writer sees that guidance, not the prior drafts, judgments, or historical baseline. Nothing is written back to production instructions and no publishing tools exist in the experiment. Automated acceptance is not proof of improvement or permission to publish. A single example cannot establish a general instruction improvement. Build from the source checkout and run from the source agent’s project:
The result directory preserves every role’s exact request/response, each round’s judgment and instruction revision, frozen inputs, hashes, and the final status. The optional historical baseline is loaded only after all model calls finish. This prototype is separate from the public agentuse test command.