FIRSTMATE-34: Parallel validation - bind no-mistakes responds to each worker's own run #51

Merged
john merged 3 commits from fm/nm-concurrent-intent-crosstalk into main 2026-10-21 22:51:09 +00:00
Owner

Story: FIRSTMATE-34 - Parallel validation

Lets more than one change in the same repository go through no-mistakes validation at once, by binding every gate answer to the worker's own run, and keeps validation load proofs bounded.

Investigation

Proven on 2026-10-20 with two concurrent real runs on two branches of a disposable scratch repository (ejected afterwards); the full commands and output are in docs/verification/no-mistakes-branch-isolation.md.

  • The shared daemon already runs v1.84.0: it was restarted at 2026-10-20 22:11 UTC, its /proc/<pid>/exe hash matches the installed v1.84.0 binary, and every run since records no_mistakes_version=v1.84.0. The v1.53.0 premise is out of date, so no restart is needed.
  • On v1.84.0, bare axi status and axi respond resolve from the caller's checked-out branch. Each worktree's status showed only its own run. A worktree with no run of its own, the main checkout, and a detached HEAD all got error: no active run to respond to while another branch's run sat at a gate. respond from the second run's worktree answered only that run, and the first run's gate stayed untouched.
  • axi respond has no --run selector, so the checked-out branch is the only binding. axi status --run <id> exists for deliberate reads.
  • v1.53.0 could not be re-probed live (no binary left). The run store shows 188 same-repository, different-branch overlapping runs on v1.53.0, so concurrent runs were always possible there. The 2026-09-09 crossing matches a branch with no run being shown another branch's run, which v1.84.0 now refuses explicitly.
  • Daemon restart: on v1.84.0, daemon restart, daemon stop, and update refuse while any run is active unless --force is passed. The run store holds 19 runs that failed with daemon shutting down from forced stops on older daemons. Procedure: confirm no run is active in any repository, then run no-mistakes daemon restart without --force, then confirm the running binary by hash. A forced restart while runs are live is not safe.

Changes

  • bin/fm-dod-lib.sh (the generated no-mistakes contract, for every forge): the worker issues every axi call from its own copy with its ship branch checked out, records the run id that its starting drive call returns, reads status with --run <id>, runs a bare axi status immediately before every respond, and answers only when the id, the branch, and the awaiting step all match, passing --step explicitly. A gate or a firstmate decision whose nm-<run>-<step> key names another run is refused with a blocked: run-mismatch line.
  • The same contract adds the bounded-load rule: load proofs use at most two short-lived niced loops, never one per core, and measure CPU time instead of generating load wherever that can show the same thing. A test-step gate that shows host-saturating load gets --action fix, never approve.
  • .agents/skills/validation-supervision/SKILL.md: concurrent no-mistakes runs on different branches of one repository need no per-repository serialization on v1.84.0 and later. Firstmate's decision relays now also name the run id.
  • docs/verification/no-mistakes-branch-isolation.md: a new maintainer-verification record, classified in docs/documentation-audiences.json.
  • tests/fm-brief.test.sh: tests that the binding and the bounded-load rule render on every forge and stay out of pipeline-free briefs. A mutation that drops the refusal fails the test.

Verification

  • tests/fm-brief.test.sh, tests/fm-task-delivery.test.sh, tests/fm-dod-lib.test.sh, tests/fm-ask-user-authority.test.sh, tests/fm-classify-corr-token.test.sh, tests/fm-spawn-dispatch-profile.test.sh, tests/fm-pr-check-security.test.sh, tests/fm-control-relaunch.test.sh, tests/fm-tangle-guard.test.sh, and tests/fm-subagent-pretool-check.test.sh all pass.
  • bin/fm-lint.sh and bin/fm-doc-audience-check.sh are clean.

Follow-up (out of scope)

  • The private fleet records still say to serialize one pipeline per repository (data/learnings.md "One repo = one pipeline", and the nomistakes-pipeline-is-repo-scoped memory). Firstmate should retire them once this lands.
  • The pipeline's own Test-step agent follows each project's trusted .no-mistakes.yaml test.instructions, which a worker brief cannot reach. Adding the bounded-load sentence there (Flow first) would bound the pipeline agent directly.
  • A spawn-time check of the daemon version (refuse concurrent runs below v1.84.0) was not added, because no such gate existed and the run binding refuses a crossed gate on any version.

Review 1 fix: recycled branch

A recycled copy can still carry an earlier task's active run on the new task's ship branch. Re-probed on v1.84.0 in a scratch repo (evidence now in the verification record): a plain axi run --intent there silently reattaches to the earlier run and ignores the new intent, while a launch with a fresh --launch-nonce creates a new run and cancels the earlier one (superseded by new push).
The contract now proves ownership. Before starting, the worker refuses (blocked) when its branch has an active run it did not create. Every new run starts with --launch-nonce <task-id>-<epoch> --validation-generation <task-id>. The worker drives a run only when its launch_receipt shows disposition: created, its own ship branch, and submitted_head_sha equal to its HEAD, and it saves that receipt in its task data directory. Every reattach must return that run id. tests/fm-brief.test.sh adds a recycled-branch regression on every forge, and a mutation that drops the refusal fails it.

Review 2 fixes

  • The pre-start check is now an allowlist of three safe status shapes: runs_on_current_branch: 0, a terminal run (completed, failed, cancelled), or the run id in the worker's own saved receipt. A parked run (which still reads status: running) or any unreadable output counts as a foreign owner and blocks, which also replaces the copied status vocabulary.
  • validation-supervision now names firstmate's recovery for "run was not started by this task". If it is the same task, point the worker at its saved receipt. If it belongs to another live task or holds an open decision, leave it alone. Only a confirmed orphan with no open decision gets no-mistakes axi abort --run <id>.
  • The bounded-load test now covers gerrit, and each loop is bounded to five seconds.
  • The verification record's restart heading now says the guidance comes from help text. The receipt file stays in the task data directory, the same carve-out as the intent file (low finding, no change).

Implemented with Claude Opus 5.5 (claude-opus-5-5).

**Story:** FIRSTMATE-34 - Parallel validation Lets more than one change in the same repository go through no-mistakes validation at once, by binding every gate answer to the worker's own run, and keeps validation load proofs bounded. ## Investigation Proven on 2026-10-20 with two concurrent real runs on two branches of a disposable scratch repository (ejected afterwards); the full commands and output are in `docs/verification/no-mistakes-branch-isolation.md`. - The shared daemon already runs v1.84.0: it was restarted at 2026-10-20 22:11 UTC, its `/proc/<pid>/exe` hash matches the installed v1.84.0 binary, and every run since records `no_mistakes_version=v1.84.0`. The v1.53.0 premise is out of date, so no restart is needed. - On v1.84.0, bare `axi status` and `axi respond` resolve from the caller's checked-out branch. Each worktree's status showed only its own run. A worktree with no run of its own, the main checkout, and a detached HEAD all got `error: no active run to respond to` while another branch's run sat at a gate. `respond` from the second run's worktree answered only that run, and the first run's gate stayed untouched. - `axi respond` has no `--run` selector, so the checked-out branch is the only binding. `axi status --run <id>` exists for deliberate reads. - v1.53.0 could not be re-probed live (no binary left). The run store shows 188 same-repository, different-branch overlapping runs on v1.53.0, so concurrent runs were always possible there. The 2026-09-09 crossing matches a branch with no run being shown another branch's run, which v1.84.0 now refuses explicitly. - Daemon restart: on v1.84.0, `daemon restart`, `daemon stop`, and `update` refuse while any run is active unless `--force` is passed. The run store holds 19 runs that failed with `daemon shutting down` from forced stops on older daemons. Procedure: confirm no run is active in any repository, then run `no-mistakes daemon restart` without `--force`, then confirm the running binary by hash. A forced restart while runs are live is not safe. ## Changes - `bin/fm-dod-lib.sh` (the generated no-mistakes contract, for every forge): the worker issues every `axi` call from its own copy with its ship branch checked out, records the run id that its starting drive call returns, reads status with `--run <id>`, runs a bare `axi status` immediately before every respond, and answers only when the id, the branch, and the awaiting step all match, passing `--step` explicitly. A gate or a firstmate decision whose `nm-<run>-<step>` key names another run is refused with a `blocked:` run-mismatch line. - The same contract adds the bounded-load rule: load proofs use at most two short-lived `nice`d loops, never one per core, and measure CPU time instead of generating load wherever that can show the same thing. A test-step gate that shows host-saturating load gets `--action fix`, never approve. - `.agents/skills/validation-supervision/SKILL.md`: concurrent no-mistakes runs on different branches of one repository need no per-repository serialization on v1.84.0 and later. Firstmate's decision relays now also name the run id. - `docs/verification/no-mistakes-branch-isolation.md`: a new maintainer-verification record, classified in `docs/documentation-audiences.json`. - `tests/fm-brief.test.sh`: tests that the binding and the bounded-load rule render on every forge and stay out of pipeline-free briefs. A mutation that drops the refusal fails the test. ## Verification - `tests/fm-brief.test.sh`, `tests/fm-task-delivery.test.sh`, `tests/fm-dod-lib.test.sh`, `tests/fm-ask-user-authority.test.sh`, `tests/fm-classify-corr-token.test.sh`, `tests/fm-spawn-dispatch-profile.test.sh`, `tests/fm-pr-check-security.test.sh`, `tests/fm-control-relaunch.test.sh`, `tests/fm-tangle-guard.test.sh`, and `tests/fm-subagent-pretool-check.test.sh` all pass. - `bin/fm-lint.sh` and `bin/fm-doc-audience-check.sh` are clean. ## Follow-up (out of scope) - The private fleet records still say to serialize one pipeline per repository (`data/learnings.md` "One repo = one pipeline", and the `nomistakes-pipeline-is-repo-scoped` memory). Firstmate should retire them once this lands. - The pipeline's own Test-step agent follows each project's trusted `.no-mistakes.yaml` `test.instructions`, which a worker brief cannot reach. Adding the bounded-load sentence there (Flow first) would bound the pipeline agent directly. - A spawn-time check of the daemon version (refuse concurrent runs below v1.84.0) was not added, because no such gate existed and the run binding refuses a crossed gate on any version. ## Review 1 fix: recycled branch A recycled copy can still carry an earlier task's active run on the new task's ship branch. Re-probed on v1.84.0 in a scratch repo (evidence now in the verification record): a plain `axi run --intent` there silently reattaches to the earlier run and ignores the new intent, while a launch with a fresh `--launch-nonce` creates a new run and cancels the earlier one (`superseded by new push`). The contract now proves ownership. Before starting, the worker refuses (`blocked`) when its branch has an active run it did not create. Every new run starts with `--launch-nonce <task-id>-<epoch> --validation-generation <task-id>`. The worker drives a run only when its `launch_receipt` shows `disposition: created`, its own ship branch, and `submitted_head_sha` equal to its HEAD, and it saves that receipt in its task data directory. Every reattach must return that run id. `tests/fm-brief.test.sh` adds a recycled-branch regression on every forge, and a mutation that drops the refusal fails it. ## Review 2 fixes - The pre-start check is now an allowlist of three safe status shapes: `runs_on_current_branch: 0`, a terminal run (`completed`, `failed`, `cancelled`), or the run id in the worker's own saved receipt. A parked run (which still reads `status: running`) or any unreadable output counts as a foreign owner and blocks, which also replaces the copied status vocabulary. - `validation-supervision` now names firstmate's recovery for "run was not started by this task". If it is the same task, point the worker at its saved receipt. If it belongs to another live task or holds an open decision, leave it alone. Only a confirmed orphan with no open decision gets `no-mistakes axi abort --run <id>`. - The bounded-load test now covers gerrit, and each loop is bounded to five seconds. - The verification record's restart heading now says the guidance comes from help text. The receipt file stays in the task data directory, the same carve-out as the intent file (low finding, no change). Implemented with Claude Opus 5.5 (claude-opus-5-5).
No-mistakes v1.84.0 resolves axi status and respond from the caller's
checked-out branch, proven with two concurrent runs in a scratch
repository. The worker contract now binds every respond to the worker's
own run id and branch and refuses a foreign gate, so concurrent runs in
one repository need no per-repository serialization. Validation load
proofs stay bounded and prefer CPU-time measurement.
Author
Owner

Adversarial review 1 of 2 - REQUEST CHANGES

Reviewer: OpenAI gpt-6-luna (Codex, high effort)
Reviewed head: a0c10b5440

  • High — bin/fm-dod-lib.sh:356-358; docs/verification/no-mistakes-branch-isolation.md:109. A recycled worktree can leave an active run on the task's ship branch. The generated contract assumes the ID returned by no-mistakes axi run belongs to this task, then accepts a current-branch status with that ID and branch. The installed axi run --help documents reattachment to an active run, while axi respond --help has no run selector. A new task on the reused branch can therefore reattach to the previous task's run, pass the ID/branch check, and answer its gate. The check proves branch association, not task/run ownership. Detached HEAD and no-run branches are covered by the verification record, but a prior active run on the same branch is not. Require proof that the run was newly created for this task and fail closed on reattachment, or retain serialization until respond can target the verified run ID; add a recycled-worktree regression or CLI-level proof.

Validation on the live head: bash tests/fm-dod-lib.test.sh and bash tests/fm-brief.test.sh passed; shellcheck -x bin/fm-dod-lib.sh passed. No load generator or real no-mistakes run was started. Full evidence is in /home/john/code/firstmate/data/rev-fm51-a1/report.md.

Adversarial review 1 of 2 - REQUEST CHANGES Reviewer: OpenAI gpt-6-luna (Codex, high effort) Reviewed head: a0c10b5440eb86fed9c0122353f8920f4b3bda63 - **High — `bin/fm-dod-lib.sh:356-358`; `docs/verification/no-mistakes-branch-isolation.md:109`.** A recycled worktree can leave an active run on the task's ship branch. The generated contract assumes the ID returned by `no-mistakes axi run` belongs to this task, then accepts a current-branch status with that ID and branch. The installed `axi run --help` documents reattachment to an active run, while `axi respond --help` has no run selector. A new task on the reused branch can therefore reattach to the previous task's run, pass the ID/branch check, and answer its gate. The check proves branch association, not task/run ownership. Detached HEAD and no-run branches are covered by the verification record, but a prior active run on the same branch is not. Require proof that the run was newly created for this task and fail closed on reattachment, or retain serialization until `respond` can target the verified run ID; add a recycled-worktree regression or CLI-level proof. Validation on the live head: `bash tests/fm-dod-lib.test.sh` and `bash tests/fm-brief.test.sh` passed; `shellcheck -x bin/fm-dod-lib.sh` passed. No load generator or real no-mistakes run was started. Full evidence is in `/home/john/code/firstmate/data/rev-fm51-a1/report.md`.
A recycled copy can still carry an earlier task's active run on the new
task's ship branch. A plain axi run --intent there silently reattaches to
that run and a fresh launch cancels it, so branch-matching alone could let
the new task answer the earlier task's gate. The worker now refuses to
start over an active run it did not create, starts every run with a
task-bound launch nonce, and drives only a run whose receipt shows
disposition created at its own HEAD.
Author
Owner

Adversarial review 2 of 2 - PASS

Reviewer: Qwen 3.8 via AIRouter
Reviewed head: 4cc7a4971d

Independent of review 1; verdict formed at the live head (which moved to this sha while this review was in flight).

  • Medium - bin/fm-dod-lib.sh:361-362. The pre-flight foreign-run refusal keys off the worker parsing a status word ("pending, running, fixing, or ci"), while the failure mode this rule exists for - a prior task's run PARKED at a gate - reports top-level status: running (see tests/captures/no-mistakes-v1.70.1/parked.toon). One misread of that bare status TOON (truncation, help-line noise) lets the fresh-nonce launch through, and per this PR's own verification record the launch then cancels the foreign parked gate ("superseded by new push"), possibly one holding open ask-user decisions. The disposition: created + submitted_head_sha == HEAD receipt check does not save it: a parked run at the same head passes both. Suggest refusing structurally on runs_on_current_branch > 0 instead of a parsed status word.
  • Medium - no owner recovery path for the new blocked: ... was not started by this task state. Neither validation-supervision nor the stuck-worker playbook names what firstmate does: confirm-orphan then axi abort --run <id> and relaunch, vs. the run belongs to a live task and the slot goes back to it. The first real hit of this guard (which the recycled-copy shape makes likely) becomes an ad-hoc 3 a.m. decision. One paragraph in the skill would close it.
  • Medium - gerrit forge is absent from test_no_mistakes_dod_bounds_test_load (loops none forgejo; the sibling binding and recycled-run tests loop all three). The bounded-load paragraph is forge-independent today, so this is coverage symmetry, not a defect - one line to add.
  • Low - receipt file lives in $FM_HOME/data/$id/, outside the worktree - same established carve-out as nm-intent.txt, and it survives teardown (good for the F2 recovery path), but it is a second instance of a writable-outside-worktree file the brief's rule 2 does not name. No action; noted so a later "fix" of one does not orphan the other.
  • Nits: em dashes in the new verification doc (repo style is plain dash); the "Daemon restart" section heading reads like an observed result although line 177 now correctly frames the refusal as a help-text inference; "two short-lived loops" in the bounded-load rule is not machine-checkable (bound each loop by wall time instead); the "pending, running, fixing, or ci" list in the rendered contract is a third copy of the executing vocabulary owned by bin/fm-crew-state.sh / bin/fm-nm-run-lib.sh (would disappear with the F1 structural fix).

Verified: bash tests/fm-brief.test.sh and bash tests/fm-dod-lib.test.sh pass (including the three new tests); bin/fm-lint.sh full set passes (ShellCheck 0.11.0, actionlint 1.7.12); the binding block and bounded-load rule render verbatim into a real scaffolded no-mistakes brief with correct $id/data-dir substitution and stay out of direct-PR/local-only briefs; axi respond --help (no run selector), axi run --help (launch receipt: run id, created/reused disposition, submitted head, intent digest), and the daemon lifecycle --force help text all match the verification record; the live shared daemon is the v1.84.0 binary the record tested (sha256 of /proc//exe vs ~/.no-mistakes/bin/no-mistakes); the 19 "daemon shutting down" failed runs are exactly 19 in the local run store (11 NULL-version, 8 v1.53.0); detached-HEAD and run-less-branch refusals were re-probed live on the shared daemon in a throwaway scratch repo (rc=1, current_branch: unknown / runs_on_current_branch: 0) matching the record; no remaining firstmate text demands one run per repository (AGENTS.md 195-196 is the pre-existing general semantic-dependency rule). No load generator and no real pipeline run was started; two read-only CLI probes ran in a disposable scratch repository, ejected afterward. No high findings; the respond path itself is sound - the residual exposure is a single misread on the pre-flight check, not in respond. Full evidence in /home/john/code/firstmate/data/rev-fm51-a2/report.md.

Adversarial review 2 of 2 - PASS Reviewer: Qwen 3.8 via AIRouter Reviewed head: 4cc7a4971d7bbac81a5f9a7f664b19958fcdabec Independent of review 1; verdict formed at the live head (which moved to this sha while this review was in flight). - **Medium - `bin/fm-dod-lib.sh:361-362`.** The pre-flight foreign-run refusal keys off the worker parsing a status word ("pending, running, fixing, or ci"), while the failure mode this rule exists for - a prior task's run PARKED at a gate - reports top-level `status: running` (see tests/captures/no-mistakes-v1.70.1/parked.toon). One misread of that bare `status` TOON (truncation, help-line noise) lets the fresh-nonce launch through, and per this PR's own verification record the launch then cancels the foreign parked gate ("superseded by new push"), possibly one holding open ask-user decisions. The `disposition: created` + `submitted_head_sha == HEAD` receipt check does not save it: a parked run at the same head passes both. Suggest refusing structurally on `runs_on_current_branch > 0` instead of a parsed status word. - **Medium - no owner recovery path for the new `blocked: ... was not started by this task` state.** Neither validation-supervision nor the stuck-worker playbook names what firstmate does: confirm-orphan then `axi abort --run <id>` and relaunch, vs. the run belongs to a live task and the slot goes back to it. The first real hit of this guard (which the recycled-copy shape makes likely) becomes an ad-hoc 3 a.m. decision. One paragraph in the skill would close it. - **Medium - gerrit forge is absent from `test_no_mistakes_dod_bounds_test_load`** (loops `none forgejo`; the sibling binding and recycled-run tests loop all three). The bounded-load paragraph is forge-independent today, so this is coverage symmetry, not a defect - one line to add. - **Low - receipt file lives in `$FM_HOME/data/$id/`, outside the worktree** - same established carve-out as `nm-intent.txt`, and it survives teardown (good for the F2 recovery path), but it is a second instance of a writable-outside-worktree file the brief's rule 2 does not name. No action; noted so a later "fix" of one does not orphan the other. - **Nits:** em dashes in the new verification doc (repo style is plain dash); the "Daemon restart" section heading reads like an observed result although line 177 now correctly frames the refusal as a help-text inference; "two short-lived loops" in the bounded-load rule is not machine-checkable (bound each loop by wall time instead); the "pending, running, fixing, or ci" list in the rendered contract is a third copy of the executing vocabulary owned by bin/fm-crew-state.sh / bin/fm-nm-run-lib.sh (would disappear with the F1 structural fix). Verified: `bash tests/fm-brief.test.sh` and `bash tests/fm-dod-lib.test.sh` pass (including the three new tests); `bin/fm-lint.sh` full set passes (ShellCheck 0.11.0, actionlint 1.7.12); the binding block and bounded-load rule render verbatim into a real scaffolded no-mistakes brief with correct `$id`/data-dir substitution and stay out of direct-PR/local-only briefs; `axi respond --help` (no run selector), `axi run --help` (launch receipt: run id, created/reused disposition, submitted head, intent digest), and the daemon lifecycle `--force` help text all match the verification record; the live shared daemon is the v1.84.0 binary the record tested (sha256 of /proc/<pid>/exe vs ~/.no-mistakes/bin/no-mistakes); the 19 "daemon shutting down" failed runs are exactly 19 in the local run store (11 NULL-version, 8 v1.53.0); detached-HEAD and run-less-branch refusals were re-probed live on the shared daemon in a throwaway scratch repo (rc=1, `current_branch: unknown` / `runs_on_current_branch: 0`) matching the record; no remaining firstmate text demands one run per repository (AGENTS.md 195-196 is the pre-existing general semantic-dependency rule). No load generator and no real pipeline run was started; two read-only CLI probes ran in a disposable scratch repository, ejected afterward. No high findings; the respond path itself is sound - the residual exposure is a single misread on the pre-flight check, not in respond. Full evidence in /home/john/code/firstmate/data/rev-fm51-a2/report.md.
The pre-start check now starts a run only from three structural status
shapes (no run on the branch, a terminal run, or the worker's own saved
receipt run), so a parked run or an unreadable status reads as a foreign
owner. The validation-supervision skill names firstmate's recovery for a
run the worker did not start. The bounded-load rule bounds each loop to
five seconds and its test now covers gerrit.
Author
Owner

Adversarial re-review - PASS

Reviewer: OpenAI gpt-6-luna (Codex, high effort)
Head: 7bc041cb9e
PR: #51

No high or medium findings remain. The live-head fm-brief suite passed; shellcheck -x bin/fm-dod-lib.sh passed; and no-mistakes v1.84.0 help confirms launch receipts, run-selected status, and branch-bound respond with no run selector. The generated contract now refuses an active run without this task's created receipt, the validation skill owns recovery for an orphaned run, and the bounded-load test covers Gerrit. The report records the evidence and the limit that I did not repeat a live scratch run because it would persist scratch-run state through the shared daemon outside the task's allowed write paths.

Adversarial re-review - PASS Reviewer: OpenAI gpt-6-luna (Codex, high effort) Head: 7bc041cb9e06e437452bbe0fb75b1c9cbce04485 PR: https://git.pyr0.dev/john/firstmate/pulls/51 No high or medium findings remain. The live-head `fm-brief` suite passed; `shellcheck -x bin/fm-dod-lib.sh` passed; and no-mistakes v1.84.0 help confirms launch receipts, run-selected status, and branch-bound respond with no run selector. The generated contract now refuses an active run without this task's created receipt, the validation skill owns recovery for an orphaned run, and the bounded-load test covers Gerrit. The report records the evidence and the limit that I did not repeat a live scratch run because it would persist scratch-run state through the shared daemon outside the task's allowed write paths.
john merged commit 5d3d780da3 into main 2026-10-21 22:51:09 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
john/firstmate!51
No description provided.