Atomic write-then-rename for the landing handoff #15

Merged
john merged 13 commits from fm/fm-quota-landing-procedure into main 2026-08-21 21:03:27 +00:00
Owner

Replaces the size-stability handoff check with an atomic write-then-rename, so a partially written file can never be mistaken for a completed handoff.

Validated through the full no-mistakes pipeline: run 01M0FX639JSVPJ3JTVY0E0EWMX, outcome passed, one auto-fix and one informational finding, nothing awaiting a decision.

Firstmate verified the mechanism by reading the code rather than trusting the report. All three required proofs exist as named tests, including test_fast_atomic_handoff_is_accepted_without_observing_the_temp - the false-negative case firstmate own design got wrong and the reviewer caught. The worker also added an edge case firstmate had not considered: test_marker_quoted_in_the_body_is_not_a_completion_signal, so a handoff that merely DISCUSSES the completion marker cannot falsely signal completion.

Replaces the size-stability handoff check with an atomic write-then-rename, so a partially written file can never be mistaken for a completed handoff. Validated through the full no-mistakes pipeline: run 01M0FX639JSVPJ3JTVY0E0EWMX, outcome passed, one auto-fix and one informational finding, nothing awaiting a decision. Firstmate verified the mechanism by reading the code rather than trusting the report. All three required proofs exist as named tests, including `test_fast_atomic_handoff_is_accepted_without_observing_the_temp` - the false-negative case firstmate own design got wrong and the reviewer caught. The worker also added an edge case firstmate had not considered: `test_marker_quoted_in_the_body_is_not_a_completion_signal`, so a handoff that merely DISCUSSES the completion marker cannot falsely signal completion.
john added 13 commits 2026-08-21 20:24:07 +00:00
Build the landing sequence the captain named: at the Claude session-window
ceiling, hand off each worker, write the durable record, clear context, close
the worker, push its committed branch, and wait for the reset. Detection
already existed (state/fm-session-budget-watch.check.sh); this adds what
happens next.

- bin/fm-land.sh: the landing procedure. Steers each session-quota worker
  (Claude or Ollama; Pi excluded - soft ceiling) to write a handoff to
  data/<id>/handoff.md, verifies it landed on disk, pushes the committed branch
  to its remote before /clear (a committed-but-unpushed branch in a pooled
  worktree is one machine failure from lost work; teardown does not refuse
  it), writes the state/.fm-landed marker, and launches a detached reset
  sentinel. Push failure is fail-loud: recorded under PUSH FAILURES and
  non-zero exit so firstmate does not treat the landing as complete.
- bin/fm-quota-reset-sentinel.sh: the no-turn wake backstop. Detached, polls
  quota-axi with no dependency on a model turn, a live agent, or a turn-end
  hook. On reset it enqueues a durable signal wake (the survives-no-agent
  guarantee - pure file I/O) plus a best-effort pane injection, then exits.
  Does not re-arm the watcher (that is the turn-end hook's job, and re-arming
  is useless with no live firstmate to deliver the wake).
- bin/fm-operational-input.sh: add the quota-reset operational-input kind so a
  sentinel-originated wake is structurally typed and marked-internal (does
  not exit away mode), not confused with captain input.
- AGENTS.md section 8: a Quota-ceiling landing stub pointing at bin/fm-land.sh
  as the procedure owner, with the inline facts a fresh session needs (what
  the budget wake triggers, what a quota-reset wake / .fm-landed marker means,
  the seam with fm-modal-aware-recovery).
- tests/fm-land.test.sh (6 cases) and tests/fm-quota-reset-sentinel.test.sh
  (5 cases): drive the machinery with mocks. The sentinel is proven with NO
  agent alive at all - the durable wake lands on disk with no firstmate
  process, no tmux, no turn hook. A fresh-session step reads only the queued
  wake + marker and confirms a resume is possible. Tests fail if the feature
  is removed.
- tests/fm-operational-input.test.sh: cover the new quota-reset kind.

Scope: absorbs nothing. fm-modal-aware-recovery (recovering an already
modal-parked worker), fm-wedge-alarm-escalation, and fm-afk-return-watcher-
handoff stay separate; this task owns not reaching the wall.
Two follow-ups the captain judged inside the accepted contract.

1. Sentinel no-live-firstmate audit (the harder defect). Dropping ensure_watcher
   fixed one instance, but the real defect was broader: the sentinel made a
   synchronous call into a component whose completion depends on a live
   firstmate. Audit EVERY external call and answer "does this terminate without
   a live firstmate?":
   - quota-axi --json: NO (network). Now runs under a hard QUOTA_TIMEOUT
     (fm_run_bounded: timeout -> gtimeout -> perl-alarm; refuses unbounded if
     no tool). A timeout is logged loudly and treated as an unreadable meter
     (skip the cycle, never fire a reset wake).
   - tmux/herdr calls (target_exists, composer_state, send_text_submit,
     pane_is_busy): cannot be PROVEN to terminate (wedged tmux). Now all run
     inside inject_tmux under fm_fn_bounded INJECT_TIMEOUT (background+kill
     bound, since shell functions cannot be exec'd by timeout). Fail loud.
   - python3, fm_wake_append (file I/O + bounded lock), env reads,
     fm_operational_input_encode, sleep, date: terminate unaided. Unchanged.
   - re-arming the watcher: already removed (blocks on a watcher wake, and a
     re-armed watcher has no delivery channel under Claude with no live firstmate).
   Proof: a new test runs the sentinel with a quota-axi that hangs 60s; with
   QUOTA_TIMEOUT=2 the sentinel hard-timeout-kills it, logs, and COMPLETES
   instead of hanging - the exact defect the 21-minute hang exposed.

2. Missing worktree check. fm-land.sh now verifies the recorded worktree resolves
   to its own git toplevel (git -C <worktree> rev-parse --show-toplevel vs the
   recorded path, realpath-compared) and refuses on mismatch, naming the actual
   mismatch (recorded=X actual=Y). A refused worker is not handed off, not
   pushed, not /cleared; its status records a refused: line and the marker lists
   it under REFUSALS; fm-land.sh exits non-zero. Proven RED against a
   deliberately drifted worker (recorded subdir != real toplevel) and GREEN
   against a correct worktree.
   Honest limit recorded in notes: this catches a mis-recorded/mis-resolved
   worktree (subdir, symlink, deleted, moved) but NOT the pure cd-drift the
   author committed (worker cd'd to a different checkout while the recorded
   worktree stayed intact) - fm-land.sh operates on the recorded path and
   cannot see the worker's shell. That cd-drift remains an open gap.

tests/fm-land.test.sh: +2 cases (drifted-worktree RED, correct-worktree GREEN).
tests/fm-quota-reset-sentinel.test.sh: +1 case (hanging quota-axi hard-timeout).
fm-lint.sh clean; fm-test-run.sh --check-coverage ok (total=144).
The size-stability heuristic could not separate "complete" from
"truncated-but-stable": a worker that wrote a partial handoff and then died or
wedged leaves a stable partial file that any size check accepts - and a worker
dying mid-write is our actual failure mode (three died mid-task to a route
outage in one night). A guard that reports success on the failure it exists for
is worse than not having it: it converts a visible loss into an invisible one.
The fix removes the heuristic entirely.

- The worker is now steered to write the handoff to <path>.tmp and atomically
  rename it into <path> on completion. Rename is atomic on POSIX, so <path>
  existing IS the completion signal - no size heuristic is needed, and a
  partial write never reaches the final path (it stays in the temp).
- await_atomic_handoff waits for the atomic protocol: captured = final present,
  temp gone, AND the temp was observed at least once (the rename protocol was
  actually followed - this is what rejects a partial file written directly to
  the final path, which appears without any temp). unverified = final present
  but temp never observed (direct write, cannot verify complete). incomplete =
  temp still present at the deadline (worker died/wedged mid-write). missing =
  neither present.
- archive_stale_handoff now also archives a leftover temp from a dead worker so
  it cannot poison the next capture.
- Verdict wording corrected to say what the code actually knows: INCOMPLETE =
  the write never completed (temp still present), UNVERIFIED = a file appeared
  at the final path without the atomic-write temp, so completeness could not be
  verified. The old "so it is truncated" overclaim is gone - asserting a fact
  the code cannot know is the same defect class in prose.

Tests prove it RED both ways, and the RED proof is itself proven: reverting
await_atomic_handoff to the old size-stability logic makes RED A fail (the old
heuristic accepts a partial file left at the final path) - the exact case that
was being silently accepted. With the atomic logic: RED A (partial file at the
final path) is rejected, not cleared, still pushed; RED B (worker killed
mid-write) leaves NO file at the final path, records INCOMPLETE, pane not
cleared, branch still pushed; GREEN (temp then rename) is captured and the pane
is cleared. Full fm-land suite (28), sentinel suite, and fm-lint all pass.
no-mistakes(document): document atomic handoff capture contract in landing section
All checks were successful
CI / Test coverage guard (pull_request) Successful in 12s
CI / Behavior portable parallel 1 (pull_request) Successful in 3m15s
CI / Lint shell scripts (pull_request) Successful in 4m28s
CI / Behavior portable parallel 2 (pull_request) Successful in 3m1s
CI / Behavior portable serial 4 (pull_request) Successful in 9m3s
CI / Behavior portable serial 2 (pull_request) Successful in 11m22s
CI / Repo invariants (pull_request) Successful in 4s
CI / Behavior portable serial 1 (pull_request) Successful in 15m51s
CI / Behavior tests (Herdr) (pull_request) Successful in 7m25s
CI / Behavior portable serial 3 (pull_request) Successful in 17m18s
226e26c433
john merged commit 8c10e2827a into main 2026-08-21 21:03:27 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
john/firstmate!15
No description provided.