Refuse teardown when a different live task owns the worktree #23

Merged
john merged 2 commits from fm/fm-teardown-worktree-ownership-guard into main 2026-08-30 00:13:15 +00:00
Owner

What was broken, and what it cost

bin/fm-teardown.sh guarded the finished task's unlanded work, but never checked whether a different, live task occupied the worktree named in that finished task's metadata. Treehouse worktrees are pooled and recycled, so that record goes stale the moment a slot is reissued.

Proven live on 2026-08-29, destructively. Four merged Conduitmail tasks were torn down to quiet a noisy digest. Three of them still recorded pooled slots that cond-auth-provider-module and cond-forms-redirect-contract were at that moment using. Teardown killed one worker's endpoint outright and hard-reset both trees mid-run — including a commit that had just applied three captain rulings, with an active validation run against it.

The teardown refusal is the only protection unlanded work has, and it inspected the wrong task entirely. It reported success while destroying running work. Two of the six candidates happened to refuse on "uncommitted changes present" — but that fired for the recycled slot's current contents, which is accidental protection, not the guard working.

Recovery was luck: the validation gate's own repository still held the branch.

What this changes

An ownership check runs before any destructive step. If a different live task owns the worktree, teardown refuses and names that task, so the operator can act rather than guess.

--force alone does not bypass it. A separate --discard-live-owner is required, and must match. That separation is deliberate: this is a different hazard from "this task's own work is unlanded", and sharing an escape hatch would mean someone forcing past the familiar refusal and silently getting this one too.

A stale slot — where the recorded worktree no longer belongs to that task at all — clears the task's state without touching the tree.

Tests

Six, colocated. The two that decide whether this guard is worth having:

  • live-owner refusal, which fails against current code — the scenario that actually happened;
  • own-pane still allows — because a guard that also refuses a task's own worker blocks every legitimate cleanup, which is how safety features get switched off.

Plus: --force alone refuses, a matching discard overrides, a mismatched discard refuses, and a stale slot clears state without touching the tree.

Note

Since the incident, firstmate has been running this check by hand before every teardown — reading live panes, comparing recorded worktrees, refusing anything occupied. That manual procedure has been safe through six teardowns and is what this encodes. Cleanup has otherwise been suspended all day, which is why finished work has been re-announcing itself in every digest.

Pipeline passed with zero findings across review, test, document and lint. Pushed at 8d76eb93.

## What was broken, and what it cost `bin/fm-teardown.sh` guarded the **finished** task's unlanded work, but never checked whether a **different, live** task occupied the worktree named in that finished task's metadata. Treehouse worktrees are pooled and recycled, so that record goes stale the moment a slot is reissued. **Proven live on 2026-08-29, destructively.** Four merged Conduitmail tasks were torn down to quiet a noisy digest. Three of them still recorded pooled slots that `cond-auth-provider-module` and `cond-forms-redirect-contract` were *at that moment* using. Teardown killed one worker's endpoint outright and hard-reset **both** trees mid-run — including a commit that had just applied three captain rulings, with an active validation run against it. The teardown refusal is the only protection unlanded work has, and it inspected the wrong task entirely. It reported success while destroying running work. Two of the six candidates happened to refuse on "uncommitted changes present" — but that fired for the *recycled slot's current contents*, which is accidental protection, not the guard working. Recovery was luck: the validation gate's own repository still held the branch. ## What this changes An ownership check runs **before any destructive step**. If a different live task owns the worktree, teardown refuses and **names that task**, so the operator can act rather than guess. `--force` alone does **not** bypass it. A separate `--discard-live-owner` is required, and must match. That separation is deliberate: this is a different hazard from "this task's own work is unlanded", and sharing an escape hatch would mean someone forcing past the familiar refusal and silently getting this one too. A stale slot — where the recorded worktree no longer belongs to that task at all — clears the task's state **without touching the tree**. ## Tests Six, colocated. The two that decide whether this guard is worth having: - **live-owner refusal**, which fails against current code — the scenario that actually happened; - **own-pane still allows** — because a guard that also refuses a task's *own* worker blocks every legitimate cleanup, which is how safety features get switched off. Plus: `--force` alone refuses, a matching discard overrides, a mismatched discard refuses, and a stale slot clears state without touching the tree. ## Note Since the incident, firstmate has been running this check **by hand** before every teardown — reading live panes, comparing recorded worktrees, refusing anything occupied. That manual procedure has been safe through six teardowns and is what this encodes. Cleanup has otherwise been suspended all day, which is why finished work has been re-announcing itself in every digest. Pipeline passed with zero findings across review, test, document and lint. Pushed at `8d76eb93`.
Treehouse worktrees are pooled and recycled, so a finished task's
state/<id>.meta can go stale the moment its slot is reissued: the recorded
worktree path then points at a DIFFERENT live task's work, and teardown's
hard-reset/return would destroy it. The finished task's own landed-work check
inspects the RECORDED worktree's current contents - the wrong task's work -
and can report success while destroying running work (observed 2026-08-29:
tearing down four merged tasks to quiet a noisy away-mode digest destroyed two
live workers' trees mid-run, killing one's active no-mistakes run).

Encode the manual ownership check firstmate ran by hand ever since: before any
destructive step, enumerate every OTHER task meta whose recorded worktree
resolves to the same canonical dir and refuse when one of them is LIVE, naming
that live task. A pane in the candidate's own worktree that belongs to this
same task is not a refusal - only a different live task's claim is.

A non-live other claim means the slot is stale: clear the task's state without
touching the tree (skip run abort, process reap, branch delete, and treehouse
return) while endpoint kill and state cleanup still run.

--force does not bypass the live-ownership refusal on its own - it is a
different hazard from 'this task's own work is unlanded'. Add
--discard-live-owner <task-id>, which must name the actual live owner and
pairs with --force, so the operator accepts discarding that live task's work.

Tests stand up two tasks sharing one pooled slot against a real tmux server
with a fake agent process (comm=claude) so fm_backend_agent_state reports it
alive: the landed task's teardown refuses and leaves the live tree untouched
(fails against current code), teardown still succeeds when the only live pane
is the task's own, --force alone does not bypass, --force
--discard-live-owner=<matching> overrides and --discard-live-owner=<wrong>
does not, and a stale (non-live) other claim clears state without touching the
tree.
no-mistakes(document): docs: note live-worktree ownership guard in teardown architecture
All checks were successful
CI / Test coverage guard (pull_request) Successful in 13s
CI / Behavior portable parallel 2 (pull_request) Successful in 3m4s
CI / Behavior portable parallel 1 (pull_request) Successful in 3m12s
CI / Lint shell scripts (pull_request) Successful in 4m35s
CI / Behavior portable serial 1 (pull_request) Successful in 13m45s
CI / Behavior portable serial 4 (pull_request) Successful in 11m10s
CI / Repo invariants (pull_request) Successful in 4s
CI / Behavior portable serial 3 (pull_request) Successful in 13m39s
CI / Behavior portable serial 2 (pull_request) Successful in 14m20s
CI / Behavior tests (Herdr) (pull_request) Successful in 7m30s
8d76eb939a
john merged commit 6eaab43975 into main 2026-08-30 00:13:15 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
john/firstmate!23
No description provided.