john/firstmate
Watch
1
0
Fork
You've already forked firstmate
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

PYR0D-394: Fix away-mode worker ceiling and Codex session-lock anchor #45

Closed
john wants to merge 8 commits from fm/fm-dispatch-fix into main
pull from: fm/fm-dispatch-fix
merge into: john:main
Conversation 2 Commits 8 Files changed 9 +1058 -433
john commented 2026-10-13 01:54:40 +00:00
Owner
Copy link

Story: PYR0D-394 - Fix away-mode worker ceiling and Codex session-lock anchor

What was wrong (the Oct-9/12 outage)

Two firstmate defects compounded overnight: (1) the away-mode worker ceiling counted stale in-flight records - 14 dead windows against a cap of 4 - so nothing dispatched; (2) the session lock anchored to the immortal codex app-server --managed-daemon pid, so every new session was refused for 15+ hours until the stale anchor was reclaimed by hand.

What changed

  • Away cap counts only live workers (red test 17613fea, fix 024b83ec; incident reproduction 14 -> 4).
  • Codex app-server daemon excluded from harness-pid liveness, so a daemon-anchored lock is stale-reclaimable; anchor selection documents the app-server-hosted case (red tests ccb1f6c5, fix 4bc4b5b8; live daemons verified reclaimable).
  • Docs: abb4e570, a7ea9469.

Known trade-off (accepted in review)

A daemon-anchored lock is reclaimable even while its conversation may still be live: the displaced session fails its next ownership/Stop check and stays read-only. Codex conversations have no process of their own, so conversation liveness is not checkable without coupling the lock library to codex internals, which the accepted intent does not require.

Validation

no-mistakes run 01M4SA9HC3CRVAKPZC39BWGZ3B passed (one review finding accepted as the documented trade-off above; the pipeline PR/CI steps skip on Forgejo). Full local suite: the 18 failures reproduce identically on clean main bins (environmental, pre-existing); both incident reproductions proven red before and green after.

**Story:** PYR0D-394 - Fix away-mode worker ceiling and Codex session-lock anchor ## What was wrong (the Oct-9/12 outage) Two firstmate defects compounded overnight: (1) the away-mode worker ceiling counted stale in-flight records - 14 dead windows against a cap of 4 - so nothing dispatched; (2) the session lock anchored to the immortal `codex app-server --managed-daemon` pid, so every new session was refused for 15+ hours until the stale anchor was reclaimed by hand. ## What changed - Away cap counts only live workers (red test `17613fea`, fix `024b83ec`; incident reproduction 14 -> 4). - Codex app-server daemon excluded from harness-pid liveness, so a daemon-anchored lock is stale-reclaimable; anchor selection documents the app-server-hosted case (red tests `ccb1f6c5`, fix `4bc4b5b8`; live daemons verified reclaimable). - Docs: `abb4e570`, `a7ea9469`. ## Known trade-off (accepted in review) A daemon-anchored lock is reclaimable even while its conversation may still be live: the displaced session fails its next ownership/Stop check and stays read-only. Codex conversations have no process of their own, so conversation liveness is not checkable without coupling the lock library to codex internals, which the accepted intent does not require. ## Validation no-mistakes run `01M4SA9HC3CRVAKPZC39BWGZ3B` passed (one review finding accepted as the documented trade-off above; the pipeline PR/CI steps skip on Forgejo). Full local suite: the 18 failures reproduce identically on clean main bins (environmental, pre-existing); both incident reproductions proven red before and green after.
The 2026-10-12 overnight away window refused every dispatch for 8.5
hours: in the away session's environment no liveness or verdict read
could classify any record, so the cap counted all fourteen in-flight
records, every one long finished. Red now; the fix lands beside it.
The away spend cap counted a record as spend whenever neither its
liveness read nor its run-verdict read could classify it, which is the
2026-10-12 overnight window: the away session's environment could not
reach the backend tools, every read answered unverified or unreadable
with a backend-unreachable verdict, and all fourteen in-flight records -
every one long finished - held a cap of 4 for 8.5 hours, so no worker
ran.

When both reads are unreadable, the record's own durable status log is
the remaining evidence: a last status verb only a finished worker writes
(done, failed, landed, landed-failed - a landing parks the worker) frees
the record even then. Any other record still counts under the same
condition, so an uncertain verdict still refuses, and every existing
fail-safe direction of the count survives.
The 2026-10-12 incident: a firstmate session on Codex anchored the
home's lock to the codex app-server daemon, the daemon outlived the
session, and every later session was refused read-only for 15+ hours
until manual reclamation. Red now: the daemon counts as a live owner,
no codex conversation id is trusted, and a dead conversation's lock on
a live daemon is not reclaimable by a new session.
A firstmate session run on Codex anchored the home's lock to the codex
app-server daemon (codex app-server --managed-daemon), which hosts every
codex conversation and outlives each one. Once the session's window
closed, the recorded pid still read as a live harness, every later
session was refused read-only for 15+ hours, and only manual
reclamation freed the home.

Three rules, all in the session-lock lib that owns harness identity:

- A daemon is exactly a codex run whose first positional argument is
  app-server. ps flattens argv, so words alone cannot decide this: a
  worker whose single launch-brief argument mentions app-server must
  stay a session. The daemon still matches the harness name, so the
  ancestry walk finds it, but the walk extends through it like a Claude
  run, because it is never the session itself.
- fm_session_lock_anchor_pid prefers the outermost non-daemon harness
  pid, so a real codex session process above a daemon anchors its own
  lock exactly as before; a daemon-hosted conversation, whose only
  harness ancestor is the daemon, still records the daemon pid on line
  1.
- A daemon-shaped pid is never a live owner (fm_harness_pid_alive), so
  a lock its dead session left is reclaimed through bin/fm-lock.sh's
  ordinary stale-owner path. Mirroring Claude's trusted session id,
  codex sessions prove CODEX_SESSION_ID from a codex-shaped ancestry
  (verified: codex-cli 0.162.1 hands the id to exec'd commands), and
  fm-lock records it beside the lock in state/.lock-session.

An untrusted id, a non-codex ancestry, or no id at all changes nothing:
every fail-safe direction of the lock survives, and a session-shaped
codex process anchors, holds, and refuses a second live session exactly
as before.
The session-lock sidecar now records a trusted codex session id beside
Claude's, and a codex app-server daemon never counts as a live owner,
so the turn-end guard's ownership description gains both facts.
no-mistakes(document): Name the codex id in stale lock docs
Some checks failed
CI / Test coverage guard (pull_request) Successful in 4m43s
Details
CI / Behavior portable parallel 2 (pull_request) Successful in 18m34s
Details
CI / Behavior portable parallel 1 (pull_request) Successful in 26m15s
Details
CI / Lint 1 (pull_request) Successful in 28m45s
Details
CI / Lint 2 (pull_request) Failing after 30m0s
Details
CI / Behavior portable serial 4 (pull_request) Failing after 16m21s
Details
CI / Behavior portable serial 1 (pull_request) Successful in 24m12s
Details
CI / Behavior portable serial 2 (pull_request) Successful in 21m21s
Details
CI / Behavior portable serial 3 (pull_request) Successful in 22m34s
Details
CI / Behavior portable serial 7 (pull_request) Successful in 20m38s
Details
CI / Behavior portable serial 6 (pull_request) Successful in 20m47s
Details
CI / Behavior portable serial 8 (pull_request) Successful in 21m3s
Details
CI / Repo invariants (pull_request) Successful in 8s
Details
CI / Behavior portable serial 5 (pull_request) Failing after 30m0s
Details
CI / Behavior tests (Herdr) (pull_request) Successful in 15m35s
Details
CI / Behavior portable serial 9 (pull_request) Successful in 29m50s
Details
5c1791a3ae
john commented 2026-10-13 04:58:36 +00:00
Author
Owner
Copy link

REQUEST CHANGES. Standalone review: data/review-45-sol/report.md. P1: conversations sharing a Codex app-server PID can both pass lock ownership despite different session IDs, so the displaced session does not fail closed. P2: landed-failed workers are freed from the away cap even though fm-land.sh leaves them uncleared for recovery. The report includes reproductions, gates, and the red forge CI status at 5c1791a3.

REQUEST CHANGES. Standalone review: `data/review-45-sol/report.md`. P1: conversations sharing a Codex app-server PID can both pass lock ownership despite different session IDs, so the displaced session does not fail closed. P2: `landed-failed` workers are freed from the away cap even though `fm-land.sh` leaves them uncleared for recovery. The report includes reproductions, gates, and the red forge CI status at `5c1791a3`.
john closed this pull request 2026-10-13 06:17:55 +00:00
john commented 2026-10-13 06:17:56 +00:00
Author
Owner
Copy link

Closed by captain order 2026-10-13: the Claude-first routing change makes this PR not worth its fix cycle (the P1 same-daemon lock ownership hole needs a real rework). The branch fm/fm-dispatch-fix preserves both fixes (away-cap live-worker counting; codex daemon never anchors a lock) and the Sol review (data/review-45-sol/report.md) for whoever picks the work up. The underlying outage defects remain real - this only closes the PR.

Closed by captain order 2026-10-13: the Claude-first routing change makes this PR not worth its fix cycle (the P1 same-daemon lock ownership hole needs a real rework). The branch fm/fm-dispatch-fix preserves both fixes (away-cap live-worker counting; codex daemon never anchors a lock) and the Sol review (data/review-45-sol/report.md) for whoever picks the work up. The underlying outage defects remain real - this only closes the PR.
Some checks failed
CI / Test coverage guard (pull_request) Successful in 4m43s
Details
CI / Behavior portable parallel 2 (pull_request) Successful in 18m34s
Details
CI / Behavior portable parallel 1 (pull_request) Successful in 26m15s
Details
CI / Lint 1 (pull_request) Successful in 28m45s
Details
CI / Lint 2 (pull_request) Failing after 30m0s
Details
CI / Behavior portable serial 4 (pull_request) Failing after 16m21s
Details
CI / Behavior portable serial 1 (pull_request) Successful in 24m12s
Details
CI / Behavior portable serial 2 (pull_request) Successful in 21m21s
Details
CI / Behavior portable serial 3 (pull_request) Successful in 22m34s
Details
CI / Behavior portable serial 7 (pull_request) Successful in 20m38s
Details
CI / Behavior portable serial 6 (pull_request) Successful in 20m47s
Details
CI / Behavior portable serial 8 (pull_request) Successful in 21m3s
Details
CI / Repo invariants (pull_request) Successful in 8s
Details
CI / Behavior portable serial 5 (pull_request) Failing after 30m0s
Details
CI / Behavior tests (Herdr) (pull_request) Successful in 15m35s
Details
CI / Behavior portable serial 9 (pull_request) Successful in 29m50s
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
john/firstmate!45
No description provided.