PYR0D-355: Reimplement the away-mode watchdog (idle-with-work wake, stall and dead-supervisor alarms) #43

Merged

john merged 14 commits from fm/fm-afk-watchdog-reimplement into main 2026-10-02 15:29:23 +00:00

Conversation 0 Commits 14 Files changed 10 +945 -1

john commented 2026-10-02 11:31:13 +00:00

Owner

Copy link

Story: PYR0D-355 - Away mode: reimplement our own watchdog (idle-with-work and dead-supervisor alarms)

What was wrong

On 2026-10-01 away mode did no work for 4h21m. One of the two causes was that nothing gave firstmate a turn when the fleet went idle with ready work queued. The fork's earlier private watchdog had caught the same shape on 09-29, but it was lost in the upstream rebuild.

What changed

• New bin/fm-afk-watchdog.sh, started with the away daemon (fm-afk-start.sh / fm-afk-launch.sh). It runs only while an away record exists.
• Idle-with-work wake: when no ordinary task is working and fm-tasks-axi.sh ready is non-empty, it wakes firstmate once per idle period through the durable wake queue, carrying the away words' read-back.
• Stall alarm: it pages only when work is waiting and nothing is working.
• Dead-supervisor alarm: it pages through the wedge-alarm channels when supervision stops beating.
• It stands down where the upstream supervision host runs. It checks that through the shared fm_supervision_host_enabled gate, never by whether the config file exists.

Decisions made in review (firstmate)

• Any ordinary task whose verdict is working, from any source (pane, run-step), counts as busy and healthy. A busy fleet never pages.
• Secondmates are neutral: they neither vouch for the fleet nor count against it.
• Done or terminal tasks are not stalled, and the stall check is skipped under a quiet record, so a finished fleet is never paged.
• The interrupting page is rate-limited to WATCHDOG_REPEAT_SECS on its own marker, while the durable wake retries every poll.
• The upstream half of "lets have both" is PYR0D-356 (PR 42, merged), so this PR stays fork-local.

How it was checked

tests/fm-afk-watchdog.test.sh covers the idle wake, the stall alarm (busy pane, run-step, secondmate, quiet and done cases), alarm rate limits, and the host gate (absent file on a Claude primary, and "off"). It also went through the no-mistakes pipeline.

Follow-ups

• Retire the stopgap custom check state/away-idle-nudge.check.sh (home-private) once this merges.
• A live overnight trial on a real away session.

**Story:** PYR0D-355 - Away mode: reimplement our own watchdog (idle-with-work and dead-supervisor alarms) ## What was wrong On 2026-10-01 away mode did no work for 4h21m. One of the two causes was that nothing gave firstmate a turn when the fleet went idle with ready work queued. The fork's earlier private watchdog had caught the same shape on 09-29, but it was lost in the upstream rebuild. ## What changed - New `bin/fm-afk-watchdog.sh`, started with the away daemon (`fm-afk-start.sh` / `fm-afk-launch.sh`). It runs only while an away record exists. - **Idle-with-work wake:** when no ordinary task is working and `fm-tasks-axi.sh ready` is non-empty, it wakes firstmate once per idle period through the durable wake queue, carrying the away words' read-back. - **Stall alarm:** it pages only when work is waiting and nothing is working. - **Dead-supervisor alarm:** it pages through the wedge-alarm channels when supervision stops beating. - It stands down where the upstream supervision host runs. It checks that through the shared `fm_supervision_host_enabled` gate, never by whether the config file exists. ## Decisions made in review (firstmate) - Any ordinary task whose verdict is `working`, from any source (pane, run-step), counts as busy and healthy. A busy fleet never pages. - Secondmates are neutral: they neither vouch for the fleet nor count against it. - Done or terminal tasks are not stalled, and the stall check is skipped under a quiet record, so a finished fleet is never paged. - The interrupting page is rate-limited to WATCHDOG_REPEAT_SECS on its own marker, while the durable wake retries every poll. - The upstream half of "lets have both" is PYR0D-356 (PR 42, merged), so this PR stays fork-local. ## How it was checked `tests/fm-afk-watchdog.test.sh` covers the idle wake, the stall alarm (busy pane, run-step, secondmate, quiet and done cases), alarm rate limits, and the host gate (absent file on a Claude primary, and "off"). It also went through the no-mistakes pipeline. ## Follow-ups - Retire the stopgap custom check `state/away-idle-nudge.check.sh` (home-private) once this merges. - A live overnight trial on a real away session.

john added 11 commits 2026-10-02 11:31:14 +00:00

PYR0D-355: Reimplement the away-mode watchdog so an idle fleet with ready work wakes the away session ... fdb535540a

A fork-local, independent watchdog (bin/fm-afk-watchdog.sh) runs beside every away
record on the daemon, supervision-host and Pi postures. It appends one durable
check: afk-idle wake per idle period, carrying the away record's read-back, when no
ordinary task is working and the ready queue is non-empty. It alarms a dead daemon,
a dead watcher (host path: a stale beat only) and a stalled fleet once per failure
class, resets its markers on a new away entry, and exits when the record is gone.
Away entry and the daemon entry start it; a singleton lock makes extra starts safe.

Merge remote-tracking branch 'origin/main' into fm/fm-afk-watchdog-reimplement ... c7aee16b07

# Conflicts:
#	docs/scripts.md

no-mistakes(review): Fix watchdog ready parsing, wake routing, stall checks, host gate 5d4f13a35b

no-mistakes(document): Note watchdog alarms in wedge-alarm channel doc 75ab447c51

Merge remote-tracking branch 'origin/main' into fm/fm-afk-watchdog-reimplement ... 89b080e3aa

# Conflicts:
#	.agents/skills/afk/SKILL.md

Test the watchdog's host gate for an absent file on a Claude primary and an off file f0da9b3b61

no-mistakes(review): Harden watchdog stall, alarm-retry and takeover race handling 47712870db

no-mistakes(review): Treat any working task as healthy; rate-limit active alerts separately fc7fa92a61

no-mistakes(review): Stop busy secondmates vouching for the fleet in stall check 4ecb27fa09

no-mistakes(review): Count only busy-pane working verdicts; make secondmates stall-neutral b28592b835

no-mistakes(review): Count working verdicts from any source as busy b416f45c7a

john added 2 commits 2026-10-02 12:05:25 +00:00

Fix the watchdog's host missing-beat alarm, secondmate stall freshness and crew-state failure handling 3133d77e4c

no-mistakes(review): Treat inconclusive crew-state unknowns as unknown; key episodes on entered_epoch cb92bf608a

john added 1 commit 2026-10-02 13:13:11 +00:00

Rename the watchdog launch seam so it no longer trips the gate guard's stock-layout rule b833a1c1d9

john merged commit c1ada66ade into main 2026-10-02 15:29:23 +00:00

john referenced this pull request from a commit 2026-10-02 15:29:25 +00:00

PYR0D-355: Reimplement the away-mode watchdog (idle-with-work wake, stall and dead-supervisor alarms) (#43)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

john/firstmate!43

No description provided.