Away-mode watchdog: notice a dead supervisor, and retract its alarm record on a fresh entry #27
Loading…
Reference in a new issue
No description provided.
Delete branch "fm/fix-22-watchdog-liveness-v2"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Supersedes #22, which cannot be updated without a force-push.
Why a new PR rather than updating #22
The validation pipeline's own rebase step — an automatic part of every run — rebased this work onto a main that had since gained #20 and #23. PR 22's branch is an ancestor of the original commit but not of the rebased head, so updating #22 in place would require a force-push.
Firstmate verified that every file PR 22 changes exists in this head, so nothing is lost. The alternatives were rejected deliberately: force-pushing rewrites a branch nobody authorised rewriting, and cherry-picking onto #22's branch would land content the pipeline never validated.
What this delivers
The away-mode watchdog notices a dead supervisor, serialises its stale-lock takeover, and fails closed on identity — plus the fix the captain decided tonight: a fresh
/afkentry now retracts the durablecheck afk-watchdogwake record, not merely its rate-limit marker.Without that retraction, a supervisor crash during an away session could resurface an "away mode is still broken" alert after the incident was resolved, costing a turn on a stale alarm. The code immediately beside it already did exactly this for the sibling wedge alarm; this brings the two into line.
The test was extended to assert the record is retracted, not only that the marker cleared — the previous test passed while the record survived, which is why this shipped.
Validation
Full pipeline passed. Note for the record: two earlier runs on this work died with an opaque parse error that was not a content fault — it was caused by a change to Pi's default model, and swapping the model resolved it.
state/afk-watchdog.sh was a live, untracked, hand-written file: invisible to any crewmate worktree, untested, and one rm from gone for a component whose entire job is detecting away-mode failure. Promotes it to bin/fm-afk-watchdog.sh and fixes two confirmed faults: (a) alarm() ran bin/fm-wake-drain.sh with output discarded before appending its own record. That drains and deletes the durable wake queue on every successful call, destroying every other pending record (sixteen hourly alarms on 2026-08-27 left exactly one surviving record). The call served no purpose and is simply removed; alarm() now only ever appends its own check:afk-watchdog record. (b) The alarm's only delivery path was a durable record read at the next session start, so the one component that detects an away-mode outage could not interrupt the outage it was reporting. It now also fires the active-alert channels through the same config/wedge-alarm mechanism the away-mode injection wedge alarm uses, extracted out of bin/fm-supervise-daemon.sh into bin/fm-wedge-alarm-lib.sh so both alarm sources share exactly one delivery implementation instead of a second, parallel one. bin/fm-wake-drain.sh's own contract is untouched; no caller changes. Colocated tests in tests/fm-afk-watchdog.test.sh include a regression test that reproduces the sixteen-alarms-one-record queue loss against the old behavior and passes against the fix.