Promote the away-mode watchdog into tracked bin/, and stop it failing silently in five different ways #22
Loading…
Reference in a new issue
No description provided.
Delete branch "fm/fm-afk-watchdog-untracked-and-eats-queue"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why this was a defect and not a chore
state/afk-watchdog.shwas a live, hand-written, untracked file in the gitignored private state directory — invisible to any worker's worktree, untested, unreviewed, present in no other firstmate home, and onermfrom gone. For the component whose entire job is detecting away-mode failure, that fragility was the defect.The two faults it was promoted to fix
It destroyed the durable wake queue it exists to protect.
alarm()ran the wake drain with output discarded before appending its own record; the drain moves the queue aside, prints it, then removes it, restoring only on non-zero exit. So every alarm silently destroyed every pending wake. Evidence: sixteen hourly alarms on 2026-08-27 left exactly one surviving record — each had eaten the previous.Its only delivery path was a record read at the next session start. The one component that detects an away-mode outage could not interrupt the component having it. It shouted sixteen times into a log file. It now routes through the existing wedge-alarm channels rather than a second parallel mechanism.
Five review rounds, all of them finding real silence
Every round found another way this thing could fail to report — which is worth stating plainly, because a monitor that fails quietly is worse than no monitor:
One known limitation, stated rather than engineered around
A rapid exit-and-re-enter inside the legacy watchdog's own poll window can briefly run both copies. It is one-time and transitional — only possible while a pre-this-PR hand-started watchdog is alive — self-limiting, and cannot recur once every watchdog on a machine is lock-based. Fixing it would need either signalling the legacy process (forbidden) or a heuristic guess at whether one is running, and a guess like that outlives the problem and then misfires. Documented instead.
Deliberately not fixed here
A recycled pid can mask a dead lock owner, which would permanently disable whatever the lock guards. The general fix belongs at the shared lock helper — used by the daemon lock, the wake-queue lock and the watcher lock — and changing that primitive for one task's benefit is far too much blast radius. This PR uses a local identity check mirroring the pattern the supervisor daemon already proves. The general problem is filed separately and looks like the same root cause as upstream issue 3279.
Tests
The central one reproduces the sixteen-alarms-one-record loss and proves a pending wake now survives an alarm. Plus: a live foreign-home daemon with a dead local one must alarm; two rapid entries must produce exactly one watchdog; a refresh against an active legacy session must launch none; a new session's failure inside the rate-limit window must alarm while a mid-session refresh must not reset it. All fail against the pre-fix code.
Pipeline passed clean — review, test, document, lint — and pushed at
c2f959ce.prandciskipped because the tool has no Forgejo provider; this PR is what fires CI.state/afk-watchdog.sh was a live, untracked, hand-written file: invisible to any crewmate worktree, untested, and one rm from gone for a component whose entire job is detecting away-mode failure. Promotes it to bin/fm-afk-watchdog.sh and fixes two confirmed faults: (a) alarm() ran bin/fm-wake-drain.sh with output discarded before appending its own record. That drains and deletes the durable wake queue on every successful call, destroying every other pending record (sixteen hourly alarms on 2026-08-27 left exactly one surviving record). The call served no purpose and is simply removed; alarm() now only ever appends its own check:afk-watchdog record. (b) The alarm's only delivery path was a durable record read at the next session start, so the one component that detects an away-mode outage could not interrupt the outage it was reporting. It now also fires the active-alert channels through the same config/wedge-alarm mechanism the away-mode injection wedge alarm uses, extracted out of bin/fm-supervise-daemon.sh into bin/fm-wedge-alarm-lib.sh so both alarm sources share exactly one delivery implementation instead of a second, parallel one. bin/fm-wake-drain.sh's own contract is untouched; no caller changes. Colocated tests in tests/fm-afk-watchdog.test.sh include a regression test that reproduces the sixteen-alarms-one-record queue loss against the old behavior and passes against the fix.Pull request closed