FIRSTMATE-7: Stop firstmate CI shard timeouts and timing flakes #48

Merged

john merged 6 commits from fm/fm-ci-shard-fix into main 2026-10-15 12:28:26 +00:00

Conversation 0 Commits 6 Files changed 12 +277 -167

john commented 2026-10-15 06:32:32 +00:00

Owner

Copy link

Story: FIRSTMATE-7 - Stop firstmate CI shard timeouts and timing flakes
Author: Claude Sonnet 5.5 (claude-sonnet-5-5) via Claude Code

Why

PRs 45, 46 and 47 failed CI regardless of their content: serial shards 5 and 9 need about 30 minutes against the 30-minute cap, and shards 4, 6 and 1 fail on fixed wall-clock windows in tests. The serial split was balanced on stale GitHub-runner timing hints while the hub runner is 1.2x to 2.5x slower.

What

• Rebalance the serial weights in bin/fm-test-run.sh from the slowest successful hub timings of runs 132-134: 91 stale hints replaced, 18 unhinted scripts hinted. All nine shards now pack to about 23.1-23.4 min (were 29-30 on the hub). fm-watch-triage is not split: it is about 23 min alone, which is already the shard average, so a split cannot lower the longest shard.
• Timeout: forge serial shards 30 -> 40 min as a documented exception; tests/fm-forge-workflow.test.sh and docs/fm-test-portable-shards.md updated.
• De-flake: the set +e/set -e toggles in tests/fm-procevent.test.sh leaked errexit for about 2400 lines, so failures aborted silently; removed, plus a guard that fails if it leaks again. New fm_test_scaled helper (tests/lib.sh) scales the waits that expect success in fm-procevent, fm-watch-arm and fm-watch-triage; the forge serial lane exports FM_TEST_TIME_SCALE=3. fm-contributions freezes the clock for its one-second-budget shim test.
• Out of scope: the hub runner slowdown (INFRA-82).

Fork-only vs shared with upstream

Fork-only: .forgejo/workflows/ci.yml, tests/fm-forge-workflow.test.sh, the forge-specific doc passages. Shared with upstream (merge-conflict surface): the hint table in bin/fm-test-run.sh and the test edits.

Evidence

Mutations: serial timeout back to 30 fails the forge guard; re-adding set -e fails the new errexit guard. fm-procevent, fm-contributions, fm-watch-arm, fm-watch-triage pass locally (each in its own PID namespace, under nice); lint and doc-audience checks clean.

**Story:** FIRSTMATE-7 - Stop firstmate CI shard timeouts and timing flakes **Author:** Claude Sonnet 5.5 (claude-sonnet-5-5) via Claude Code ## Why PRs 45, 46 and 47 failed CI regardless of their content: serial shards 5 and 9 need about 30 minutes against the 30-minute cap, and shards 4, 6 and 1 fail on fixed wall-clock windows in tests. The serial split was balanced on stale GitHub-runner timing hints while the hub runner is 1.2x to 2.5x slower. ## What - **Rebalance** the serial weights in `bin/fm-test-run.sh` from the slowest successful hub timings of runs 132-134: 91 stale hints replaced, 18 unhinted scripts hinted. All nine shards now pack to about 23.1-23.4 min (were 29-30 on the hub). `fm-watch-triage` is not split: it is about 23 min alone, which is already the shard average, so a split cannot lower the longest shard. - **Timeout:** forge serial shards 30 -> 40 min as a documented exception; `tests/fm-forge-workflow.test.sh` and `docs/fm-test-portable-shards.md` updated. - **De-flake:** the `set +e`/`set -e` toggles in `tests/fm-procevent.test.sh` leaked errexit for about 2400 lines, so failures aborted silently; removed, plus a guard that fails if it leaks again. New `fm_test_scaled` helper (`tests/lib.sh`) scales the waits that expect success in fm-procevent, fm-watch-arm and fm-watch-triage; the forge serial lane exports `FM_TEST_TIME_SCALE=3`. fm-contributions freezes the clock for its one-second-budget shim test. - Out of scope: the hub runner slowdown (INFRA-82). ## Fork-only vs shared with upstream Fork-only: `.forgejo/workflows/ci.yml`, `tests/fm-forge-workflow.test.sh`, the forge-specific doc passages. Shared with upstream (merge-conflict surface): the hint table in `bin/fm-test-run.sh` and the test edits. ## Evidence Mutations: serial timeout back to 30 fails the forge guard; re-adding `set -e` fails the new errexit guard. fm-procevent, fm-contributions, fm-watch-arm, fm-watch-triage pass locally (each in its own PID namespace, under nice); lint and doc-audience checks clean.

john added 2 commits 2026-10-15 06:32:32 +00:00

FIRSTMATE-7: Stop firstmate CI shard timeouts and timing flakes ... 821d70fd6c

Rebalance the serial shards on hub-measured weights, give the forge serial
shards a 40-minute bound, end the errexit leak in the procevent test and scale
its fixed wall-clock windows for slow CI hosts.

no-mistakes(document): Point CONTRIBUTING test-timing guidance at fm_test_scaled 70ace7c2c2

john added 2 commits 2026-10-15 08:29:28 +00:00

FIRSTMATE-7: Update the scheduling fixture to the refreshed serial hints ... 4fd1a58f00

The rebalanced hub-measured hints reorder the two hinted scripts the fixture
schedules, so expect the new hint-descending order.

no-mistakes(review): Refresh stale --changed timeout headroom rationale 66490a2f56

john added 2 commits 2026-10-15 10:57:11 +00:00

FIRSTMATE-7: Stop a stalled runner start from failing the procevent fixtures ... 7028374202

A listener start that stalled longer than a fixture's short owner lease made the
runner's guard refuse to initialise, so the runner exited before recording
itself and the fixture failed with no clue why. The bound fixture now hands its
first guard read a lease aged zero and retries the start like a supervision
cycle would, the other short-lease fixtures keep the owner present until the
runner is up, and the confirm and wait windows that expect success stretch with
FM_TEST_TIME_SCALE.

no-mistakes(review): Drop bound fixture relaunch loop; fail on first launch b21a86ac36

john merged commit c0567a878d into main 2026-10-15 12:28:26 +00:00

john referenced this pull request from a commit 2026-10-15 12:28:27 +00:00

FIRSTMATE-7: Stop firstmate CI shard timeouts and timing flakes (#48)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

john/firstmate!48

No description provided.