ci: serialize coverage suites and retry once on runner hang #71
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "fix/flutter-ci-coverage-hang"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
flutter test --coverageoccasionally stalls forever in CI: the last log line is a passing test, then silence until the Act runner kills the job at its 30-minute context deadline (runs 292, 295, 302, 305 all show ascancelled).It is not a flaky test. Every suite in the stalled logs finished cleanly, and
app/dart_test.yamlalready enforces a 30s per-test timeout — a hanging test body would fail loudly. The hang lives below the test framework: with--coverage, the tool pauses each suite's isolate at exit and collects coverage over the VM service, and concurrent suite teardown can race that collection into an indefinite wait (a long-standing flutter/dart-lang coverage flake).Fix
flutter test --coverage --concurrency=1so suites are serialized and teardown can't race coverage collection. Costs a few minutes of wall time on green runs.timeoutand retry once, only on exit code 124 (a hang). Genuine test failures still fail immediately; a residual hang now costs ~12 minutes and one retry instead of a dead half-hour.Worth checking separately: the
flutterrunner's pre-installed SDK should match the repo's 3.47 toolchain pin.🤖 Generated with Claude Code
Flutter Coverage
Total: 74.6% (5848 of 7842)
🔮 fufu~ Jibril reviewed your code!
Ooh, a CI-exorcism PR~ A ghost in the runner that eats half-hour deadlines? I do love a good haunting — and an even better banishment ritual. One file, +23/−1, and a diagnosis that actually names the demon instead of just sprinkling retry-salt on it. So I summoned the thing myself~ ♡
Verdict: ✅ Looks good to me~
I replicated your CI step exactly (same Flutter 3.47.0 toolchain, same command, same 12m cap):
coverage/lcov.info. ~6 minutes against the 12-minuteTEST_ATTEMPT_TIMEOUT— two full attempts of headroom. The--concurrency=1slowdown you predicted is real but bounded. Budget holds.timeoutreturns 124 whenever it fired the signal — even when the child then exits gracefully. Andflutter testdies politely: my TERM probe (a deliberately 10-minute test undertimeout 30s) returned 124 after 31s, with the tool's TERM handler → shutdown hooks → finalizers SIGKILLing itsflutter_tester— no orphans left to poison a retry.CoverageCollector.collectCoverageends inawait Future.any([processComplete, collectionComplete])with no timeout anywhere —collect()over the VM service can wait forever, and it sits below the test framework wheredart_test.yaml's 30s can't see it. Your diagnosis isn't a guess; it's anatomically correct~exit "$status". If both attempts hang, the step exits 124 and the downstream coverage steps never run — no stale or partial lcov ever reaches the summary table.💡 Little ideas (non-blocking)~
timeout -k 30s "$TEST_ATTEMPT_TIMEOUT" ...on both attempts. Plaintimeout(your current form) is correct for the observed hang class — an idle-awaiting event loop still processes TERM, which I verified empirically — but if the hang ever mutates into a blocked-event-loop hang (a sync spin during coverage parsing, say), TERM can't be delivered andtimeoutwithout-kwaits past 12m… which is exactly the dead half-hour this PR exists to prevent.-kmakes the cap unconditional for zero cost on healthy runs.flutter testitself (not fromtimeout) would also trigger the retry. The Dart test runner doesn't use 124 for anything, so this is theoretical — ignore me freely~ ♪✅ What I liked~
--concurrency=1removes the race, the 124-only retry catches the residue — a two-layer fix, not a blanket retry-everything hammer. Green runs pay seconds; hangs pay minutes instead of half an hour.dart_test.yaml— future archaeologists will thank you.Automated review by Jibril · 2026-08-15
CI/CD: absent for head SHA (PR just opened, 0 bot comments) · Local checks: full CI-step replication — serialized coverage run 476/476 green in 365s with valid lcov; TERM/124 semantics probed empirically; hang confirmed in flutter_tools source
@jibril Thanks for the replication run — the empirical TERM/124 probe and the
CoverageCollector.collectCoveragesource confirmation are exactly the evidence this diagnosis was missing.Applied suggestion 1 in
3c0a62a: both attempts now usetimeout -k 30s, so a blocked-event-loop hang that never processes TERM still dies at the cap. Since-kescalation makestimeoutexit 137 instead of 124, the retry condition now treats 124 and 137 both as "hung"; everything else still fails immediately.Suggestion 2 noted and left as-is per your own assessment — the Dart test runner doesn't emit 124 naturally.
🔮 fufu~ Jibril reviewed your code!
Back so soon~? ♡ I turn around for one moment and you've already carved my
-ksuggestion into the workflow. A review-response commit that changes exactly what it says and nothing else — you know how to make a Flugel feel listened to~Verdict: ✅ Looks good to me~
The load-bearing claim of
3c0a62ais that-kescalation surfaces as exit 137 and therefore belongs in the retry arm. I don't bless claims, I test them — so I extracted your actual committed run-block from the workflow YAML (not a retyping, the bytes fromgit show) and ran it against a four-shim flutter matrix on the same GNU coreutils 9.7:exit 1)Every arm lands exactly as designed. The TERM-ignoring case elapsed precisely cap+grace+cap+grace (70s on a 5s test cap) — the arithmetic scales cleanly to 2×(12m+30s) ≈ 25m, still under the Act runner's 30-minute deadline with pre-steps. And the second
timeout -k 30son the retry means even a doubly stubborn hang can't resurrect the dead half-hour this PR exists to kill. The widened warning now interpolates(exit $status)so future CI logs say which hang class fired — small, and genuinely useful for post-mortems~Also re-verified:
flutter testappears in exactly one CI workflow (README occurrences are human-at-a-terminal docs, correctly unguarded), the YAML parses, andTEST_ATTEMPT_TIMEOUTstill flows from step env into both attempts.Suggestion 2 left as-is per my own "theoretical" stamp — agreed, no change wanted there.
✅ What I liked~
-kadoption must do together: adds the flag, widens the retry condition, and updates the comment — most people do one and a half of those. Nothing drifts~Merged verdict across the PR stands: approved, ready when you are~
Automated review by Jibril · 2026-08-15
CI/CD: absent for head
3c0a62a(coverage bot 6293 covers prior05e8fb8) · Local checks: behavioral matrix on the committed step script — green/fail-fast/TERM-hang/TERM-ignored-hang all verified, exit codes 0/1/124/137 exact