Orihon.UseCases.Tests intermittently hangs the test host under CPU contention (pre-existing) #67

New issue

Closed

opened 2026-07-26 17:58:24 +02:00 by scarlet · 0 comments

scarlet commented

2026-07-26 17:58:24 +02:00

Member

Found while chasing a CI failure on PR #65. Filing separately because it is pre-existing on main and unrelated to that PR's content.

Symptom

A full dotnet test tests/Orihon.UseCases.Tests -c Release run occasionally never finishes — the test host hangs indefinitely (killed at a 70 s outer timeout). It is a hang, not a slow run: healthy runs take ~2 s.

In a captured hang, 229 of 240 tests had completed and the 11 missing ones were the entire RunEngineTests class. Every other class, including all the engine-heavy AnnotationRunTests, had passed. So one test in that class stops progressing and blocks the rest of its (sequential) class while the assembly waits for it.

Reproduction and rates

Each figure is from a loop of full-assembly Release runs with --no-build and a 70 s per-run timeout:

Configuration	Hangs
`origin/main` (`4a5e776`), idle machine	0 / 15
`origin/main`, with 6 background CPU hogs (`yes > /dev/null`)	1 / 20
PR #65 branch, idle machine	2 / 25
PR #65 branch, its new sfx tests excluded	0 / 20
PR #65 branch, engine change reverted, sfx tests kept	2 / 20

Reading: the trigger is CPU/thread-pool contention, not any particular change. Main reproduces it once given artificial load; the PR branch reproduces it without extra load simply because its added tests raise concurrency. Reverting the PR's RunEngine change does not help, and removing its tests does — both consistent with load being the variable.

Where it most likely sits

RunEngine.WhenIdleAsync() (Task.WhenAll(inFlight.Values)) is the only unbounded await the engine tests use; every other wait in those tests is a 10 s UntilAsync deadline that would fail rather than hang. Several RunEngineTests cases await it after driving a stage to a terminal state. A task that never completes — or a snapshot taken as tasks are being re-scheduled — would hang there forever with no diagnostic.

Notes for whoever picks this up

dotnet test --blame-hang masks the bug: 25 runs with blame instrumentation and 10 more under forced CPU load never reproduced it, so the hang dump route needs something else (attach dotnet-stack/dotnet-dump to the live hung host instead).
Running RunEngineTests alone never reproduced it (30 runs) — it needs the whole assembly's cross-class parallelism.
Suggested first moves: give WhenIdleAsync an optional timeout for test use (or a WaitAsync(TimeSpan) at the call sites) so the failure mode becomes a readable assertion instead of a hang, then re-run the loop to see which test surfaces.

Impact

CI is red at random with no message beyond a timeout, and a hung run costs the whole job's wall clock. Not a production-path defect as far as this investigation went — the engine itself was exercised heavily throughout without misbehaving.

🤖 Generated with Claude Code

Found while chasing a CI failure on PR #65. Filing separately because it is **pre-existing on main** and unrelated to that PR's content. ## Symptom A full `dotnet test tests/Orihon.UseCases.Tests -c Release` run occasionally never finishes — the test host hangs indefinitely (killed at a 70 s outer timeout). It is a hang, not a slow run: healthy runs take ~2 s. In a captured hang, **229 of 240 tests had completed** and the 11 missing ones were the entire `RunEngineTests` class. Every other class, including all the engine-heavy `AnnotationRunTests`, had passed. So one test in that class stops progressing and blocks the rest of its (sequential) class while the assembly waits for it. ## Reproduction and rates Each figure is from a loop of full-assembly Release runs with `--no-build` and a 70 s per-run timeout: | Configuration | Hangs | |---|---| | `origin/main` (4a5e776), idle machine | 0 / 15 | | `origin/main`, with 6 background CPU hogs (`yes > /dev/null`) | **1 / 20** | | PR #65 branch, idle machine | 2 / 25 | | PR #65 branch, its new sfx tests excluded | 0 / 20 | | PR #65 branch, engine change reverted, sfx tests kept | 2 / 20 | Reading: the trigger is **CPU/thread-pool contention**, not any particular change. Main reproduces it once given artificial load; the PR branch reproduces it without extra load simply because its added tests raise concurrency. Reverting the PR's `RunEngine` change does not help, and removing its tests does — both consistent with load being the variable. ## Where it most likely sits `RunEngine.WhenIdleAsync()` (`Task.WhenAll(inFlight.Values)`) is the only **unbounded** await the engine tests use; every other wait in those tests is a 10 s `UntilAsync` deadline that would fail rather than hang. Several `RunEngineTests` cases await it after driving a stage to a terminal state. A task that never completes — or a snapshot taken as tasks are being re-scheduled — would hang there forever with no diagnostic. ## Notes for whoever picks this up - `dotnet test --blame-hang` **masks the bug**: 25 runs with blame instrumentation and 10 more under forced CPU load never reproduced it, so the hang dump route needs something else (attach `dotnet-stack`/`dotnet-dump` to the live hung host instead). - Running `RunEngineTests` alone never reproduced it (30 runs) — it needs the whole assembly's cross-class parallelism. - Suggested first moves: give `WhenIdleAsync` an optional timeout for test use (or a `WaitAsync(TimeSpan)` at the call sites) so the failure mode becomes a readable assertion instead of a hang, then re-run the loop to see which test surfaces. ## Impact CI is red at random with no message beyond a timeout, and a hung run costs the whole job's wall clock. Not a production-path defect as far as this investigation went — the engine itself was exercised heavily throughout without misbehaving. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

scarlet referenced this issue

2026-07-26 17:59:21 +02:00

feat: staged annotation — the core pass and the optional sfx pass (ADR 0023) #65

scarlet referenced this issue

2026-07-26 18:12:14 +02:00

feat: staged annotation — the core pass and the optional sfx pass (ADR 0023) #65

scarlet referenced this issue

2026-07-26 19:48:20 +02:00

feat: the setup agent sees the chapters the user organized #69