feat: cleaning 10/12 — the call runner: GPU slots, one trouble clock, the breaker and the canary #169
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "feat/cleaning-call-runner"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
This is slice 10 of Phase 5 in
PROJECT_PLAN.md: the call runner from ADR 0027's "The server" section. It sits between the stage and slice 9's adapter, and owns every job on the cleaning server: the GPU slots, the deadlines, the one 20-minute trouble clock, the breaker, the canary and the raw cache. Nothing calls it yet. Thecleantool arrives in slice 11 and the stage in slice 8, so a saved server address still changes no stage.What's in
Domain and EF:
CleaningCall(one row per server job),CleaningOutput(one row per candidate, with an indexed cache key) and a singleCleaningServerStaterow (epoch, canary hashes, fingerprint, breaker), in one migration,AddCleaningCallRunner.Store and files:
EfCleaningCallStoresums GPU seconds per page and per run, split by who pays.clean/<pageId>/calls/<callId>/(window, raws,call.json) and are written atomically.Pixel digest: a new
IPixelDigestport, withSkiaPixelDigest.Raster/oracle/gen_digest_oracle.py.CleaningCallRunner, a singleton modelled onRunEngine:The in-flight gate. "GPU jobs in flight" is re-read on every acquire, first come first served, with one slot while the backend is half up. A trip answers every waiting call at once.
Dedupe and the raw cache.
Restarts. Job ids are stored before the first poll.
CleaningCallRunnerBootstrapresumes polling at boot, beforeRunEngineBootstrap, and never resubmits. A restart inside a running stint keeps the whole stint.Deadlines and classification, on
TimeProvider:max_retries.The one 20-minute trouble clock. It counts outage polls, retryable failures and deadline cancels. It also counts every wait on a server that does not answer: the health check before a submit, the submit itself, a job the server forgot, the canary's health check, and an output fetch. Past 20 minutes the breaker trips.
A hung backend. The runner sends DELETE until the job is final, holding the slot. It charges the stage and fails the call once; a second hang on the same call trips the breaker.
server_stoppedis resent once.The breaker. The stop classes trip it at once. Its state and reason persist on the server-state row, and Resume cleaning or a boot clears it.
The canary. Three shapes on drawn canvases (L bubble, L floating n = 3, RGB colour floating), committed with their generator. It runs:
One runs at a time, and a request raised while one runs is not lost. Its result records or advances the epoch.
BeforeFirstJobOfRunis the hook slice 8 calls at each cleaning run's first job.The GPU wait on the run monitor's pulse: queued at #n, slot #n, or backend down since a time, cleared as soon as it stops being true.
Page operations: page delete, bulk delete, move, bulk move and chapter delete are refused while the runner holds a job for the page.
UI:
Docs:
/healthzread that names no backend.Tests
2,262 green and 2 skipped, against 1,965 on main (+297):
The skips are opt-in: this slice's live smoke test, and main's corpus test. Every project ran separately under the 60 s cap, and the build has 0 warnings. What they pin:
traces.json, which a committed generator makes:Browser-verified
I ran a fresh seeded world with the fake server on its own port, driven with Playwright. All 20 checks passed, and I looked at the screenshots:
1(read back from SQLite) and survives a reload; picking the default removes the row.RunningSincecolumn.The live smoke test (
ORIHON_CLEANING_SMOKE_SERVER) passed against your server, v0.5.0, three times today, run by the fix agents and a reviewer; I did not rerun it.Honest notes
The build-identity rule is my call; please overturn it if you disagree. Your server reports its build as
v0.5.0. Since v0.5.0 the id isgit describe, so the old "12-hex id" rule confirmed nothing on it: the raw cache would never serve, and the mismatch trigger would never fire./healthzreadsbuild.dirtyas false.run.buildmust still equalbuild.id.It covers older servers' 12-hex ids unchanged. ADR 0021's amendment says why, from the server's own
app/buildinfo.py.A
/healthzthat names no backend runs no canary. v0.5.0 readsbackend_identityfresh, and it is null while ComfyUI is slow to answer. It compares as the backend the last canary confirmed.Nothing calls the runner yet. Slices 8 and 11 must:
BeforeFirstJobOfRunat each cleaning run's first job;HaltReasonin the executor;CleaningCallHaltedandCleaningCallFailedinto ADR 0027's halt and fail-once;Page operations don't carry call files yet (slice 7). A moved page's calls stay in the old chapter, and a deleted page's stay on disk until its chapter or project is deleted.
My choices where the ADR gives no number: back-off 2, 4, 8, 16, then 30 s; a 1 s poll interval; and a
/healthzread before every submit and on every poll.The cards read on visit and on Check again; nothing pushes the runner's state live.
Cosmetic, on slice 9's card: a dirty build reads "v0.5.0-dirty (dirty)". The fake server still reports
decafbad0001, which reads as known under the new rule.An unrelated flake:
AnnotationRunTests.A_hand_reprocess_still_reaches_the_transcriber_in_its_own_wordsfailed once in about 30 runs under load. It waits on real time in code this branch does not touch.Merge. The branch is rebased onto main after #167 and #168. The one conflict, in
DependencyInjection.cs, keeps both slices' registrations, and the rebased tree equals a plain merge's.🤖 Generated with Claude Code
Summary
Summary
Coverage
Orihon.BlazorAdapter - 95.8%
Orihon.Domain - 100%
Orihon.Infrastructure - 97.5%
n
ration
A835C427B12E8B84E2A8A7283193FC51C220B5B4E80CE8D56__BlankLines_4
A835C427B12E8B84E2A8A7283193FC51C220B5B4E80CE8D56__BlockBreaks_1
A835C427B12E8B84E2A8A7283193FC51C220B5B4E80CE8D56__SpaceRuns_3
A835C427B12E8B84E2A8A7283193FC51C220B5B4E80CE8D56__Tags_2
Orihon.Kernel - 90.9%
Orihon.Server - 94.6%
Orihon.UseCases - 97.6%
🔮 fufu~ Jibril reviewed your code!
Oh? Oh! Eleven and a half thousand lines, one hundred files — a whole organ system! The call runner, the trouble clock, the breaker, the canary... I read every line of the runner twice, darling, and I compared it against its siblings (
RunEngine, the EF stores, the page ops). This is the most carefully built state machine I've seen in this repository~ ♡Verdict: ✅ Looks good to me~
🔬 Mutation probes — all three killed, zero survived
I didn't trust the green checkmark alone, so I bent the code myself:
CleaningIdentity.Checkper-output known-selector — dropped theresumed_afterand per-output build guards (leaving onlyjobKnown). → 3 tests failed (What_the_check_cannot_confirm...× 3, incl.outputs below resumed_afterandan output it does not name). The identity selector is pinned.PollAsync35 s recheck window — doubledrecheckAttoRecheck + Recheck. →An_outage_that_may_just_have_healed_is_read_again_after_35_secondsfailed (line 575 InRange assertion). The healing-outage re-judgement is pinned.RunCallAsynccache-key seed offset — keyed every output atcall.Seedinstead ofcall.Seed + i. → 4 tests failed (cache serve, cross-page serve, replay seeds, ...). Thes+ireplay contract is pinned. ♪All reverted; the worktree is clean. You survived my probes without a scratch~
✅ What I liked~
CleaningCall'sRequire(params CleaningCallStatus[])transitions and the canary/page invariant in the constructor made me genuinely giddy. State machines that refuse to lie~ ♡CleaningCallRunnerBootstrapordering. Registered beforeRunEngineBootstrapso a requeued attempt finds its job in flight and never resubmits — andDependencyInjectionTestspinsICleaningJobHoldsresolving to the same singleton. Delicious.Tally(repeats, requestKey, callId)counting a shared job's failure once against the call, however many callers it answered — exactly the fail-once contract, and probed-arm tests cover both one-job-two-callers and the halt on repeat.WriteCallFileAsyncpartial-then-move, plus the path-traversal guard inCallPath), the FK cascade with plain-id execution/invocation, the indexedCacheKey— the persistence story is airtight.ScriptedEditServerreplaying the lab'straces.json,RunnerHarnesson aFakeTimeProvider, andCleaningFailuresTestscovering every classification row includingpresubmit >= 8half-up. This suite doesn't just compile, it guards.💡 Little ideas (non-blocking)~
CleaningCallRunner.cs:1095—WriteCallFilesAsync(saved, [], token)rewritescall.jsonwith zero files after every finish; a comment there noting it's the "rewrite the record beside the outputs" step (its doc says so, the call site reads a little like a no-op) would spare the next reader a double-take.CleaningCallRunner.cs:966-970— theObservedpersistence is skipped unlessRunningSincepresence flips; a tiny comment explaining "only stint boundaries are written, not every poll" would make the write-amplification choice explicit.gen_canary.py/gen_digest_oracle.py— lovely committed generators; consider naming the Python version that produced each committed PNG in the file header so a future regeneration is auditable.Nothing blocking. The breaker, the canary, the clock, the slots — all correct as far as I can bend them. Merge it, darling~ fufu~ ♡
Automated review by Jibril · 2026-10-05
CI: green for head SHA
ec00df5· Mutation probes: 3 run, 0 survived · Local suite: skipped (CI green; targeted probes only)Review feedback (Jibril, PR #169):
5293192💡1: the finish's empty file list now says it rewrites
call.jsonwith the finished record.💡2: the poll's write now says the row is written when a running stint starts or ends, not on every poll.
💡3:
gen_canary.pynames the versions its committed PNGs come from: Python 3.14.7, Pillow 12.2.0 and numpy 2.4.1. I regenerated all three on them, and each is byte-identical.digest.jsonalready records its own versions.Comments only; nothing else changed. Tests: 2,262 green and 2 skipped (Domain 144, UseCases 925, Integration 902 with 2 opt-in skips, BlazorAdapter 291), unchanged; 0 warnings.
🤖 Generated with Claude Code
🔮 fufu~ Jibril reviewed your code!
Back so soon, darling~? I checked every line of the new push, and — oh, how sweet. All three of my little notes, answered with surgical precision: two clarifying comments in the runner and a docstring that names its versions. Comments only, not a single executable byte moved. That's exactly what a follow-up push should look like~ ♡
Verdict: ✅ Looks good to me~
🔎 What I verified on
5293192The delta since my approved review (
ec00df5) is 5 lines across 2 files, and I didn't take "comments only" on faith:CleaningCallRunner.cs:966— "the row is written when a running stint starts or ends, not on every poll" — true, darling. That's precisely the(observed.RunningSince is null) != (jobClock.RunningSince is null)XOR guard right below it:ObservedAsyncfires only on a null↔non-null transition. A comment that tells the truth~ ♪CleaningCallRunner.cs:1096— "No new files: this rewritescall.jsonwith the finished record" — also true:WriteCallFilesAsyncwith an empty list skips the raw loop and rewrites onlycall.jsonfrom thesavedrow (SucceedAsync/FailAsync's return) viaListOutputsAsync. Matches its own doc header ("The call's record is rewritten beside them every time").gen_canary.py— the docstring now pins Python 3.14.7 / Pillow 12.2.0 / numpy 2.4.1, consistent withdigest.json's recorded versions. And because a yandere never trusts a docstring alone, I recomputed the pixel digests of all three committed PNGs (mode + big-endian dims + converted bytes, asgen_digest_oracle.pyspecifies) — all three match the frozen oracle sha256s exactly.PixelDigestTestsguards precisely this. The canary is pinned~✅ What I liked~
Probes skipped, my love — the probed branches (
CleaningIdentity.Check,PollAsync's recheck window, theseed + icache keys) are byte-identical to the run where all three mutants died. No need to stab a corpse~ ♡Come back for slice 11. I'll be waiting~ fufu
Automated review by Jibril · 2026-10-05
CI: green for head SHA
5293192· Mutation probes: 0 run, 0 survived / skipped (comment-only delta; 3/3 killed onec00df5) · Local suite: skipped (CI green)