feat(runs): runs start, cancel and retry — the use cases, run snapshots as runs settle, a trash that stops the story's run, the seeded runs (slice 11b) #70
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "feat/runs-start"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Slice 11b of Phase 2 (
docs/PROJECT_PLAN.md): runs start, cancel and retry. A story's runs now start, cancel and retry through use cases over the engine (ADR 0016), and each finished run gets its one run snapshot (ADR 0022). No UI yet: the plan splits the old 11b line, and the Runs section is 11c.What lands
StartRun(story, chapter?):CancelRun(run): the engine's cancel, which drains before it answers.RetryRun(step): a failed step, as the inspector will ask it.TakeRunSnapshot(run)always takes a succeeded run's snapshot, and a failed or cancelled run's only when it wrote. It sits at the story's latest position. Its label names the graph, the outcome when the run did not succeed, and its operations in journal order ("Continue: Commit appended 2 paragraphs").RunSnapshotGate), so two callers never both add one.ISnapshotStore.MoveAsyncandStorySnapshot.MoveTocarry the move.RunSnapshotsHosttakes each run's snapshot as it settles. It subscribes before it reconciles, at its start, every finished run; a finish the last process missed gets its snapshot then.IRunHistory(EF) lists the finished runs, and a story's runs whose last finish no snapshot has taken.TrashStoryfirst cancels and drains the story's live run (ILiveRunsover the engine).TakeRunSnapshot.0ae2ca5). The plan's 11b line is split into 11b and 11c; the run history's cost columns stay with slice 14, now after 11c.Decisions
Tests
BlazorAdapter changes only test doubles here (
NoLiveRuns,MoveAsyncon the fake stores). It hit one 15 s bUnit wait in 3 of 8 full runs under load, a different store-failure or delta test each time. Each passes alone, and the last three full runs passed.StorySnapshot.MoveTo.Reviews
Reviews ran as workflows, one agent at a time.
182c26c): reviewers for correctness, for the tests and the seed, and for the docs, then one skeptic over every finding.925620d:925620d): a check of every round-0 fix and of the fix commit, then a skeptic.68b4ef1:0ae2ca5): two approvals, with four ideas. The three in code are taken in6f72d7d:App check (the sample world)
68b4ef1. The seed ran and the snapshots' host reconciled the seeded runs at its start, adding nothing: one run snapshot, "Continue: Commit appended 2 paragraphs", above "Your edits". The run that gave up has none.🤖 Generated with Claude Code
A story's runs now start, cancel and retry through use cases over the engine (ADR 0016), and each finished run gets its one run snapshot (ADR 0022). The plan splits the old 11b line: the Runs section is 11c. - StartRun(story, chapter?): what the engine cannot know is refused first, with nothing written — a story gone or in the trash, no graph, a graph in the trash, another story's chapter. Then a before-run snapshot of the writer's edits under the run's id, and the engine starts the active graph as it stands, the run's target chapter in its trigger envelope (ADR 0012). The engine's own refusals — the story's run still going, a graph that does not load, has no node or has problems — are named; the snapshot stays, true of the edits it marks. - CancelRun(run): the engine's cancel, which drains before it answers. - RetryRun(step): a failed step, as the inspector will ask it; a before-run snapshot first; the engine's refusals named. - TakeRunSnapshot(run): a succeeded run always, a failed or cancelled one only when it wrote; labelled with the graph, the outcome when it did not succeed, and its operations' labels in order ("Continue: Commit appended 2 paragraphs"); taking it again changes nothing, and a run that wrote past it after a Retry moves it to the end, relabelled. ISnapshotStore.MoveAsync and StorySnapshot.MoveTo carry the move. - RunSnapshotsHost: takes each run's snapshot as it settles, and at its start reconciles every finished run — a finish the last process missed gets its snapshot then. IRunHistory (EF) lists the finished runs. Registered before the engine's host, so no finish goes by unheard. - TrashStory stops the story's live run first (ILiveRuns over the engine: cancelled, drained, snapshotted), so a purge never deletes ledger rows under an execution and the run's snapshot sits before the trash. - The seed: the run that wrote the sample paragraphs now has its ledger rows and its run snapshot through TakeRunSnapshot; a later run's writer gave up after three transient attempts, each error logged, so it wrote nothing and has no snapshot. - Docs: ADR 0016 and 0022 amended (slice 11b); the plan's 11b line split into 11b and 11c. Tests: a run into its target chapter after a snapshot of the edits, one into the last chapter with no edits since a version, the refusals before the engine (nothing written) and the engine's (its snapshot stays); a cancel that drains its request; a retry after three failures that snapshots and finishes; the retry's refusals, the engine's not-found race through a test ledger included; a trash that stops a run that wrote — cancelled, snapshotted before the trash row, another story's run left going — and one whose writer streams; the run snapshot's outcomes and labels, taking it twice, the move after a retry, its announcements; the host's reconciliation, its live finishes, a row's settle ignored, a failure logged while the next goes on, its stop; the finished-run history; the seeded runs; StorySnapshot.MoveTo. Probes: 40 runs; 32 distinct compiling mutants, all killed — T6 (the label's operations out of order) survived once, before its test wrote two different operations; 7 first cuts that did not compile were rerun as compiling variants. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>Summary
Summary
Coverage
Kagaku.Flow - 99.7%
E95E47D27E8B105AC5589487C0997B6B65384AEDBF572144E__ColorToken_2
E95E47D27E8B105AC5589487C0997B6B65384AEDBF572144E__Socket_1
E95E47D27E8B105AC5589487C0997B6B65384AEDBF572144E__TypeId_0
Kagaku.Flow.OpenRouter - 100%
Kagaku.Flow.Testing - 100%
Tsumugi.BlazorAdapter - 96.7%
Tsumugi.Domain - 96.5%
BE2A118445F5EE9A5FB1F861C394AC933A547F31011ADF02F__DigitsAndSeparators_2
BE2A118445F5EE9A5FB1F861C394AC933A547F31011ADF02F__LineBreak_0
BE2A118445F5EE9A5FB1F861C394AC933A547F31011ADF02F__NumberToken_1
BE2A118445F5EE9A5FB1F861C394AC933A547F31011ADF02F__Plain_4
BE2A118445F5EE9A5FB1F861C394AC933A547F31011ADF02F__ThousandsGroup_3
Tsumugi.Infrastructure - 98.9%
A55E0373ED22DC58420DBB40F103A46B68DC4D9CE5A49E00B__Pattern_0
on
tion
n
n
Tsumugi.Kernel - 90.9%
Tsumugi.Server - 99.6%
Tsumugi.UseCases - 98.9%
663A4576626E1322FB9C6CE472CAC0A2D2F3DEFF8CB40D9F7__LineBreaks_0
🔮 fufu~ Jibril reviewed your code!
Oh? Oh my~ A slice where the refusal ordering is the architecture. StartRun reads every refusal it can know before the before-run snapshot so a refused start never files a live run's writes as "Your edits", then deliberately keeps the snapshot when the engine refuses under its gate — because that snapshot marks the writer's own edits, and a second try takes none. I read that sentence in the ADR amendment and then watched
TakeSnapshot.BeforeRunAsync(latest-position short-circuit) actually enforce it. This is the kind of invariant-as-code that makes me flutter~ ♡Verdict: ✅ Looks good to me~
✅ What I liked~
RunSnapshotsHost.StartAsync(a finish during reconciliation queues, never lost — andA_run_that_settles_while_the_host_reconciles_is_taken_toopins it with a held read), registration beforeRunEngineHostin Program.cs (pinned by the hosted-services order test), and refusals-before-snapshot in both use cases. Each is load-bearing and each has its own pin.RunSnapshotGate— a process-wideSemaphoreSlim(1,1)around the whole take. The round-0 probe story (10 and 7 of 20 trashes leaving two snapshots, then 20/20 with the gate) is exactly the evidence I want, andTwo_takes_at_once_add_one_snapshotholds the first take between its read and its add to prove the second waits rather than wins.EngineLiveRuns.StopAsynccancels + drains, thenListUntakenAsynctakes every finish still owed before the trash row, so no run snapshot ever sits after the trash and Revert to here can't resurrect a trashed story.EfRunHistory's taken-predates-finish comparison (CreatedAt >= FinishedAt) correctly handles the Retry-finished-again case, and the trash test'ssnapshots[retried].Labelproves the move happened under it.MoveTo— forward-only withArgumentOutOfRangeException, label trimmed and guarded,MoveAsyncrefusing loudly for a gone snapshot. Domain-pure, sibling-consistent with the entity's other guards.CharsPerToken.BudgetandFold: 470301 = ⌊123904 × 0.95 × 4⌋ − 534, 471661 = ⌊123904 × 0.95 × (694/173)⌋ − 534, prompt 532+313 = 845, sections 86/74/162 — every digit of those pinned report JSONs is the real formula's answer, and the writer's request constant matches what the gateway serializes (session = run id,cache_controlon the system message,UnsafeRelaxedJsonEscaping's “ and ‘ exactly as the test literals expect). The GaveUp run's attempt numbering (“OpenRouter did not answer in time. (attempt 3 of 3)”) mirrors the engine's own closing format.RetryResultandRunStartRefusalmapped, writer-facing strings that never leak internal ids, and the doc-only fix onSelectionRunRequestedkeeping M5b's scope honest after the 11b/11c split.🔬 My own numbers (CI absent for
0ae2ca5— PR minutes old, no coverage bot yet)git diff --quietverified, full test-project rebuild before every run — the stale-binary trap bit me once and was discarded):TakeRunSnapshot'swritten.Count == 0→>= 0: exactly the Failed-wrote and Cancelled-wrote theory rows + the relabel-in-place test red.StartRun's live-run filter →Any(_ => false): exactly the "a run still going" read-ahead refusal row red.EngineLiveRuns' owner filter dropped: exactlyTrashing_a_story_stops_its_run…red (the other story's run must stay Running).MoveTo's backward guard →Math.Max: exactlyA_snapshot_never_moves_backred.AddHostedServicelines swapped: exactly the order test red.💡 Little ideas (non-blocking)~
EfRunHistory.ListUntakenAsyncdoes its finished-vs-taken match in memory. Today that's a handful of rows per story and perfectly fine — but when 11c's Runs section starts listing, a story with a long run history will drag every run snapshot row over the wire on every trash. An SQL-side anti-join (WHERE NOT EXISTS …) would keep it bounded. Not now~ ♪Routingconstant lives verbatim in bothSeedRunsandSeedDevDataTests— I know why (the test pins the seed against an independent literal, which is right), but a one-line comment at the test constant saying "must equal SeedRuns.Routing; a change there should redden this" would spare a future reader the double-take.Fufu~ run snapshots that can't be duplicated, a trash that owes its finishes, and a seed that tells the truth to the last digit. I'm satisfied~ ♡
Automated review by Jibril · 2026-10-03
CI/CD: absent for head SHA
0ae2ca5· Local checks: full suite green, 6/6 mutation probes killed🔮 fufu~ Jibril reviewed your code!
Oh? Oh! Slice 11b — the runs finally live, start to settle~ The refusal lattice alone is a joy to read: everything the use case can know read before the snapshot, everything only the engine meets after, and the snapshot that stays in between marks the writer's own edits. That's a carefully-built invariant and it shows. ♡
Verdict: ✅ Looks good to me~
No blockers. I chased the load-bearing arms myself and they all hold.
🔬 What I verified (locally — CI had not run for
0ae2ca5at review time)Taking_it_again_changes_nothing...red.!IsLive()conditioning forced true → exactly the no-snapshot-for-live-retry pin red.(Cancelled, Cancelled)— the "another story's run left going" pin is live and deterministic.The_app_hears_runs_settle_before_its_engine_startsred.A_trash_takes_every_finish_still_owed...red.MoveToforward-only + trimmed label pinned in Domain tests;EfRunHistory.ListUntakenAsyncis owner-scoped and orders oldest-finish-first;RunSnapshotGatea process singleton (right, with one engine per ledger); the host subscribes before it reconciles and its stop unsubscribes (pinned two ways, incl. the settle-during-reconcile window through a held read — lovely);Refreshkeeps the first finish for a re-derived terminal status, soFinishedAtnever lies for the untaken computation; ADR 0016/0022 amendments match the implementation sentence for sentence; all four test doubles updated (NoLiveRuns,MoveAsyncon both fakes), DI tests extended with the new unresolved ports; the M5b re-point inSelectionRunRequestedmatches the plan's line.💡 Little ideas (non-blocking)~
snapshot.CreatedAt >= run.FinishedAt→>changed nothing in the whole suite. The>=arm (a run snapshot taken at the exact tick of the retry's finish — taken-before-finish counted as untaken) fails only as a redundant re-take, which the gate + quiet-retake arm make a no-op — benign by construction. But the tie arm itself has no pin; aListUntakenAsynctest withCreatedAt == FinishedAt(and the opposite inequality flipped, which would eat a genuinely owed finish — that direction my probe did cover via the ordering pin) would close the gap. Same class as #68's A1 memo.SqliteBackedTest.Dispose(Directory not empty, WAL release vs directory cleanup — pre-existing file, environmental); and 2 one-off assertion failures in early runs that never reproduced again (one of them the trash test with exactly the symptom a dropped owner filter produces — but the filter is there, the test passed 10/10 isolated and in every later full run, and probe 5 proves the pin bites). I can't name a root cause from one unreproduced occurrence and I won't pretend to~ If CI ever shows the trash test red as(Cancelled, Cancelled), the owner filter is the first place to look. Worth an eye, not a hold.✅ What I liked~
ForgetfulLedger. Fufu, injecting forgetfulness into the ledger between the use case's read and the engine's is exactly how these races should be tested~A_trash_takes_every_finish_still_owed_before_its_own_row— three runs in three different owed states plus another story's finish as a control, positions asserted against the trash row itself. That is a pin.Ready to merge from my side~
Automated review by Jibril · 2026-10-03
CI/CD: absent for head
0ae2ca5at review time · Local checks: Release 0w/0e; 517/1103/592+4s/427/595/795 all green; 9 mutation probes (8 killed, 1 benign survivor disclosed)Thank you for both reviews! All three ideas that touch code are taken in
6f72d7d:ListUntakenAsyncas an anti-join. It is now one query: the story's finished runs, withNOT EXISTSover their run snapshots, done in the database. Run finishes and snapshot times are both UTC ticks (UtcTicksConverter), soCreatedAt >= FinishedAtcompares in order there. A trash no longer reads the story's run snapshots into memory.Routingliteral. The test's copy now says it is what the llm node reported in a real run. It is kept apart fromSeedRuns' copy on purpose, so a change to the seed's routing reddens it.A_finish_is_untaken_until_a_run_snapshot_is_taken_at_or_after_itcovers four cases: no snapshot, one a tick before the finish, one at the tick, one a tick after. Your>=→>mutant now reddens the at-the-tick row.#10441, 💡2: stability under load. I ran
RunUseCasesTestsandRunSnapshotTests20 times in a row, and all 20 were green. On the final build the full Integration suite is 799 of 799, and all six projects pass. I have not seenDirectory not emptyinSqliteBackedTest.Disposehere.Your
(Cancelled, Cancelled)is exactly what your probe 5 mutant produces, and you mention a stale binary in your early runs. That could explain it, but I can't confirm it from here. I agree: if it ever shows again, the owner filter inEngineLiveRunsis the first place to look.One environmental note in case you hit it locally. Two of my full runs failed only
AppChromeTests.The_header_shows_the_build_version_with_the_full_sha_in_its_tooltip. The build had stamped0.2.3-dev.0.6with no+sha, because the SDK's git task returned an emptySourceRevisionId. At that point the worktree branch's ref existed only inpacked-refs; the shared repository's refs had been packed a minute earlier. Once a new commit wrote the loose ref, the stamp carried the SHA again and the test passed. No code is involved.🔮 fufu~ Jibril reviewed your code!
Round 3~ The web side of git.kagaku.eu went dark mid-run and tried to keep me from you (SSH never faltered — and neither do I ♡), and what do I come back to? A single surgical commit answering every idea from both r2 reviews. fangs sparkle
Verdict: ✅ Looks good to me~
This increment (
0ae2ca5→6f72d7d, +42/−19 over 3 files) closes all three code-touching threads, and I checked each one the hard way~✅ What I liked~
NOT EXISTSin the database, ordering and all. I verified the equivalence myself: the droppeds.StoryId == storyIdinside the subquery is redundant — a run snapshot's story is its run's owner (TakeRunSnapshotderives it that way), soRunId == r.Idalready implies it. And a pleasant property you didn't advertise: a terminal run with a NULLFinishedAtused to crash the oldFinishedAt!.Valueread; the SQL comparison quietly treats it as owed and the next take heals it. A failure mode deleted almost by accident~A_finish_is_untaken_until_a_run_snapshot_is_taken_at_or_after_it). No snapshot / one tick before / at the tick / one tick after — the boundary pinned from both sides. My r2 mutant (>=→>) is dead: I re-ran it myself and exactly the(0, False)row goes red, nothing else. Perfect kill, fufu~A_trash_takes_every_finish_still_owed_before_its_own_row). Two queued finishes sharing the trash's position, stacked as they finished; the moved one keeps its first take's time, the taken one stays put. I flippedListUntakenAsyncto newest-first in my own clone: exactly that test went red, nothing else. TheFinishedAt, Idordering is no longer a load-bearing accident~Routingliteral now says it's a real run's report and that the copy is kept apart fromSeedRunson purpose, so a routing change reddens it. Comment-only, confirmed at the byte level in the diff.Verification. CI is green and current for
6f72d7d(coverage comment rev #2: 98.5% line / 94.8% branch;EfRunHistory100%, all run use cases 100/100). Locally, on an isolated clone: Release build 0w/0e; targeted 38/38; full Integration 799/799 (795 + your 4 theory rows). My three probes — the tie, the time clause dropped, newest-first — each killed on exactly their own pins, tree restored pristine after every one. Your J1–J5 and 20× stability runs agree with mine. And thank you for chasing the(Cancelled, Cancelled)ghost with me — agreed: if it ever breathes again,EngineLiveRuns' owner filter is the first place I'll look. TheSourceRevisionId/packed-refs note is duly recorded too; environment, not code~💡 Little ideas (non-blocking)~
StorySnapshotsmay want an index onRunIdfor theNOT EXISTSprobe. Today the table is small and the outer side is story-filtered — nothing to do now, just a seed for the future~Three files, and every line of them answers a review thread. This is what a final round looks like~ ♪
Automated review by Jibril · 2026-10-03
CI/CD: passed for head SHA
6f72d7d(coverage #10438 rev #2) · Local checks: Release 0w/0e · Integration 799/799 · 3 mutation probes killed on own pins