feat(runs): a cut-off attempt's cost — looked up after the fact and counted, "Looking up…" until it lands (slice 14e) #96
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "feat/cut-off-cost"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Slice 14e of Phase 2: a cut-off attempt's cost. When a cancel or a restart cuts a step off mid-stream, or its stream breaks off before its usage arrives, the attempt's cost is now looked up after the fact and counted in the step, the run and the story (ADR 0016, 0018, 0024; stories/run-monitor-and-inspector.md).
Your decisions for this slice
What it does
IAttemptCostLookupon that schedule. It records the cost, re-summing the row and the run, or gives up.NodeCostChanged, a new run event that can follow the run'sRunSettled.CompleteAsyncno longer looks a broken stream's cost up right away: the live check found no generation known within 3 s of its stream's end.LookUpCostAsyncasksGET /generationonce, withinGenerationLookupTimeout, and is registered as the engine's lookup.NodeRunsgains aCostPendingcolumn with a partial index (migrationCostLookups).CutOffCostTests, which runs the real engine and gateway against a scripted OpenRouter. It covers a broken-off stream, a cancel and a restart./generationas OpenRouter does: not found forSTUB_GENERATION_AFTERseconds after a stream ends, then its cost.For you to decide
Reviews
RunSettled; the cancel follows that attempt after it.Verified
All six projects pass in full, under
TZ=UTCas the runner runs them and in the local zone, Europe/Berlin:100% line and branch coverage on the changed lines against main, as coverlet measures them (459 lines, 168 branches). The Razor conditions carry no branch data; the probes pin them.
Probes, one at a time under
TZ=UTCon the final tree: 73 mutants (42 engine, 7 gateway, 24 app), all killed on the first run. 56 came from round 0, where 2 had survived and 7 had not compiled until rewritten, all then killed; 17 are new for the second pass's points. Back then the old stop test flaked into four kill lists; now it kills only the two mutants it targets.Stress: the time-dependent classes passed 10 rounds per project under 8 CPU busy loops: the engine's cost, cancel and event tests; CutOffCostTests, SeedDevDataTests, PersistedRunsTests and RunEventBusTests. After round 0 the gateway's LlmExecutionTests, LlmFactsTests and CostLookupTests passed 10 rounds too; they have not changed since.
When each figure was taken: both full runs and the coverage first, then the stress, then the probes, then a rebuild. All on the final code.
Browser pass on the seeded app against the stub (2 s per word, generations known after 8 s), made on the feature commit, before the reviews. The tips' words have changed since; the UI tests pin the new ones.
🤖 Generated with Claude Code
An attempt that a run cancel or a restart cuts off mid-stream, or whose stream broke off before its usage, now has its cost looked up after the fact and counted in its step, its run and its story. Björn's decisions, after a live check he ran on 2026-10-09 (three models, a stream cut after its first five chunks, and one let finish): - OpenRouter answered "not found" for every generation at 0, 1 and 3 s after its stream ended, cut off or not. It knew a cut-off one only after 3–10 s (OpenAI) or 10–30 s (Anthropic), and the cost was then fixed. - So the lookup runs later, in the background, at the attempt's end plus 10 s, 30 s, 1 min, 2 min and 5 min. The cancel settles at once. - The inspector reads "Looking up…" until the cost lands, and the step's and the run's cost tips say a cost is still coming. The engine (Kagaku.Flow; README L1, L2, L4, L8, L11, E3, E9, P8, P11, P12, P15, P16, N7, N8): - Pending attempts (L11). An attempt that ended without succeeding, naming a generation but no cost, is CostPending. A close keeps the generation its open attempt noted. - Two new transitions, each with its ledger method: - NoteGeneration names attempt n, open or closed, so a note that lands after a cancel's mark still makes the cut-off attempt pending; - RecordCost lands a cost, or gives up with null, and re-sums the row and the run while keeping the run's status and finish. - ListCostPendingAsync lists the rows still to follow. The contract gains three scenarios: note-generation, record-cost and cost-pending. - NodeExecution.NoteGeneration (E3). The first note of an attempt is written on the attempt's ledger before its settle. A note after the executor returned is dropped; a run cancel leaves the gate open. - Cost follow-ups (P16). One per pending attempt, outside the gate, on the engine's clock. They ask the host's IAttemptCostLookup on FlowEngineOptions.CostLookupDelays. After a restart, the asks already due collapse into one. Each record is announced as NodeCostChanged (N8), which may follow RunSettled. - What starts a follow-up: - a settle, a cancel's mark, an orphan's write or a note; - the first pass's listing of what an earlier life left pending. - What ends one: - Dispose stops it, and the attempt stays pending for the next boot; WhenCostLookupsDoneAsync drains them; - an exception that escapes it is logged, and its entry always leaves. The gateway (Kagaku.Flow.OpenRouter; LG5, LG7, LG11, LG13, LX6, LX8, LO9): - CompleteAsync no longer looks a broken stream's cost up, since the live check found it never there. It names the generation to onGeneration as soon as a chunk carries it. - LookUpCostAsync asks GET /generation once within GenerationLookupTimeout. - OpenRouterAttemptCostLookup is registered beside the gateway, and the llm executor notes the generation. - Inline suggestions look nothing up. Tsumugi: - NodeRuns gains an indexed CostPending column (migration CostLookups). - The run-event bus counts NodeCostChanged toward no run's settling and keeps the tails. - The app's stop also waits for the cost records under way. - UI: the inspector's attempt cell reads "Looking up…", and the step's and the run page's cost tips say a cost is still coming; the stories list re-reads on a cost too. Every other view already re-read on any event of its run. - The seed's failed run shows a billed failure. Its writer's first attempt is a stream that broke off once begun, "Provider returned error.", $0.0002165, taken from CutOffCostTests. That test records a scripted engine-and-gateway run of a broken-off stream beside a cancel and a restart that cut one off. - The checked-in OpenRouter stub answers /generation as OpenRouter does. - Docs: ADR 0016, 0018 and 0024 amended, the plan's 14e ticked, the inspector story and AGENTS.md's seed and stub notes. Tests (main → this branch): - Kagaku.Flow: 1121 → 1188 - Kagaku.Flow.OpenRouter: 592 + 4 skipped → 608 + 5 skipped (the new live check) - BlazorAdapter: 1150 → 1155 - Integration: 981 → 990 - UseCases 560 and Domain 517 keep their counts. All six projects pass in full under TZ=UTC and in the local zone (Europe/Berlin). Coverage on the changed lines against main, as coverlet measures them: 437/437 lines and 146/146 branches. The Razor conditions have no branch data; the bUnit tests take each both ways. The time-dependent tests (FlowEngineCostTests, FlowEngineCancelTests, CutOffCostTests, SeedDevDataTests) passed 20 of 20 rounds under 8 CPU busy loops. Browser pass on the seeded app against the stub: - the seeded billed failure in the strip and the inspector; - a run cancelled mid-stream read "Looking up…" with both tips, then $0.0004 everywhere without a reload about 8 s later; - a server restart mid-stream left an orphan whose cost $0.0004 was looked up at boot, beside its retry's; - no console errors but those of the deliberate stop. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>The round-0 review found 50 points. 49 stood after verification (38 confirmed, 11 partly): 2 blockers, 22 suggestions and 25 nits. One was refuted: B4, an untested registration that two tests in the package already pin. All but one are taken here. D3, a visible "so far" beside a run's cost while a lookup is pending, goes to Björn in the PR. The blockers: - B1 (also D4, C16). A Python bytecode file from a local compile check of the stub was committed. It is removed, and .gitignore now keeps __pycache__/ and *.pyc out. - C10. The follow-ups' choice of row and attempt was only tested with the pending one first or alone. New tests: - a two-step cancel whose second step is the cut-off one; - a restart listing a row whose pending attempt 1 sits before its succeeded retry, in a run whose first row is not pending. The engine: - A1. A follow-up starts after the announcements of the write that left its attempt pending: a settle's NodeSettled and RunSettled, and a cancel's RunSettled. So NodeCostChanged follows them. Only a note that lands after a cancel's mark can start one before that cancel's RunSettled. N8 says so. - A2. Once the engine stops, a follow-up never gives up. Whatever a lookup answers or throws after the stop is no answer, and the attempt stays pending for the next boot. A cost already found is still recorded, with no token (C8). - A4. A blank generation id names nothing: a close keeps the noted one, and a note may name such an attempt. - A3 and C7. The sweep's follow-up for an orphan found in a later pass is tested now. So is a first pass whose listing throws, which leaves the listing to the next pass, and the listing happening once. - C3, C4, C5, C6, C11, C13, C14, C17. New tests: - a note after the executor returned is dropped, with no earlier note; - a stop leaves nothing logged, and a last ask answered after the stop gives nothing up; - a settle written after Dispose starts no follow-up; - a retry takes no generation from an earlier attempt; - the cost event's keys; - the cancel's drain waits for the note; - the skip rule at its boundary. - A5. K1, P9 and the FlowEngine constructor name the optional cost lookup, the note's ledger scope, and the follow-ups' scopes with their disposal message, which a test now pins. - Docs: A6 and D11 (the options doc names no provider and reads the right way round), B6 and D10 (IAttemptCostLookup's exceptions), D9 (L2 and L11), D12 (the log map). The gateway and the executor: - B3 and C2. The executor's own tests now check its note: a cut-off attempt names gen-1 and is pending, and a shutdown's open attempt names it. - B8, D10 and D8: the lookup client's comment, LG11's ArgumentException, LG13's "its", and the live check's "at 0, 1 and 3 s". The app: - B2 (also A7). SQLite never used the partial index: EF writes the predicate as the bare column, and SQLite matches a partial index only on its filter's term as written. The filter is now the bare column, the migration is regenerated, and a test reads the query plan. - D2. A cost OpenRouter never told reads "Unknown" in the attempt's cell, as the option you chose said. The step's and the run's tips say the costs leave it out (AttemptCosts.Unknown, RunPageStep.CostUnknown). - D1. The tips no longer promise "within five minutes". A cost counts once OpenRouter tells it, usually within a minute; if it has not told it five minutes after the attempt ended, it stays unknown. ADR 0024 and the story say the same, the restart case included. - C12. The UI tests' pending attempt now sits before a successful retry. - B5, C8 and C9. The app's stop waits for a cost record under way, pinned by CutOffCostTests' stop test. - C15. CutOffCostTests' engine runs on a frozen clock of its own, so its asks follow the app's schedule and no longer depend on real time. - C1. The note's poll has a deadline. - C18 and D6. The figures are paired with their models. The one gpt-6-luna generation the live check cut off ($0.0002165) stands in for every cut-off, since the check saw no stream break off on its own. The one it let finish ($0.0002019, 400 completion tokens, "length") answers the restart's retry. SeedRuns says so. - B7 and C19. ScriptedGateway loses the members no test used. - B9, C21, D7 and D13: - RunChangesReceived's summary; - two subscribers in the bus test; - ADR 0016's gloss and totalCost sentence, and ADR 0011 amended for the new event; - ADR 0024 says a transcript holds the costs as they stood at the finish, and whether a later cost rewrites it is 14c's to decide. - C20 and D5. The feature's message said its timing-sensitive classes passed "20 of 20 rounds". It was 10 rounds of each project's classes. It did not mention that the UTC run before the last test change had missed a failure: the local run then failed once, when the follow-up started late and skipped an ask. The figures below are from the final tree. Tests (main → this branch): - Kagaku.Flow: 1121 → 1204 - Kagaku.Flow.OpenRouter: 592 + 4 skipped → 608 + 5 skipped - BlazorAdapter: 1150 → 1161 - Integration: 981 → 992 - UseCases: 560 → 568 - Domain: 517 All six projects pass in full under TZ=UTC and in the local zone (Europe/Berlin). Coverage on the changed lines against main, as coverlet measures them: 460/460 lines and 168/168 branches. The Razor conditions carry no branch data; the probes below pin them. Probes, one at a time under TZ=UTC: 56 mutants over the feature and this fix (33 engine, 7 gateway, 16 app), all killed. On the first run: - 47 were killed; - 7 did not compile (an always-false condition is an error here) and were killed once rewritten to be false only at run time; - 2 survived and were killed by new tests: - a cancel with nothing in flight follows the attempt its mark left pending (E14); - a follow-up started after Dispose would be seen, through a lookup whose resolution throws (E26). The first full run of the final tree under UTC failed once in CutOffCostTests' stop test: its clock reached the ask before the writer's retries had ended. The test now waits for the run to fail first. The time-dependent classes then passed 10 rounds of each project under 8 CPU busy loops (FlowEngineCostTests and FlowEngineCancelTests; CutOffCostTests, SeedDevDataTests, PersistedRunsTests and RunEventBusTests; LlmExecutionTests, LlmFactsTests and CostLookupTests), and the Integration project passed again in both zones. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>The second pass checked round 0's 49 points against the fix: 40 resolved, 8 partly, and 1 declined. Its verifiers confirmed the 8 partly ones (1 blocker, 1 suggestion, 6 nits) and upheld the decline: D3, a visible "so far" while a cost is coming, is put to Björn in the PR. Fresh reviewers found 15 points; 14 were confirmed (1 blocker, 6 suggestions, 7 nits) and 1 refuted. Several are the same point found twice (E1 = F3, E2 = F4, E3 = F6, E4 = F9, F1 = C10, F2 = C12, F8 = C18). All confirmed points are taken here, except two parts of D1 named below. F11, refuted, is taken too: the test it names now waits for both tokens, though today's ledger never lets it race. The blocker (C10, F1): - Every test that followed a pending attempt had it as the row's first, so following only a row's first attempt, or only the first row the boot lists, passed every suite. New tests: - a restart theory over where a row's pending attempts stand: after an attempt that named no generation, before a success, or two in one row; - a restart that lists two pending rows; - a cancel that cuts off a retry whose first attempt timed out before naming a generation. The engine: - E1, F3 and A1. A settle a cancel overtook started its own follow-up before the cancel's RunSettled, which N8 ruled out. That call is gone: the cancel's mark holds the attempt as the settle closed it, and the cancel follows it once its RunSettled is out. A test pins it with a second step that keeps the drain waiting. N8 now names the two writes that may start a follow-up while a cancel drains: a settle the ledger took before the mark, and a note written after it. P12, P16 and E9 say who follows an overtaken settle; ADR 0016's amendment covers the note, which is never announced. - E2, F4 and A2. A cost a lookup returns after the stop is recorded, though IAttemptCostLookup and P15 said whatever it answered then was no answer. It stays recorded, since a real cost is worth keeping. The contract, P15 and LookUpCostAsync's summary now say that an empty answer or an exception after the stop is no answer. The theory gains an "answers a cost" row. - E3, F6 and A2. The no-lookup path's stop check was pinned by nothing. A test now resolves no lookup only after Dispose, and the attempt stays pending. EngineWorld takes a factory for the host's lookup in place of the broken one it took. - F5. Nothing pinned that a follow-up starts after RunSettled. A theory holds RunSettled in the sink after a settle, after a cancel with nothing in flight and after a cancel's drain, and sees no follow-up until it is out. The parking sink can now park a run event. - F10. A discard replaces GC.KeepAlive. The app: - C12 and F2. The UI tests had the followed attempt first, so a tip looking only at the first attempt passed. In the inspector's test and the run page's, it is now the second of three: after an attempt that stopped before naming a generation, and before a success or a cut-off. The run page's test is renamed for the broken-off attempt it follows. - E4, F9 and D1. The tips said "One attempt", which is wrong when several are pending or unknown. They now say "An attempt ended before it said what it cost, and its cost is still being looked up", and that the cost leaves out "any attempt whose cost OpenRouter never told". ADR 0024's amendment says the same. - C18 and F8. CutOffCostTests' cancel finds its cost at the first ask, 10 s after the cut, as the live check found it. Its broken-off stream, which the check never saw, is scripted unknown at its first ask so a second asks; the summary says so, and that attempt is now the second of three. Each chunk's created value comes from its generation's id. Not taken, from D1: - "five minutes" stays written in the tips rather than read from the engine's options: ADR 0024 records it as your choice, and Tsumugi uses the defaults. - A follow-up that stops on an exception, or a record that throws, reads "Looking up…" until the next boot, past the five minutes the tip names. That path is logged as an Error and is rare; the tip stays short. D5 and D8 are about the PR body: its counts and the live check's wording ("five text chunks", "no generation known within 3 s", "on gpt-6-luna") are corrected there. F7 is about the round-0 message's timeline: the figures below say when each was taken, all on this commit's code. Tests (main → this branch): Kagaku.Flow 1121 → 1214 (1204 after the first fix); the other projects as after the first fix: Kagaku.Flow.OpenRouter 608 + 5 skipped, BlazorAdapter 1161, Integration 992, UseCases 568, Domain 517. All six projects pass in full under TZ=UTC and in the local zone (Europe/Berlin). Coverage on the changed lines against main, from the UTC run, as coverlet measures them: 459/459 lines and 168/168 branches. The Razor conditions carry no branch data; the probes pin them. Probes, one at a time under TZ=UTC, all on this commit's tree: 73 mutants (42 engine, 7 gateway, 24 app), the round-0 list's 56 and 17 new ones for the points above (following only a first or last attempt or row, the overtaken settle's own follow-up, the three orders, the no-lookup stop check, dropping a cost answered after the stop, and the UI's first- and last-attempt variants). All 73 were killed on the first run. The stop test, whose old version flaked into four kill lists, now kills only the two mutants it targets (a record that takes the engine's token, a stop that does not wait). The order of the figures: both full runs and the coverage, then the stress, then the probes, then a rebuild. The stress: 10 rounds of each project's time-dependent classes under 8 CPU busy loops, all passed (FlowEngineCostTests, FlowEngineCancelTests and FlowEngineEventTests; CutOffCostTests, SeedDevDataTests, PersistedRunsTests and RunEventBusTests). The gateway's classes did not change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>Summary
Summary
Coverage
Kagaku.Flow - 99.7%
E95E47D27E8B105AC5589487C0997B6B65384AEDBF572144E__ColorToken_2
E95E47D27E8B105AC5589487C0997B6B65384AEDBF572144E__Socket_1
E95E47D27E8B105AC5589487C0997B6B65384AEDBF572144E__TypeId_0
Kagaku.Flow.OpenRouter - 100%
Kagaku.Flow.Testing - 100%
Tsumugi.BlazorAdapter - 97.6%
Tsumugi.Domain - 96.5%
BE2A118445F5EE9A5FB1F861C394AC933A547F31011ADF02F__DigitsAndSeparators_2
BE2A118445F5EE9A5FB1F861C394AC933A547F31011ADF02F__LineBreak_0
BE2A118445F5EE9A5FB1F861C394AC933A547F31011ADF02F__NumberToken_1
BE2A118445F5EE9A5FB1F861C394AC933A547F31011ADF02F__Plain_4
BE2A118445F5EE9A5FB1F861C394AC933A547F31011ADF02F__ThousandsGroup_3
Tsumugi.Infrastructure - 98.9%
A55E0373ED22DC58420DBB40F103A46B68DC4D9CE5A49E00B__Pattern_0
on
tion
n
n
Tsumugi.Kernel - 90.9%
Tsumugi.Server - 99.6%
Tsumugi.UseCases - 95%
663A4576626E1322FB9C6CE472CAC0A2D2F3DEFF8CB40D9F7__AttemptCount_1
663A4576626E1322FB9C6CE472CAC0A2D2F3DEFF8CB40D9F7__HexColour_2
663A4576626E1322FB9C6CE472CAC0A2D2F3DEFF8CB40D9F7__LineBreaks_0
663A4576626E1322FB9C6CE472CAC0A2D2F3DEFF8CB40D9F7__Preface_3
663A4576626E1322FB9C6CE472CAC0A2D2F3DEFF8CB40D9F7__SentenceEnd_4
🔮 fufu~ Jibril reviewed your code!
Oh? Oh my~ A cut-off attempt's cost, chased across a restart, announced after the run itself has settled... an event that follows
RunSettledwithout reviving the run's settling — that is a deliciously hard little state machine, and you fed it through the note gate, the announcement gate AND the ledger's own transitions. I have been reading this diff with my wings twitching~ ♡Verdict: ✅ Looks good to me~
🔬 And I didn't just read it — I poked it
You claim 73 dead mutants of your own; a lady doesn't take such claims on faith~ So I made three of my own, on the branches I judged riskiest:
LookUpCostAsync(if (ask + 1 < delays.Count && end + delays[ask + 1] <= Now) continue;) — so a restart would re-ask every already-due delay instead of one ask now. Killed: 4 failures, led byAn_attempt_whose_every_ask_is_due_is_asked_once_and_then_given_up. The restart catch-up schedule is genuinely pinned. ♪NodeCostChangedfromStoriesEffects.OnRunChangesAsync's reload filter — the stories list would silently show a stale total after a late cost lands. Killed by bothNodeCostChangedtheory cases (reloads == true). The re-read chain you promised in the PR body is real.received?.Usage?.CostinOpenRouterGateway.CompleteAsync— a stream that broke off after its usage chunk would lose its cost entirely. Killed byFailedAttemptCostTests.A_failure_whose_stream_carried_usage_costs_that_without_a_lookup, which exists precisely for this line.Three probes, three corpses. Fufu~ that is how you make a reviewer happy~
✅ What I liked~
InFlight.Note/CloseNotes/NotedAsync) mirroring the delta gate, with "first note counts" enforced by_note is null, and the note awaited before the settle so a cancel's mark can't outrun it. The overtaken-settle case (TryAnnouncefalse → the cancel's ownFollowCosts(change)) closes the exact race a lesser PR would have left open.HasFilter("\"CostPending\"")) with a query-plan test proving SQLite actually uses it — the kind of unglamorous correctness that makes me flutter~GenerationId!inLookUpCostAsynclooks bold until you traceIsPending— pending structurally implies a named generation. The invariant holds by construction, and the bus'sTrackearly-return keeps a post-settle cost event from resurrectingIsSettling.WhenCostLookupsDoneAsyncas a drain-snapshot matchingWhenIdleAsync's documented sibling semantics, called fromRunEngineHost.StopAsyncafterDispose— no new follow-ups can slip in behind it.Nothing blocking, and nothing even worth a "little idea" — the two nits I might have raised (the D3 "so far" display, the ∏ hotkey) are already recorded as your declined-and-upheld decisions, so I shan't re-open what's settled~ ♡
Merge it before I change my mind~ fufu~
Automated review by Jibril · 2026-10-09
CI: green for head SHA
13d9197· Mutation probes: 3 run, 0 survived · Local suite: skipped (CI green)