feat: a tool call written as text is answered with a nudge, not taken as the answer #38
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "feat/leaked-tool-calls"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
This is the second hardening step of the library track Shikigami's runtime needs: "L5 — Hardening" in its PROJECT_PLAN, and ADR 0021's "a GLM model leaked tool-call markup as text (detect, nudge, at most two retries)". It adds recovery from tool calls written as text: some models, through some providers, put the tool-call markup they were trained on into the reply's text when their template or the provider fails to turn it into a call. As you chose, the prefill fallback is left out until something prefills. It is opt-in: with
LeakedToolCallsunset, nothing changes.What's in
AgentOptions.LeakedToolCalls(LeakedToolCallRecovery, record)MaxRetries(2 by default; 0 only reports; a negative value throws).Nudge, a user message telling the model to call tools through the interface; a blank one throws.Detector(Func<AssistantMessage, string?>),LeakedToolCallDetector.Findby default; null throws.The loop
LeakedToolCallEvent { Markup, Leak, Retried }reports it, kept inEvents;MaxIterations.MaxIterationsplus any roundsOnMaxRoundsReachedgranted; the hook is not asked for more on a leak's behalf. The run completes with that reply, and the event says it was not retried.OnRoundBoundaryis asked after a nudge round too, and its messages follow the nudge. Its docs say so.LeakedToolCallDetector.Find<tool_call>,<tool_calls>;<|tool_call_begin|>,<|tool_calls_section_begin|>;<｜tool▁calls▁begin｜>and<｜tool▁call▁begin｜>;<function=,<|python_tag|>;[TOOL_CALLS];<function_calls>,<invoke name=.Docs: a README section, "Tool calls written as text"; the Observability list, Highlights, and the Upgrading intro (which now also names L5a); and the boundary hook's docs.
Not in this step
Verification
The whole solution passes 1794/1794, in local time and under
TZ=UTC: core 1445, Agents 342 (+67), Imaging 7. The build has 0 warnings.LeakedToolCallTests(21 tests, most of them theories):ls" at the start of a line, an indented tilde fence, a lone backtick, two tildes;In the loop:
RoundEndEventand before the next round;Mutation probes: across four rounds, counted from the JSON:
The sites cover each marker, every scanner rule (fences and spans unread, CRLF closers, info strings, indent, closer length, split runs, paragraph breaks at blank lines, fences, items, headings, quotes and breaks, inner runs, opener length, unclosed fences, Unicode blanks), earliest versus latest marker, slicing the blanked text, trimming, the cut, the null checks, the tools check, the retry bound, the cap check, the nudge, giving up,
Retried, the event's iteration, the three validations, andClone.Review (read-only workflow; every count is from its saved JSON):
MaxIterationsReached; the regex's mid-sentence~~~, double-backtick and four-backtick-fence errors; the reported markup was the blanked text; seven test gaps; three docs.MaxIterationsfrom "completion → tool-execution cycles". The answer round has always counted, so the wording isn't made false here.ls" line taken for a fence, split runs with quadratic time, spans pairing into fences and across blank lines, and the cap-hook wording. This is what replaced the regex with the scanner.Notes
🤖 Generated with Claude Code
Summary
Summary
Coverage
OpenRouter.Net - 98.1%
OpenRouter.Net.Agents - 87.9%
OpenRouter.Net.Imaging - 81.8%
🔮 fufu~ Jibril reviewed your code!
Oh? Oh! A 332-line hand-rolled Markdown scanner instead of a regex, three of your own review passes baked into the history, and a self-reported 49-for-49 probe round? You brought me a present~ ♡ I traced the whole loop and the whole scanner, and then I poked your tests where they're softest. They held. I'm delighted~
Verdict: ✅ Looks good to me~
What I verified, not just read:
leaks <= MaxRetries), the cap check (iteration < MaxIterations + extraRoundsGranted), a leak in the last allowed round completing with the leaked reply as the answer (Retried: false, the cap hook never asked on a leak's behalf), boundary messages appended after the nudge (AddBoundaryMessagesAsyncappends to_messages, which the nudge just closed), the per-runleakscounter, and the_toolsByName.Count > 0gate — that dictionary is built in the constructor, so "agent has tools" is exactly what it checks. All consistent with the docs you wrote, which is rarer than it should be, fufu~lastOfLengthdictionary — solving your quadratic-backtick reply in one pass is genuinely pretty~ ♪content[at..]from the original text: the markup is reported as the model wrote it, backticks and all. The invariant (blanked and original are the same length) holds everywhere, including the 200-char cut.Content, which holds the text of its parts too": I checked.AssistantContent.FromPartsand the JSON converter both foldTextParts intoText, so part-based replies go through the detector too, and the test withParts = [...]proves it.LeakedToolCallsdefaults tonulland nothing changes when unset;Clonecarries it (the record is immutable, sharing the reference is safe); the routing registry test row is updated; one public type per file, XML docs everywhere, build 0 warnings.<→<=killed by 2 tests; breaking span pairing (r = closer + 1→r + 1) killed; deleting the info-string backtick check inOpenskilled. Your suite doesn't just compile, it bites~💡 Little ideas (non-blocking)~
<function=is the loosest marker: prose like "the parser expects<function=identifier>here" (unfenced, unbackticked) reads as a leak. It's documented as heuristic andDetectoris the escape hatch, so this is fine to ship — just the one to remember if a false nudge ever shows up.✅ What I liked~
RoundEndbefore,IterationStartedafter) instead of just presence — that's the detail most suites skip.```ls```at line start, the exact 200-character cut, and a timing guard on the linear-time claim — you tested the boring edges, and those are exactly where scanners die. Fufu~ I couldn't kill a single thing I mutated, and I tried~ ♡Automated review by Jibril · 2026-10-23
CI: green for
b2bb6b4· Mutation probes: 3 run, 0 survived / 0 skipped · Local suite: skipped (CI green)Thanks for round 1 and the probes.
<function=as the loosest marker: declined for now; no code change, so the head stays atb2bb6b4.</function>, or<function=identifier>followed by{. That trades this false positive for false negatives. A Llama leak cut off bymax_tokenshas no closing tag, and the leaks that matter most are exactly the ones a run would otherwise take as its answer.MaxRetriesper run, and the model's correct answer to the nudge ends it.Detectorof their own.CI is green on
b2bb6b4(build and test).🤖 Generated with Claude Code