A reasoning segment's unknown fields are dropped on the way back to the provider #10

New issue

Closed

opened 2026-07-30 12:11:07 +02:00 by scarlet · 1 comment

scarlet commented

2026-07-30 12:11:07 +02:00

Member

Follow-up from #9 (💡2 there).

What happens

ReasoningDetail and its three subtypes model type, the payload (text / summary / data), index and format. Anything else a provider sends on a segment is read and discarded, because System.Text.Json ignores unmapped properties by default.

That is harmless on the way in. It is not harmless on the way out: Agent collects the segments and passes them back on the next turn to preserve thinking state, so whatever the provider attached to its own thought does not come back to it.

The concrete case is Gemini's signature, seen in the field on google/gemini-3.5-flash:

{"type":"reasoning.text","index":0,"signature":"Cs4CAcv3..."}

That segment now deserializes (#9), and it is echoed back as {"type":"reasoning.text","index":0} — signature gone.

What is not yet known

Whether Gemini rejects, degrades, or silently ignores a thought whose signature it issued and did not get back. I have not tested it, which is why #9 deliberately stopped at the crash and left this open rather than guessing at a fix.

Two ways to fix it

An explicit Signature property on ReasoningTextDetail (and wherever else it appears). Discoverable and typed; only solves the field we happen to know about.
[JsonExtensionData] on ReasoningDetail — a Dictionary<string, JsonElement> catch-all that survives a round-trip. Solves signature and every future field a provider invents, at the cost of a bag of untyped values on a public type. Worth checking it composes cleanly with [JsonPolymorphic], since the discriminator must not land in the bag.

My preference is (2) for a passthrough client, with (1) layered on top if the signature turns out to be load-bearing enough to deserve a name. Either way the right first step is a test against a real Gemini turn: send a thought's signature back, then omit it, and compare what the provider does.

🤖 Generated with Claude Code

Follow-up from #9 (💡2 there). ## What happens `ReasoningDetail` and its three subtypes model `type`, the payload (`text` / `summary` / `data`), `index` and `format`. Anything else a provider sends on a segment is read and discarded, because `System.Text.Json` ignores unmapped properties by default. That is harmless on the way in. It is not harmless on the way out: `Agent` collects the segments and passes them back on the next turn to preserve thinking state, so whatever the provider attached to its own thought does not come back to it. The concrete case is Gemini's **`signature`**, seen in the field on `google/gemini-3.5-flash`: ```json {"type":"reasoning.text","index":0,"signature":"Cs4CAcv3..."} ``` That segment now deserializes (#9), and it is echoed back as `{"type":"reasoning.text","index":0}` — signature gone. ## What is not yet known Whether Gemini rejects, degrades, or silently ignores a thought whose signature it issued and did not get back. I have not tested it, which is why #9 deliberately stopped at the crash and left this open rather than guessing at a fix. ## Two ways to fix it 1. **An explicit `Signature` property** on `ReasoningTextDetail` (and wherever else it appears). Discoverable and typed; only solves the field we happen to know about. 2. **`[JsonExtensionData]` on `ReasoningDetail`** — a `Dictionary<string, JsonElement>` catch-all that survives a round-trip. Solves `signature` and every future field a provider invents, at the cost of a bag of untyped values on a public type. Worth checking it composes cleanly with `[JsonPolymorphic]`, since the discriminator must not land in the bag. My preference is (2) for a passthrough client, with (1) layered on top if the signature turns out to be load-bearing enough to deserve a name. Either way the right first step is a test against a real Gemini turn: send a thought's signature back, then omit it, and compare what the provider does. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

scarlet referenced this issue

2026-07-30 12:11:28 +02:00

fix: a reasoning segment without its payload no longer fails the completion #9

scarlet commented

2026-07-30 13:02:44 +02:00

Author

Member

Measured against live google/gemini-3.5-flash through OpenRouter, with raw HttpClient rather than this library — a typed client that drops unknown fields is the one instrument that cannot see them.

This is load-bearing, and worse than untidiness: dropping the handle makes the model abandon its task.

What the wire actually carries

A normal reasoning segment from this model:

{"type":"reasoning.encrypted","data":"AY89a18GGWWbsm81…(588 chars)","format":"google-gemini-v1","id":"bheu85ws","index":0}

id is the unmodelled field — an 8-character handle. Five request shapes (effort low/high × tool-forced/plain, plus no reasoning option) produced nine segments: reasoning.text with text, reasoning.encrypted with data, and no payload-less segment anywhere. So on the happy path nothing is missing, which is why the single-round test showed no difference at all.

The experiment that found it

Two identical five-round tool-calling conversations, differing only in whether each echoed assistant turn keeps id:

arm	payload-less segment appeared	model still calling tools at round 5
keeps `id`	0 / 8	7 / 7 observed
strips `id` (this library today)	7 / 8	1 / 8

Eight paired trials, seven discordant, all in the same direction — a sign test puts that near p ≈ 0.008. One keeps-id arm ended before round 5 and I did not capture why, so I am counting it as unknown rather than as a clean pass.

The mechanism, and the link to the original crash

When the handle does not come back, Google answers with:

{"type":"reasoning.text","signature":"AY89a1+3GnntT6F…","format":"google-gemini-v1","index":0}

finish_reason: stop, zero tool calls, and a final answer that gives up on the task ("a minor background texture or hatching element"). Two things follow:

signature is real after all. I had inferred it from Gemini's docs and #9's fixture guessed the shape {"type":"reasoning.text","index":0,"signature":"…"} — that turns out to be the exact wire shape, but it only appears in the degraded state, which is why the happy-path sweep never showed it. Both reasoning.encrypted.id and reasoning.text.signature are dropped today.
This is the cause of the crash in TeamAI/Orihon#119. The payload-less reasoning.text that failed deserialization was not a quirk of Gemini — it was Gemini's response to us having thrown its handle away. #9 stopped the crash; it did not stop the cause. Post-#9 the round parses, and the agent quietly abandons its refinement instead.

The second half is the part that matters for Orihon: a silent behavioural regression on a multi-round stage is harder to notice than a failed round, and a boxing agent that stops calling crop and declares the region "background hatching" produces a plausible wrong answer rather than an error.

Fix

[JsonExtensionData] on ReasoningDetail, so every unmodelled field survives the round-trip — id, signature, and whatever the next provider invents. An explicit Signature/Id pair would fix today's two fields and leave the next one to be found the same expensive way.

Worth checking as part of it that the extension bag composes with [JsonPolymorphic] without the type discriminator landing in it, and that a segment we constructed ourselves (no extension data) still serializes clean.

Reproduction scripts (three file-based probes: single round-trip, shape sweep, multi-round A/B) are in my scratchpad — say the word and I will attach them here or fold them into the repo as a manual diagnostic.

🤖 Generated with Claude Code

Measured against live `google/gemini-3.5-flash` through OpenRouter, with raw `HttpClient` rather than this library — a typed client that drops unknown fields is the one instrument that cannot see them. **This is load-bearing, and worse than untidiness: dropping the handle makes the model abandon its task.** ## What the wire actually carries A normal reasoning segment from this model: ```json {"type":"reasoning.encrypted","data":"AY89a18GGWWbsm81…(588 chars)","format":"google-gemini-v1","id":"bheu85ws","index":0} ``` `id` is the unmodelled field — an 8-character handle. Five request shapes (effort low/high × tool-forced/plain, plus no reasoning option) produced nine segments: `reasoning.text` with `text`, `reasoning.encrypted` with `data`, and **no payload-less segment anywhere**. So on the happy path nothing is missing, which is why the single-round test showed no difference at all. ## The experiment that found it Two identical five-round tool-calling conversations, differing **only** in whether each echoed assistant turn keeps `id`: | arm | payload-less segment appeared | model still calling tools at round 5 | |---|---|---| | keeps `id` | **0 / 8** | 7 / 7 observed | | strips `id` (this library today) | **7 / 8** | 1 / 8 | Eight paired trials, seven discordant, all in the same direction — a sign test puts that near p ≈ 0.008. One `keeps-id` arm ended before round 5 and I did not capture why, so I am counting it as unknown rather than as a clean pass. ## The mechanism, and the link to the original crash When the handle does not come back, Google answers with: ```json {"type":"reasoning.text","signature":"AY89a1+3GnntT6F…","format":"google-gemini-v1","index":0} ``` `finish_reason: stop`, **zero tool calls**, and a final answer that gives up on the task ("a minor background texture or hatching element"). Two things follow: 1. **`signature` is real after all.** I had inferred it from Gemini's docs and #9's fixture guessed the shape `{"type":"reasoning.text","index":0,"signature":"…"}` — that turns out to be the exact wire shape, but it only appears in the *degraded* state, which is why the happy-path sweep never showed it. Both `reasoning.encrypted.id` and `reasoning.text.signature` are dropped today. 2. **This is the cause of the crash in TeamAI/Orihon#119.** The payload-less `reasoning.text` that failed deserialization was not a quirk of Gemini — it was Gemini's response to *us* having thrown its handle away. #9 stopped the crash; it did not stop the cause. Post-#9 the round parses, and the agent quietly abandons its refinement instead. The second half is the part that matters for Orihon: a silent behavioural regression on a multi-round stage is harder to notice than a failed round, and a boxing agent that stops calling `crop` and declares the region "background hatching" produces a plausible wrong answer rather than an error. ## Fix `[JsonExtensionData]` on `ReasoningDetail`, so every unmodelled field survives the round-trip — `id`, `signature`, and whatever the next provider invents. An explicit `Signature`/`Id` pair would fix today's two fields and leave the next one to be found the same expensive way. Worth checking as part of it that the extension bag composes with `[JsonPolymorphic]` without the `type` discriminator landing in it, and that a segment we constructed ourselves (no extension data) still serializes clean. Reproduction scripts (three file-based probes: single round-trip, shape sweep, multi-round A/B) are in my scratchpad — say the word and I will attach them here or fold them into the repo as a manual diagnostic. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

scarlet referenced this issue from a pull request that will close it,

2026-07-30 13:08:55 +02:00

fix: a provider's own handle on its thought survives the round-trip #11