Assistant degrades to vision-model captions from turn 2 (vision-mode captioning regression) #79

New issue

Closed

opened 2026-08-16 11:33:10 +02:00 by matikane · 0 comments

matikane commented

2026-08-16 11:33:10 +02:00

Member

Root cause (traced on main 88f846a):

#70 (5b510be, "image replay spend") wired the caption describer and maxInlineImages: 3 into vision mode as a token-save optimization: on every run, _describeConversationImages rewrites history images beyond the newest 3 in place to caption TextParts, and _syncHistory persists that rewrite. Turn 1's images are newest → inline; on turn 2, once history holds >3 images, the model's own view of older images is destroyed and replaced by vision-model captions. ADR 0024 still specifies the captioner "only engages when the main model actually lacks image input" — #70 changed that behavior without updating the ADR. #76's 200k-char history budget (which counts images at a flat 4k chars) already bounds image accumulation, making the destructive in-place captioning redundant.

Two hardening items from #70's own Follow-ups:

Vision detection is not sticky: blindMode is recomputed each turn from _modelHasVision (fails closed on any /models listing failure, 5-min cache). A transient listing error ≥5 min into a session flips a vision-capable model into full blind mode mid-session.
Failed captions are cached for the app run: a transient vision-model error bakes (image description failed: ...) into history permanently.

Fix (restores ADR 0024 semantics):

Vision mode: model keeps pixels in history — describer only in blind mode.
Sticky per-session vision detection (first successful lookup wins for the session; failures keep the previous verdict).
Blind mode only: don't cache failed captions (retry next time).
Remove the now-dead DOUJIN_MANAGER_AGENT_MAX_INLINE_IMAGES knob; rely on #76's history budget.

Symptom discriminators (for verification): with >3 viewed images, turn 1 answers from pixels; turn 2 answers about the oldest come from a caption; with ≤3 images nothing is captioned. If ≤3-image sessions also degrade, the detection flap is also live.

Since the 2026-08-15/16 assistant update, the agent behaves wrongly from turn 2 in image-heavy conversations: it stops seeing library images itself and works only from captions produced by the configured vision model (visible as repeated `describe_image` chips). **Root cause** (traced on main `88f846a`): #70 (`5b510be`, "image replay spend") wired the caption describer and `maxInlineImages: 3` into **vision mode** as a token-save optimization: on every run, `_describeConversationImages` rewrites history images beyond the newest 3 in place to caption TextParts, and `_syncHistory` persists that rewrite. Turn 1's images are newest → inline; on turn 2, once history holds >3 images, the model's own view of older images is destroyed and replaced by vision-model captions. ADR 0024 still specifies the captioner "only engages when the main model actually lacks image input" — #70 changed that behavior without updating the ADR. #76's 200k-char history budget (which counts images at a flat 4k chars) already bounds image accumulation, making the destructive in-place captioning redundant. Two hardening items from #70's own Follow-ups: 1. **Vision detection is not sticky**: `blindMode` is recomputed each turn from `_modelHasVision` (fails closed on any `/models` listing failure, 5-min cache). A transient listing error ≥5 min into a session flips a vision-capable model into full blind mode mid-session. 2. **Failed captions are cached for the app run**: a transient vision-model error bakes `(image description failed: ...)` into history permanently. **Fix** (restores ADR 0024 semantics): - Vision mode: model keeps pixels in history — describer only in blind mode. - Sticky per-session vision detection (first successful lookup wins for the session; failures keep the previous verdict). - Blind mode only: don't cache failed captions (retry next time). - Remove the now-dead `DOUJIN_MANAGER_AGENT_MAX_INLINE_IMAGES` knob; rely on #76's history budget. **Symptom discriminators** (for verification): with >3 viewed images, turn 1 answers from pixels; turn 2 answers about the oldest come from a caption; with ≤3 images nothing is captioned. If ≤3-image sessions also degrade, the detection flap is also live.

matikane referenced this issue from a commit

2026-08-16 12:15:15 +02:00

fix: vision-mode main model keeps pixels; captioner is blind-mode only

matikane referenced this issue from a pull request that will close it,

2026-08-16 12:15:31 +02:00

fix: vision-mode main model keeps pixels; captioner is blind-mode only #80