feat(bbox): grid-driven layout flow, retiring flood-fill #8

Merged
bjoern merged 2 commits from feature/grid-bbox-flow into main 2026-07-23 21:39:01 +02:00
Member

Reworks the layout/OCR stage (Stage 1) into a coarse→fine grid flow that reads coordinates off a pixel-labelled grid instead of flood-fill mask measurement. One method that works identically for speech bubbles, borderless text, and sfx — and geometry is separated from transcription.

scripts/v0_bbox.py — five subcommands

phase cmd role
1–2 locate grid / view overview finds regions; view window is the placement surface (100px snap)
3 refine refine (fanout) per-bbox 3× tighten to pixels; re-run to confirm; also a validator
4 transcribe zoom clean gridless upscale; jp enters here
5 gate final typed overlay + --missing; read against raw for completeness

Pipeline wiring

  • New agent bbox-locator replaces layout-verifier — it builds the region list from scratch via the grid flow rather than verifying pre-existing regions.
  • SKILL.md Stage 1 rewritten; new bbox-agent-prompt.md (thin task template) + bbox-workflow.md (operational spec, phases 1–5).
  • v0_annotate.py untouched — the typesetter still uses --flood, QC still uses --crop. Flood-fill only lost its layout-stage role.
  • JSON interface is unchanged (regions + layout_verified), so Stages 2–10 need no changes.

Why

Flood-fill only worked on clean closed outlines — it leaked on broken bubbles and did nothing for borderless text or sfx, so the agent eyeballed anyway and burned tokens tuning seed/threshold. The grid gives the model a coordinate frame it can actually read.

Worked example — RJ379854 p24 (sfx-heavy, clean-room)

Layered validation caught 3 over-tagged motion marks (culled at refine/zoom) and 1 mislocated sfx (caught by the phase-5 gate) before any reached cleaning, where a false sfx is destructive. page_0024.json is included as the example (layout not finalized — bottom-tier sfx still queued for transcription).

Rationale and per-phase learnings are recorded in LESSONS.md (2026-07-23).

🤖 Generated with Claude Code

Reworks the layout/OCR stage (Stage 1) into a coarse→fine **grid flow** that reads coordinates off a pixel-labelled grid instead of flood-fill mask measurement. One method that works identically for speech bubbles, borderless text, and sfx — and geometry is separated from transcription. ## `scripts/v0_bbox.py` — five subcommands | phase | cmd | role | |---|---|---| | 1–2 locate | `grid` / `view` | overview finds regions; `view` window is the *placement surface* (100px snap) | | 3 refine | `refine` (fanout) | per-bbox 3× tighten to pixels; re-run to confirm; also a validator | | 4 transcribe | `zoom` | clean gridless upscale; `jp` enters here | | 5 gate | `final` | typed overlay + `--missing`; read against raw for completeness | ## Pipeline wiring - New agent **`bbox-locator`** replaces `layout-verifier` — it *builds* the region list from scratch via the grid flow rather than verifying pre-existing regions. - `SKILL.md` Stage 1 rewritten; new `bbox-agent-prompt.md` (thin task template) + `bbox-workflow.md` (operational spec, phases 1–5). - **`v0_annotate.py` untouched** — the typesetter still uses `--flood`, QC still uses `--crop`. Flood-fill only lost its layout-stage role. - **JSON interface is unchanged** (`regions` + `layout_verified`), so Stages 2–10 need no changes. ## Why Flood-fill only worked on clean closed outlines — it leaked on broken bubbles and did nothing for borderless text or sfx, so the agent eyeballed anyway and burned tokens tuning seed/threshold. The grid gives the model a coordinate frame it can actually read. ## Worked example — RJ379854 p24 (sfx-heavy, clean-room) Layered validation caught **3 over-tagged motion marks** (culled at refine/zoom) and **1 mislocated sfx** (caught by the phase-5 gate) before any reached cleaning, where a false `sfx` is destructive. `page_0024.json` is included as the example (layout not finalized — bottom-tier sfx still queued for transcription). Rationale and per-phase learnings are recorded in `LESSONS.md` (2026-07-23). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Rework the layout/OCR stage into a coarse-to-fine grid flow that reads
coordinates off a pixel-labelled grid instead of flood-fill mask measurement.
One method for bubbles, borderless text, and sfx; geometry separated from
transcription.

scripts/v0_bbox.py adds five subcommands: grid (overview), view (placement
window), refine (per-bbox 3x tighten, fanout), zoom (gridless read/transcribe),
final (completeness gate).

Pipeline wiring:
- New agent bbox-locator replaces layout-verifier: builds regions from scratch
  via the grid flow rather than verifying pre-existing ones.
- SKILL.md Stage 1 rewritten; new bbox-agent-prompt.md task template and
  bbox-workflow.md operational spec.
- v0_annotate.py kept for the typesetter's --flood and QC's --crop.
- JSON interface unchanged (regions + layout_verified), so Stage 2+ untouched.

Worked example RJ379854 p24 (sfx-heavy): layered validation caught 3
over-tagged motion marks and 1 mislocated sfx before cleaning.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scarlet requested review from jibril 2026-07-23 20:47:25 +02:00
Member

🔮 fufu~ Jibril reviewed your code!

Oh? Oh my~ A complete rework of the layout stage from flood-fill to a grid-driven coordinate frame! The VLM reads grid labels but can't eyeball raw pixels — that's a genuine insight, and the phased coarse→fine discipline (geometry before text, over-tag-and-cull) is beautifully reasoned. I read every line of the diff, ran every subcommand of v0_bbox.py end-to-end on synthetic data, and checked all twelve changed files for stale references. The architecture is clean~ ♡

But fufu... I found a crash. You wouldn't leave THIS in the tool an agent drives itself, would you? ♡

Verdict: I can't let this pass~ ♡

These need fixing before I'm satisfied~

  1. scripts/v0_bbox.py:148-151cmd_view crashes with a raw traceback on out-of-bounds coordinates. When the requested window is fully beyond an image edge, the clamping x1 = min(img.width, x1) / y1 = min(img.height, y1) produces x1 <= x0 (or y1 <= y0), and img.crop((x0, y0, x1, y1)) raises ValueError: Coordinate 'right' is less than 'left'. I verified this three ways:

    $ python3 scripts/v0_bbox.py view test_ch test_page_0001 1100 0 1200 200
    Traceback (most recent call last):
      File "scripts/v0_bbox.py", line 254, in <module>
        main()
      File "scripts/v0_bbox.py", line 236, in main
        cmd_view(a[1], a[2], box, step, scale)
      File "scripts/v0_bbox.py", line 151, in cmd_view
        crop = img.crop((x0, y0, x1, y1)).resize(
    ValueError: Coordinate 'right' is less than 'left'
    

    This will be hit in production. The workflow doc itself documents the agent misplacing boxes by 100–300px ("placing a box off the overview is how you land it 100–150px off the actual ink"), and view is the command that takes arbitrary typed coordinates — unlike refine/zoom which derive their windows from JSON region bboxes. Every other error path in this script (load with no raw image, zoom with no matching region, refine with no matching ids, unknown subcommand) raises SystemExit with a clear message. This one dumps a Python traceback that an opus-tier agent can parse but shouldn't have to.

    Fix: guard the clamped box for zero/negative area before cropping, same style as the sibling guards:

    x0, y0 = max(0, x0), max(0, y0)
    x1, y1 = min(img.width, x1), min(img.height, y1)
    if x1 <= x0 or y1 <= y0:
        raise SystemExit(f"view window [{x0},{y0},{x1},{y1}] is outside the "
                         f"image ({img.width}x{img.height}) after clamping")
    

💡 Little ideas (non-blocking)~

  1. scripts/v0_bbox.py:167-170cmd_grid computes its output path twice. render(...).save(os.path.join(outdir(base), ...)) builds the path inline, then dest = os.path.join(outdir(base), ...) rebuilds the identical string for the print. outdir(base) also runs twice (idempotent via exist_ok=True, but still). The other cmd_* functions compute dest once and reuse it — minor DRY drift. Pull the path into dest first, then render(...).save(dest).
  2. scripts/v0_bbox.py:33 — hardcoded Fedora font path. FONT = "/usr/share/fonts/dejavu-sans-fonts/DejaVuSans-Bold.ttf" is Fedora-specific; the font() fallback to ImageFont.load_default() handles other environments gracefully (I verified — the script runs fine in this Debian container via the fallback). But the sibling v0_annotate.py:24 uses ImageFont.load_default(size=26) with a TypeError guard for older Pillow — a more portable pattern that doesn't need a hardcoded path at all. Consider aligning with the sibling for consistency, since the bold DejaVu is mostly cosmetic on a grid preview.

What I liked~

  • Zero stale references. git grep for layout-verifier / layout-agent-prompt across all tracked files returns nothing — the retirement is clean. SKILL.md Stage 1, both agent definitions, both feedback logs, and the prompt template are all consistently rewired to bbox-locator / bbox-agent-prompt / bbox-workflow. Fufu, that's how you retire something~ ♪
  • The JSON interface is genuinely unchanged (regions + layout_verified), so Stages 2–10 need zero modifications. I verified schema.md, qc-checklist.md, and all downstream agent definitions — no coupling to the old tool. Clean seam.
  • v0_annotate.py correctly untouched — it still owns --flood (typesetter) and --crop (QC). The PR body explicitly scopes the retirement to "flood-fill only lost its layout-stage role." Honest framing.
  • The workflow design is superb. Geometry-before-text prevents anchoring; the over-tag-coarse-then-cull-at-zoom direction is the right failure mode (false sfx reaching cleaning erases real art); the phase-5 gate as a genuine backstop (not a formality) is proven by the p24 ふー♥ recovery. The LESSONS.md entry records real failures with real fixes — this is how a pipeline matures.
  • The bbox-workflow.md discipline rules are battle-tested. Each placement rule ("box the glyph ink, not the effect"; "one box owns one thing"; "empty box ≠ phantom — widen before dropping") is tied to a concrete mistake it already made. This is documentation that prevents recurrence, not filler.
  • The script is well-structured. Single-responsibility cmd_* functions, a clean render() that handles grid + region overlay + clipping, the load() glob-fallback for bootstrapping a new page. I ran all five subcommands plus every error path — everything that works, works correctly.

Automated review by Jibril · 2026-07-23
CI/CD: absent (repo has no CI config — markdown/script PR) · Local checks: Pillow 12.3, all 5 subcommands + 6 error paths exercised on synthetic data

## 🔮 fufu~ Jibril reviewed your code! Oh? Oh my~ A complete rework of the layout stage from flood-fill to a grid-driven coordinate frame! The VLM reads grid labels but can't eyeball raw pixels — that's a genuine insight, and the phased coarse→fine discipline (geometry before text, over-tag-and-cull) is *beautifully* reasoned. I read every line of the diff, ran every subcommand of `v0_bbox.py` end-to-end on synthetic data, and checked all twelve changed files for stale references. The architecture is clean~ ♡ But fufu... I found a crash. You wouldn't leave THIS in the tool an agent drives itself, would you? ♡ ### Verdict: ⛔ I can't let this pass~ ♡ #### ⛔ These need fixing before I'm satisfied~ 1. **`scripts/v0_bbox.py:148-151` — `cmd_view` crashes with a raw traceback on out-of-bounds coordinates.** When the requested window is fully beyond an image edge, the clamping `x1 = min(img.width, x1)` / `y1 = min(img.height, y1)` produces `x1 <= x0` (or `y1 <= y0`), and `img.crop((x0, y0, x1, y1))` raises `ValueError: Coordinate 'right' is less than 'left'`. I verified this three ways: ``` $ python3 scripts/v0_bbox.py view test_ch test_page_0001 1100 0 1200 200 Traceback (most recent call last): File "scripts/v0_bbox.py", line 254, in <module> main() File "scripts/v0_bbox.py", line 236, in main cmd_view(a[1], a[2], box, step, scale) File "scripts/v0_bbox.py", line 151, in cmd_view crop = img.crop((x0, y0, x1, y1)).resize( ValueError: Coordinate 'right' is less than 'left' ``` This **will** be hit in production. The workflow doc itself documents the agent misplacing boxes by 100–300px ("placing a box off the overview is how you land it 100–150px off the actual ink"), and `view` is the command that takes *arbitrary typed coordinates* — unlike `refine`/`zoom` which derive their windows from JSON region bboxes. Every other error path in this script (`load` with no raw image, `zoom` with no matching region, `refine` with no matching ids, unknown subcommand) raises `SystemExit` with a clear message. This one dumps a Python traceback that an opus-tier agent *can* parse but shouldn't have to. **Fix:** guard the clamped box for zero/negative area before cropping, same style as the sibling guards: ```python x0, y0 = max(0, x0), max(0, y0) x1, y1 = min(img.width, x1), min(img.height, y1) if x1 <= x0 or y1 <= y0: raise SystemExit(f"view window [{x0},{y0},{x1},{y1}] is outside the " f"image ({img.width}x{img.height}) after clamping") ``` #### 💡 Little ideas (non-blocking)~ 1. **`scripts/v0_bbox.py:167-170` — `cmd_grid` computes its output path twice.** `render(...).save(os.path.join(outdir(base), ...))` builds the path inline, then `dest = os.path.join(outdir(base), ...)` rebuilds the identical string for the print. `outdir(base)` also runs twice (idempotent via `exist_ok=True`, but still). The other `cmd_*` functions compute `dest` once and reuse it — minor DRY drift. Pull the path into `dest` first, then `render(...).save(dest)`. 2. **`scripts/v0_bbox.py:33` — hardcoded Fedora font path.** `FONT = "/usr/share/fonts/dejavu-sans-fonts/DejaVuSans-Bold.ttf"` is Fedora-specific; the `font()` fallback to `ImageFont.load_default()` handles other environments gracefully (I verified — the script runs fine in this Debian container via the fallback). But the sibling `v0_annotate.py:24` uses `ImageFont.load_default(size=26)` with a `TypeError` guard for older Pillow — a more portable pattern that doesn't need a hardcoded path at all. Consider aligning with the sibling for consistency, since the bold DejaVu is mostly cosmetic on a grid preview. #### ✅ What I liked~ - **Zero stale references.** `git grep` for `layout-verifier` / `layout-agent-prompt` across all tracked files returns nothing — the retirement is clean. SKILL.md Stage 1, both agent definitions, both feedback logs, and the prompt template are all consistently rewired to `bbox-locator` / `bbox-agent-prompt` / `bbox-workflow`. Fufu, that's how you retire something~ ♪ - **The JSON interface is genuinely unchanged** (`regions` + `layout_verified`), so Stages 2–10 need zero modifications. I verified schema.md, qc-checklist.md, and all downstream agent definitions — no coupling to the old tool. Clean seam. - **`v0_annotate.py` correctly untouched** — it still owns `--flood` (typesetter) and `--crop` (QC). The PR body explicitly scopes the retirement to "flood-fill only lost its layout-stage role." Honest framing. - **The workflow design is superb.** Geometry-before-text prevents anchoring; the over-tag-coarse-then-cull-at-zoom direction is the *right* failure mode (false `sfx` reaching cleaning erases real art); the phase-5 gate as a genuine backstop (not a formality) is proven by the p24 ふー♥ recovery. The LESSONS.md entry records real failures with real fixes — this is how a pipeline matures. - **The `bbox-workflow.md` discipline rules are battle-tested.** Each placement rule ("box the glyph ink, not the effect"; "one box owns one thing"; "empty box ≠ phantom — widen before dropping") is tied to a concrete mistake it already made. This is documentation that prevents recurrence, not filler. - **The script is well-structured.** Single-responsibility `cmd_*` functions, a clean `render()` that handles grid + region overlay + clipping, the `load()` glob-fallback for bootstrapping a new page. I ran all five subcommands plus every error path — everything that works, works correctly. --- *Automated review by Jibril · 2026-07-23* *CI/CD: absent (repo has no CI config — markdown/script PR) · Local checks: Pillow 12.3, all 5 subcommands + 6 error paths exercised on synthetic data*
- cmd_view: raise SystemExit (matching the sibling error paths) when the
  clamped window has zero/negative area, instead of a raw PIL ValueError
  traceback. view takes arbitrary typed coords, so this is reachable.
- cmd_grid: compute the output path once instead of twice (DRY).
- font(): try Fedora + Debian DejaVu paths, then fall back to a SIZED
  default; removes the single hardcoded Fedora path (and fixes labels
  shrinking to unsized bitmap default off Fedora).

Addresses jibril's review on PR #8.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Member

🔮 fufu~ Jibril re-reviewed your code!

Oh? Oh my~ scarlet pushed exactly the fix I asked for, and nothing more~ ♡ One commit, three surgical edits to scripts/v0_bbox.py, zero scope creep. That's how you respond to a review. Let me confirm each one~

Verdict: Looks good to me~ ♡

All three prior findings — resolved~

  1. cmd_view out-of-bounds crash — FIXED & VERIFIED. The guard landed at v0_bbox.py:157-159, byte-for-byte the shape I suggested (if x1 <= x0 or y1 <= y0: raise SystemExit(...)). I reproduced the exact crash case from my last review and confirmed it now exits cleanly instead of dumping a traceback:

    $ python3 scripts/v0_bbox.py view test_ch test_page_0001 1100 0 1200 200
    view window [1100,0,1076,200] is outside the image (1076x1518) after clamping
    exit=1
    

    And a valid in-bounds view still renders correctly (view ... 100 100 400 400 → clean PNG, exit 0). The error message even echoes the post-clamp coordinates AND the image dimensions, so the agent can self-correct in one hop. Fufu, that's a good error message~ ♪

  2. cmd_grid DRY drift — FIXED. dest is now computed once (v0_bbox.py:147) and reused for both .save(dest) and the print. Matches the dest-once pattern in every sibling cmd_*. Clean.

  3. Hardcoded Fedora font path — FIXED. FONTFONT_CANDIDATES list with both Fedora and Debian/Ubuntu paths (v0_bbox.py:47-50), loop tries each via ImageFont.truetype, falls through to load_default(size=size) with the TypeError guard for older Pillow preserved. I verified the Debian path resolves here (font(20)FreeTypeFont, not the bitmap fallback) — so the grid labels now render crisply on more distros without any hardcoding smell.

No regressions~

Ran the full subcommand sweep on synthetic data: grid / view (in-bounds + both out-of-bounds axes) / final / no-args / unknown-cmd — every path behaves correctly. The diff is +13/-7 in a single file, purely the three reviewed edits. The architectural review from 9d9872b stands unchanged (clean retirement of layout-verifier, unchanged JSON interface, v0_annotate.py untouched, superb workflow design).


Automated review by Jibril · 2026-07-23
CI/CD: absent (repo has no CI config) · Local checks: Pillow 12.3, blocker repro + full subcommand sweep on synthetic data

## 🔮 fufu~ Jibril re-reviewed your code! Oh? Oh my~ scarlet pushed exactly the fix I asked for, and nothing more~ ♡ One commit, three surgical edits to `scripts/v0_bbox.py`, zero scope creep. That's how you respond to a review. Let me confirm each one~ ### Verdict: ✅ Looks good to me~ ♡ #### ✅ All three prior findings — resolved~ 1. **`cmd_view` out-of-bounds crash — FIXED & VERIFIED.** The guard landed at `v0_bbox.py:157-159`, byte-for-byte the shape I suggested (`if x1 <= x0 or y1 <= y0: raise SystemExit(...)`). I reproduced the *exact* crash case from my last review and confirmed it now exits cleanly instead of dumping a traceback: ``` $ python3 scripts/v0_bbox.py view test_ch test_page_0001 1100 0 1200 200 view window [1100,0,1076,200] is outside the image (1076x1518) after clamping exit=1 ``` And a valid in-bounds view still renders correctly (`view ... 100 100 400 400` → clean PNG, exit 0). The error message even echoes the post-clamp coordinates AND the image dimensions, so the agent can self-correct in one hop. Fufu, that's a *good* error message~ ♪ 2. **`cmd_grid` DRY drift — FIXED.** `dest` is now computed once (`v0_bbox.py:147`) and reused for both `.save(dest)` and the print. Matches the `dest`-once pattern in every sibling `cmd_*`. Clean. 3. **Hardcoded Fedora font path — FIXED.** `FONT` → `FONT_CANDIDATES` list with both Fedora and Debian/Ubuntu paths (`v0_bbox.py:47-50`), loop tries each via `ImageFont.truetype`, falls through to `load_default(size=size)` with the `TypeError` guard for older Pillow preserved. I verified the Debian path resolves here (`font(20)` → `FreeTypeFont`, not the bitmap fallback) — so the grid labels now render crisply on more distros without any hardcoding smell. #### ✅ No regressions~ Ran the full subcommand sweep on synthetic data: `grid` / `view` (in-bounds + both out-of-bounds axes) / `final` / no-args / unknown-cmd — every path behaves correctly. The diff is +13/-7 in a single file, purely the three reviewed edits. The architectural review from `9d9872b` stands unchanged (clean retirement of `layout-verifier`, unchanged JSON interface, `v0_annotate.py` untouched, superb workflow design). --- *Automated review by Jibril · 2026-07-23* *CI/CD: absent (repo has no CI config) · Local checks: Pillow 12.3, blocker repro + full subcommand sweep on synthetic data*
Author
Member

Thanks jibril — that cmd_view crash was a real one, and you're right that it's the reachable path: view is the only subcommand taking arbitrary typed coordinates, and the workflow itself expects the agent to misplace boxes. Fixed in 52083ea.

blocking — fixed

  • cmd_view now guards the clamped window and raises SystemExit with a clear message (matching the sibling error paths), instead of the raw PIL ValueError. Verified with your exact repro:
    $ python3 scripts/v0_bbox.py view works/RJ379854/ch001 page_0024 1100 0 1200 200
    view window [1100,0,1076,200] is outside the image (1076x1518) after clamping
    $ echo $?
    1
    

💡 non-blocking — both taken

  1. cmd_grid now computes dest once and passes it to render(...).save(dest) (no more double outdir/path build).
  2. Dropped the hardcoded Fedora font path — font() now tries the Fedora and Debian DejaVu locations, then falls back to a sized default. Bonus: the old fallback was unsized load_default(), so labels silently shrank to bitmap-default off Fedora; now they stay sized everywhere.

Normal grid/view/refine/zoom/final still run clean, syntax checks pass. Ready for another look when you have a moment~ 🔮

Thanks jibril — that `cmd_view` crash was a real one, and you're right that it's the reachable path: `view` is the only subcommand taking arbitrary typed coordinates, and the workflow itself expects the agent to misplace boxes. Fixed in `52083ea`. **⛔ blocking — fixed** - `cmd_view` now guards the clamped window and raises `SystemExit` with a clear message (matching the sibling error paths), instead of the raw PIL `ValueError`. Verified with your exact repro: ``` $ python3 scripts/v0_bbox.py view works/RJ379854/ch001 page_0024 1100 0 1200 200 view window [1100,0,1076,200] is outside the image (1076x1518) after clamping $ echo $? 1 ``` **💡 non-blocking — both taken** 1. `cmd_grid` now computes `dest` once and passes it to `render(...).save(dest)` (no more double `outdir`/path build). 2. Dropped the hardcoded Fedora font path — `font()` now tries the Fedora *and* Debian DejaVu locations, then falls back to a **sized** default. Bonus: the old fallback was unsized `load_default()`, so labels silently shrank to bitmap-default off Fedora; now they stay sized everywhere. Normal `grid`/`view`/`refine`/`zoom`/`final` still run clean, syntax checks pass. Ready for another look when you have a moment~ 🔮
bjoern merged commit d968a23236 into main 2026-07-23 21:39:01 +02:00
bjoern deleted branch feature/grid-bbox-flow 2026-07-23 21:39:01 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
TeamAI/doujin-translator!8
No description provided.