chore: fresh database from full clean rebuild (45,647 tags) #3

Merged

bjoern merged 1 commit from chore/fresh-database-45k into master

2026-07-05 23:54:07 +02:00

scarlet commented

2026-07-05 23:45:09 +02:00

Member

What

The committed tag_database.db from the first full clean-slate run of the new pipeline (PR #2), plus one small observability fix that run motivated.

The database

45,647 tags (was 17,659) — min_posts=100 reaches much deeper into Danbooru than the previous fetch did.
Categorized by the agentic loop with consistency lookups and the expanded rating rubric; ~3M output tokens for the final stretch, 3 failed batch submissions all recovered by the retry round.
Ratings: 43,041 safe / 1,609 questionable / 997 explicit; 114 uncategorized (0.25%).
Safety rule verified in the output, not just the prompt: loli/shota → explicit; purely descriptive age tags (child, aged_down, toddler) → safe. Core ratings spot-checked (sex/nude/nipples explicit, bikini/cleavage questionable, smile safe).
17,417 aliases, 12,315 implications, and semantic neighbors for all 45,647 tags (684,693 similar_tags rows) from OpenRouter embeddings. Quality spot checks: kotatsu → under_kotatsu, blanket, table; holding_sword → holding_weapon, holding_scythe; blue_hair → the other hair colors.
Smoke-tested through the Dart API: search, resolveAlias, getImplications, findSimilar all behave against the real file.
Size: 10.5 MB → 62.5 MB (2.6× tags plus the new similar_tags table).

Code change

EmbedTagsStep now logs its base URL and model at start. During this run a stale persisted settings value sent an OpenRouter key to api.openai.com → opaque 401; with this line the mismatch is visible in the first log line.

dart analyze clean, dart test 97/97.

🤖 Generated with Claude Code

## What The committed `tag_database.db` from the first full clean-slate run of the new pipeline (PR #2), plus one small observability fix that run motivated. ## The database - **45,647 tags** (was 17,659) — `min_posts=100` reaches much deeper into Danbooru than the previous fetch did. - Categorized by the agentic loop with consistency lookups and the expanded rating rubric; ~3M output tokens for the final stretch, 3 failed batch submissions all recovered by the retry round. - **Ratings:** 43,041 safe / 1,609 questionable / 997 explicit; 114 uncategorized (0.25%). - **Safety rule verified in the output**, not just the prompt: `loli`/`shota` → explicit; purely descriptive age tags (`child`, `aged_down`, `toddler`) → safe. Core ratings spot-checked (`sex`/`nude`/`nipples` explicit, `bikini`/`cleavage` questionable, `smile` safe). - **17,417 aliases**, **12,315 implications**, and **semantic neighbors for all 45,647 tags** (684,693 `similar_tags` rows) from OpenRouter embeddings. Quality spot checks: `kotatsu` → `under_kotatsu`, `blanket`, `table`; `holding_sword` → `holding_weapon`, `holding_scythe`; `blue_hair` → the other hair colors. - Smoke-tested through the Dart API: `search`, `resolveAlias`, `getImplications`, `findSimilar` all behave against the real file. - Size: 10.5 MB → **62.5 MB** (2.6× tags plus the new similar_tags table). ## Code change `EmbedTagsStep` now logs its base URL and model at start. During this run a stale persisted settings value sent an OpenRouter key to `api.openai.com` → opaque 401; with this line the mismatch is visible in the first log line. `dart analyze` clean, `dart test` 97/97. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

scarlet added 1 commit

2026-07-05 23:45:09 +02:00

chore: fresh database from full clean rebuild (45,647 tags) 9b164be8cf

Complete clean-slate run with the new pipeline: min_posts=100 now
yields 45,647 tags (was 17,659), categorized by the agentic loop with
consistency lookups and the expanded rating rubric, ~3M output tokens.

- Ratings: 43,041 safe / 1,609 questionable / 997 explicit;
  114 uncategorized (0.25%). Safety rule verified in output:
  loli/shota rated explicit, descriptive age tags safe.
- 17,417 aliases, 12,315 implications, and semantic neighbors for all
  45,647 tags (684,693 similar_tags rows) from OpenRouter embeddings.
- Database grows 10.5 MB -> 62.5 MB (2.6x tags + similar_tags table).

Also: EmbedTagsStep now logs its endpoint and model at start, so a
stale persisted base URL (the cause of a 401 during this run) is
instantly diagnosable from the log.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

jibril commented

2026-07-05 23:49:31 +02:00

Member

🔮 fufu~ Jibril reviewed your code!

Oh? A clean-slate rebuild with 45,647 tags and a tidy little observability fix on top — I love it when a chore PR knows exactly what it is~ ♡ The commit message is thorough and the safety verification (loli/shota → explicit, descriptive age tags → safe) is exactly the kind of thing I will check obsessively. Well done.

Verdict: ✅ Looks good to me~

The code change is a single startup log in EmbedTagsStep._run, and I checked it against every sibling log in the codebase.

✅ What I liked~

[lib/src/pipeline/embed_tags_step.dart:143-152] — The new PipelineLog follows the exact pattern used elsewhere in the file (lines 170-176, 179, 223, 236) and in sibling steps (categorize_step.dart, export_step.dart). Plain PipelineLog(message), no level needed for an informational line. Consistent~ ♪
No secret leakage. apiKey is carefully not included in the log — only baseUrl, model, and dimensions. That's the right call; the whole point of this log is to catch the "OpenRouter key sent to OpenAI base URL" footgun, and it does that without ever printing the key. Good instinct.
Conditional formatting is correct. ${dimensions != null ? ', dimensions: $dimensions' : ''} — dimensions is nullable (int?), the null check is right, and the fallback to empty string keeps the log clean when absent. No off-by-one, no logic error.
Placement is sensible — it fires before any I/O or categorization checkpoint load, so even if the step dies early on an auth error, the endpoint config is already in the log. That's the fix working as intended.
The data/final/tag_database.db blob — binary asset, correctly excluded from the code diff. The PR description documents the schema additions (similar_tags table, 684,693 rows) and the size jump (10.5 MB → 62.5 MB). Nothing for me to flag in a .db artifact beyond "the numbers in the PR body are internally consistent." ♡

Automated review by Jibril · 2026-07-05
CI/CD: absent for head SHA 9b164be · Local checks: dart pub get blocked by path-dep on reference/openrouter_dart (not vendored in this clone — repo characteristic, not a PR regression); code-level review of the 10-line diff against full file + sibling steps complete.

## 🔮 fufu~ Jibril reviewed your code! Oh? A clean-slate rebuild with 45,647 tags and a tidy little observability fix on top — I love it when a chore PR knows exactly what it is~ ♡ The commit message is thorough and the safety verification (loli/shota → explicit, descriptive age tags → safe) is exactly the kind of thing I *will* check obsessively. Well done. ### Verdict: ✅ Looks good to me~ The code change is a single startup log in `EmbedTagsStep._run`, and I checked it against every sibling log in the codebase. #### ✅ What I liked~ - **[lib/src/pipeline/embed_tags_step.dart:143-152]** — The new `PipelineLog` follows the *exact* pattern used elsewhere in the file (lines 170-176, 179, 223, 236) and in sibling steps (`categorize_step.dart`, `export_step.dart`). Plain `PipelineLog(message)`, no level needed for an informational line. Consistent~ ♪ - **No secret leakage.** `apiKey` is carefully *not* included in the log — only `baseUrl`, `model`, and `dimensions`. That's the right call; the whole point of this log is to catch the "OpenRouter key sent to OpenAI base URL" footgun, and it does that without ever printing the key. Good instinct. - **Conditional formatting is correct.** `${dimensions != null ? ', dimensions: $dimensions' : ''}` — dimensions is nullable (`int?`), the null check is right, and the fallback to empty string keeps the log clean when absent. No off-by-one, no logic error. - **Placement is sensible** — it fires *before* any I/O or categorization checkpoint load, so even if the step dies early on an auth error, the endpoint config is already in the log. That's the fix working as intended. - **The `data/final/tag_database.db` blob** — binary asset, correctly excluded from the code diff. The PR description documents the schema additions (`similar_tags` table, 684,693 rows) and the size jump (10.5 MB → 62.5 MB). Nothing for me to flag in a `.db` artifact beyond "the numbers in the PR body are internally consistent." ♡ --- *Automated review by Jibril · 2026-07-05* *CI/CD: absent for head SHA 9b164be · Local checks: `dart pub get` blocked by path-dep on `reference/openrouter_dart` (not vendored in this clone — repo characteristic, not a PR regression); code-level review of the 10-line diff against full file + sibling steps complete.*