booru_tag_db_dart

Danbooru tag database with AI categorization and tool-based browsing. Fetches tags from Danbooru, enriches them with AI-generated categories/descriptions via openrouter_dart, and exposes the result as searchable Tool<TParams> implementations for AI agents.

Setup

git clone --recurse-submodules <repo-url>
cd booru_tag_db_dart
dart pub get

If you already cloned without --recurse-submodules:

git submodule update --init --recursive
dart pub get

Desktop App (Wizard)

The easiest way to build or refresh the database is the Flutter desktop app in app/:

cd app
flutter run -d linux --release

It walks through everything: OpenRouter API key and model (persisted locally), pipeline knobs (min posts, batch size, concurrency), and a mode choice - Refresh (update counts, categorize only new tags), Resume (continue an interrupted run), or Clean rebuild (wipe and redo everything, including the paid AI categorization - it asks for confirmation). While running it shows per-step progress, token usage with an ETA, and a live log; cancelling is graceful, so a later Resume continues where you stopped.

Data Pipeline (CLI)

The same pipeline is also available as CLI scripts. It has 6 steps (two optional); each is resumable - if interrupted (Ctrl+C cancels gracefully), re-run the same command to continue where it left off.

The final database (data/final/tag_database.db) is committed, so you only need to run the pipeline to refresh or rebuild it.

For a completely clean slate first:

# Wipes data/raw, data/intermediate, and the exported database.
# Deletes AI categorization progress - a full re-run costs API tokens!
dart run bin/clean_data.dart

Step 1: Fetch tags

Downloads tags from the Danbooru API, filtered by minimum post count.

# Default: tags with >= 100 posts (~30-50k tags)
dart run bin/fetch_tags.dart

# Smaller test batch
dart run bin/fetch_tags.dart --min-posts=10000

# Refresh: clear raw pages and re-fetch from scratch
dart run bin/fetch_tags.dart --refresh

Flag	Default	Description
--min-posts=	100	Minimum post count threshold
--refresh	off	Clear existing raw pages and re-fetch

Output: data/raw/tags_page_NNNN.json

Step 2: Fetch wiki descriptions

Fetches raw wiki body text for each tag and merges everything into a single file.

dart run bin/fetch_wikis.dart

# Refresh: update metadata for existing tags, fetch wikis for new tags only
dart run bin/fetch_wikis.dart --refresh

Flag	Default	Description
--refresh	off	Merge fresh metadata, fetch wikis for new tags only

Output: data/intermediate/merged_tags.json

Step 3: AI categorization

Uses an AI model (via OpenRouter) to assign nuanced categories, generate display names, clean up wiki markup into markdown descriptions, and flag NSFW content.

# Set your API key (or pass --api-key=...)
export OPENROUTER_API_KEY=sk-or-...

# Run with defaults (4 concurrent batches of 10)
dart run bin/categorize_tags.dart

# Customize model, batch size, and parallelism
dart run bin/categorize_tags.dart --model=google/gemini-2.5-flash --batch-size=10 --concurrency=8

# Smoke test on a handful of tags first
dart run bin/categorize_tags.dart --limit=50

# Start fresh (discard previous progress)
dart run bin/categorize_tags.dart --no-resume

Options:

Flag	Default	Description
--api-key=	$OPENROUTER_API_KEY	OpenRouter API key
--model=	openrouter/aurora-alpha	Model to use
--batch-size=	10	Tags per AI request
--concurrency=	4	Batches in flight at once
--limit=	0 (all)	Categorize at most N tags this run
--no-resume	off	Start fresh, ignoring checkpoint

The loop retries failed batches with exponential backoff (real waits on 429), validates model output against the batch (invented tag names are dropped, omitted ones get a retry round), and reports token totals plus an ETA while running.

Output: data/intermediate/categorization_progress.jsonl (append-only checkpoint; a legacy .json checkpoint is migrated automatically)

Step 4: Fetch implications (optional)

Fetches Danbooru's active tag implications (e.g. blue_hair implies hair). Used by SuggestTagsTool to skip redundant suggestions and find sibling tags, and surfaced by GetTagDetailsTool. Skipping this step just leaves the implication tables empty.

dart run bin/fetch_implications.dart

Output: data/intermediate/implications.json

Step 5: Embed tags (optional)

Embeds categorized tags via an OpenAI-compatible embeddings API. The export step then precomputes each tag's top semantic neighbors ("similar tags") - the vectors themselves never enter the final database, so it stays small and needs no runtime API access.

# OpenRouter (default - same key as categorization): ~18k tags cost cents
export OPENROUTER_API_KEY=sk-or-...
dart run bin/embed_tags.dart

# Local Ollama, no API key needed
dart run bin/embed_tags.dart --base-url=http://localhost:11434/v1 --model=nomic-embed-text --dimensions=0

Flag	Default	Description
--api-key=	$EMBEDDINGS_API_KEY / $OPENROUTER_API_KEY / $OPENAI_API_KEY	API key (optional for local endpoints)
--base-url=	https://openrouter.ai/api/v1	OpenAI-compatible endpoint
--model=	openai/text-embedding-3-small	Embedding model
--dimensions=	512	Output dimensions; 0 omits the parameter
--batch-size=	100	Texts per API request

Resumable: only new tags (or tags whose description changed) are embedded on re-runs. Tags without a description are skipped - name-only neighbors would be noise.

Output: data/intermediate/embeddings.jsonl

Step 6: Export

Validates, deduplicates, and exports the final database as SQLite (tags, FTS5 index, alias lookup, implications, and semantic neighbors when embeddings exist).

dart run bin/export_database.dart

Output: data/final/tag_database.db

Refreshing the database

To update post counts and discover new tags without losing AI categorization:

dart run bin/fetch_tags.dart --refresh
dart run bin/fetch_wikis.dart --refresh
dart run bin/categorize_tags.dart        # categorizes new tags, syncs post counts
dart run bin/fetch_implications.dart     # re-fetches all implications (fast)
dart run bin/embed_tags.dart             # embeds only new/changed tags
dart run bin/export_database.dart

fetch_tags --refresh clears raw pages and re-fetches everything (~35s). fetch_wikis --refresh merges fresh metadata into existing tags (preserving wiki bodies) and fetches wikis only for newly discovered tags. categorize_tags automatically syncs updated post counts into already-categorized entries on every resume.

Pipeline stats

Check progress at any point:

dart run bin/stats.dart

Library Usage

Loading the database

The database is a SQLite file; loading it is synchronous. Call dispose() when done to close the connection.

import 'package:booru_tag_db_dart/booru_tag_db_dart.dart';

final db = TagDatabase.loadFromFile('data/final/tag_database.db');
// ...
db.dispose();

Searching tags

final results = db.search('blue hair', limit: 10);
for (final tag in results) {
  print('${tag.displayName} (${tag.customCategory.apiString}) - ${tag.postCount} posts');
}

Browsing by category

final hairColors = db.getByCategory(CustomCategory.hairColor);
final counts = db.getCategoryCounts(); // {CustomCategory.hairColor: 42, ...}

Getting tag details

getTag accepts the canonical name or any alias:

final tag = db.getTag('ganyu_(genshin_impact)');
print(tag?.displayName);   // Ganyu (Genshin Impact)
print(tag?.description);   // Clean markdown description
print(tag?.relatedTags);   // [genshin_impact, horns, ...]

// Aliases resolve to the canonical tag:
db.getTag('76_(nanaroku)')?.name;        // nanaroku_(fortress76)
db.resolveAlias('76_(nanaroku)');        // nanaroku_(fortress76)

Finding related tags

final related = db.findRelated('blue_hair', limit: 5, maxRating: Rating.safe);

Implications

db.getImplications('blue_hair');       // [hair] - tags blue_hair implies
db.getImpliedBy('hair', limit: 10);    // [long_hair, blue_hair, ...] by post count

Similar tags (semantic)

Precomputed embedding neighbors - by meaning, not name overlap. Empty for databases exported without the embeddings step.

db.findSimilar('kotatsu', limit: 5);   // e.g. [winter_clothes, fireplace, ...]

Using as AI Agent Tools

All four tools implement Tool<TParams> from openrouter_dart and can be plugged directly into an Agent.

import 'package:booru_tag_db_dart/booru_tag_db_dart.dart';
import 'package:openrouter_dart/openrouter_dart.dart';

final db = TagDatabase.loadFromFile('data/final/tag_database.db');

final client = OpenRouterClient(
  options: OpenRouterClientOptions(apiKey: 'sk-or-...'),
);

final agent = Agent(
  client: client,
  options: AgentOptions(
    model: 'anthropic/claude-sonnet-4',
    systemPrompt: 'You help users find and compose booru tags for images.',
    tools: [
      SearchTagsTool(db),        // search_booru_tags
      BrowseCategoryTool(db),    // browse_booru_category
      GetTagDetailsTool(db),     // get_booru_tag_details
      SuggestTagsTool(db),       // suggest_booru_tags
    ],
  ),
);

final result = await agent.chat('Find tags for a girl with blue hair in a school uniform');

Tool reference

Tool	Name	Description
SearchTagsTool	search_booru_tags	Keyword search across name, aliases, and description. Supports category and rating filters.
BrowseCategoryTool	browse_booru_category	List all categories or paginate tags within a category.
GetTagDetailsTool	get_booru_tag_details	Full details for a tag by name or alias, including implied tags.
SuggestTagsTool	suggest_booru_tags	Suggest complementary tags given a current selection. Uses related-tag, implication, and semantic-neighbor data; skips tags the selection already implies.
FindSimilarTagsTool	find_similar_booru_tags	Semantically similar tags via precomputed embedding neighbors.

Custom Categories

Tags are classified into ~55 nuanced categories beyond Danbooru's basic 5:

Group	Categories
Appearance	hair_color, hair_style, hair_length, hair_accessory, eye_color, eye_feature, facial_feature, expression, body_type, body_feature, skin_feature
Clothing	clothing_top, clothing_bottom, clothing_full_body, clothing_swimwear, clothing_underwear, clothing_traditional, clothing_uniform, clothing_accessory, footwear, headwear, eyewear, handwear, legwear, neckwear
Accessories	jewelry, accessory
Pose/Action	pose, action, gesture, gaze
Setting	location, background, weather, furniture, surface
Objects	weapon, food, object, vehicle, flora, fauna
Composition	composition, framing, art_style, color_effect, text_overlay
Identity	named_character, copyright_series, artist
Count	character_count, grouping
Meta	quality, meta, uncategorized

Using the Pipeline as a Library

All pipeline steps are exposed as classes emitting typed progress events, so custom frontends (like the desktop app) can drive them:

final step = CategorizeStep(
  dataDir: 'data',
  apiKey: 'sk-or-...',
  concurrency: 8,
  cancellation: token, // CancellationToken: cancel() stops gracefully
);
await for (final event in step.run()) {
  switch (event) {
    case PipelineLog(:final message):      print(message);
    case PipelineProgress(:final done, :final total, :final detail): // ...
    case PipelineSummary(:final data):     // machine-readable result
  }
}

Steps: FetchTagsStep, FetchWikisStep, CategorizeStep, FetchImplicationsStep, ExportStep, plus cleanDataDir() for a clean slate.

Project Structure

booru_tag_db_dart/
├── lib/
│   ├── booru_tag_db_dart.dart       # Barrel exports
│   └── src/
│       ├── models/                  # TagCategory, CustomCategory, BooruTag,
│       │                            # CategorizedTag, TagDatabase
│       ├── tools/                   # SearchTagsTool, BrowseCategoryTool,
│       │                            # GetTagDetailsTool, SuggestTagsTool
│       └── pipeline/                # PipelineStep classes + events
│                                    # (shared by CLI and desktop app)
├── bin/                             # Thin CLI wrappers over the pipeline
│   ├── fetch_tags.dart              # Step 1: Fetch from Danbooru API
│   ├── fetch_wikis.dart             # Step 2: Fetch wiki descriptions
│   ├── categorize_tags.dart         # Step 3: AI categorization
│   ├── fetch_implications.dart      # Step 4: Fetch tag implications (optional)
│   ├── embed_tags.dart              # Step 5: Semantic embeddings (optional)
│   ├── export_database.dart         # Step 6: Validate and export
│   ├── clean_data.dart              # Wipe data for a clean-slate run
│   └── stats.dart                   # Pipeline statistics
├── app/                             # Flutter desktop wizard (Linux)
├── data/
│   ├── raw/                         # Raw API responses (gitignored)
│   ├── intermediate/                # Merged + progress (gitignored)
│   └── final/
│       └── tag_database.db          # Final SQLite database (committed)
└── test/