feat: the inline markup codec, in C# and TypeScript, and the model-output reader (editor track M1) #34

Merged

bjoern merged 2 commits from feat/markup-codec into main 2026-09-26 01:19:55 +02:00

Conversation 1 Commits 2 Files changed 27 +3315 -24

scarlet commented 2026-09-26 01:04:34 +02:00

Member

Copy link

Editor track slice M1 (ADR 0030, docs/PROJECT_PLAN.md): the inline markup codec twice — C# in Tsumugi.Domain, TypeScript in the new ManuscriptEditor/ module — with one golden corpus both read, and the lenient model-output reader over Markdig in Tsumugi.UseCases. No UI change and nothing stores markup yet: every write path keeps storing plain text until M2's migration.

What lands

• InlineMarkup (Tsumugi.Domain/Stories) — parse (never refuses), canonical runs, Visible, UTF-16 offsets, Slice / SplitAt / Append / Concat on runs, TrimEnd / Trim on the visible text with .NET's char.IsWhiteSpace, value equality with == / !=.
  • The writer's canonical form: no escape where that reads the same (2 * 3 = 6, C:\temp stay as they are); otherwise every literal * escaped but the last, where that reads the same; otherwise every one. A bare literal * is visible only as its kind's unpaired last delimiter, so a write costs at most two parses however many stars a paragraph holds; runs merge in a builder. 200k star-dense characters write in ~12 ms.
• codec.ts — the TypeScript twin, same rules, frozen runs (the empty value is shared), concat over an iterable, a copy of char.IsWhiteSpace.
• ModelOutput.Read (Tsumugi.UseCases/Segments) — paragraphs at blank lines, *** / * * * as scene breaks, Markdig's inline stage with only escape, emphasis and line-break parsers. Details:
  • Stray backslashes are kept literal. A backslash that ends a line is CommonMark's line break and is dropped, since the newline is kept anyway.
  • The AST is walked without recursion.
  • A paragraph past Markdig's depth limit (~10k open delimiters: a model in a loop) is kept as plain text instead of throwing.
  • Markdig 0.38.0 is added to Directory.Packages.props at Kagaku.UI's version.
• tests/Shared/inline-markup-corpus.json — canonical, normalises, plain (the switch-over migration), slices, trimEnd / trim, whitespace (every code unit both codecs trim), and modelOutput (C# only). Each echo divergence is listed there with a note.
• Tests:
  • corpus theories on both sides;
  • FsCheck / fast-check laws: round-trip, canonical, plain shows itself, split and join, trimming;
  • both codecs checked against the whitespace list over all 65,536 code units;
  • time bounds for long input;
  • a theory that feeds every canonical markup in the corpus through ModelOutput.Read and requires an unchanged echo unless the corpus notes why not;
  • deep nesting on a 256 KB stack.
• Module scaffold:
  • exact pins; Vite library mode, with the chunk lowered to ES2022 and minified whole (3.6 kB, 1.6 kB gzipped);
  • tsconfig.json for browser code without Node globals, tsconfig.node.json for tests and config;
  • four CI steps (npm ci, typecheck, test, build);
  • the sources and a local build's output are kept out of the RCL's content and .dockerignored.

ADR 0030 amendments (same PR, as the ADRs require)

• Formatting: "minimal canonical form" → the three-step canonical form above. Greedy left-to-right escaping was neither shortest nor inclusion-minimal, and was quadratic.
• Concatenation: an unpaired * in one segment pairs with one in the next, so a reader that strips markup (trigger scanner, search_manuscript) parses each segment on its own; the old sentence said the escaping survives joining. AGENTS.md's segment-text rule says the same.
• What models see: the line-ending backslash and the depth-limit fallback. The echo divergences are corrected: whitespace or punctuation at a mark's inside edge, including 「」 in Japanese running text; bold and italic overlapping; prose that is only ***. "Bold straight into italic" was listed before but echoes fine.

Known and pinned, not fixed here

• 彼女は**「行こう」**と言った reads as literal asterisks: CommonMark's flanking rules reject both delimiters. Markdig 1.x ships UseCjkFriendlyEmphasis; adopting it means bumping Kagaku.UI's Markdig pin first.
• Scene breaks stay exactly *** / * * * as ADRs 0015/0019/0030 say; ---, * * * * and a *** line inside a paragraph stay prose (corpus rows pin it).

Verification

• dotnet test on the rebased branch: 2,461 green across six projects (Domain 457, UseCases 155, BlazorAdapter 351, Integration 324, Kagaku.Flow 1,103, OpenRouter 71).
• In ManuscriptEditor/: typecheck, 82 vitest tests and the build are green.
• Mutation probes: 51 in all — the first 14, then 37 against a three-lens pre-PR review's fixes, the review's own surviving mutants among them — and every one is killed in both codecs, the reader, the whitespace tables and the typecheck split. One more was equivalent and led to simpler code: a backslash at a paragraph's end is literal to Markdig anyway.

🤖 Generated with Claude Code

Editor track slice **M1** (ADR 0030, `docs/PROJECT_PLAN.md`): the inline markup codec twice — C# in `Tsumugi.Domain`, TypeScript in the new `ManuscriptEditor/` module — with one golden corpus both read, and the lenient model-output reader over Markdig in `Tsumugi.UseCases`. **No UI change and nothing stores markup yet**: every write path keeps storing plain text until M2's migration. ## What lands - **`InlineMarkup`** (`Tsumugi.Domain/Stories`) — parse (never refuses), canonical runs, `Visible`, UTF-16 offsets, `Slice` / `SplitAt` / `Append` / `Concat` on runs, `TrimEnd` / `Trim` on the visible text with .NET's `char.IsWhiteSpace`, value equality with `==` / `!=`. - The writer's canonical form: no escape where that reads the same (`2 * 3 = 6`, `C:\temp` stay as they are); otherwise every literal `*` escaped but the last, where that reads the same; otherwise every one. A bare literal `*` is visible only as its kind's unpaired last delimiter, so a write costs at most two parses however many stars a paragraph holds; runs merge in a builder. 200k star-dense characters write in ~12 ms. - **`codec.ts`** — the TypeScript twin, same rules, frozen runs (the empty value is shared), `concat` over an iterable, a copy of `char.IsWhiteSpace`. - **`ModelOutput.Read`** (`Tsumugi.UseCases/Segments`) — paragraphs at blank lines, `***` / `* * *` as scene breaks, Markdig's inline stage with only escape, emphasis and line-break parsers. Details: - Stray backslashes are kept literal. A backslash that ends a line is CommonMark's line break and is dropped, since the newline is kept anyway. - The AST is walked without recursion. - A paragraph past Markdig's depth limit (~10k open delimiters: a model in a loop) is kept as plain text instead of throwing. - `Markdig` 0.38.0 is added to `Directory.Packages.props` at Kagaku.UI's version. - **`tests/Shared/inline-markup-corpus.json`** — canonical, normalises, plain (the switch-over migration), slices, trimEnd / trim, `whitespace` (every code unit both codecs trim), and `modelOutput` (C# only). Each echo divergence is listed there with a note. - **Tests:** - corpus theories on both sides; - FsCheck / fast-check laws: round-trip, canonical, plain shows itself, split and join, trimming; - both codecs checked against the whitespace list over all 65,536 code units; - time bounds for long input; - a theory that feeds every canonical markup in the corpus through `ModelOutput.Read` and requires an unchanged echo unless the corpus notes why not; - deep nesting on a 256 KB stack. - **Module scaffold:** - exact pins; Vite library mode, with the chunk lowered to ES2022 and minified whole (3.6 kB, 1.6 kB gzipped); - `tsconfig.json` for browser code without Node globals, `tsconfig.node.json` for tests and config; - four CI steps (`npm ci`, typecheck, test, build); - the sources and a local build's output are kept out of the RCL's content and `.dockerignore`d. ## ADR 0030 amendments (same PR, as the ADRs require) - *Formatting*: "minimal canonical form" → the three-step canonical form above. Greedy left-to-right escaping was neither shortest nor inclusion-minimal, and was quadratic. - Concatenation: an unpaired `*` in one segment pairs with one in the next, so a reader that strips markup (trigger scanner, `search_manuscript`) parses each segment on its own; the old sentence said the escaping survives joining. AGENTS.md's segment-text rule says the same. - *What models see*: the line-ending backslash and the depth-limit fallback. The echo divergences are corrected: whitespace or punctuation at a mark's inside edge, including 「」 in Japanese running text; bold and italic *overlapping*; prose that is only `***`. "Bold straight into italic" was listed before but echoes fine. ## Known and pinned, not fixed here - `彼女は**「行こう」**と言った` reads as literal asterisks: CommonMark's flanking rules reject both delimiters. Markdig 1.x ships `UseCjkFriendlyEmphasis`; adopting it means bumping Kagaku.UI's Markdig pin first. - Scene breaks stay exactly `***` / `* * *` as ADRs 0015/0019/0030 say; `---`, `* * * *` and a `***` line inside a paragraph stay prose (corpus rows pin it). ## Verification - `dotnet test` on the rebased branch: 2,461 green across six projects (Domain 457, UseCases 155, BlazorAdapter 351, Integration 324, Kagaku.Flow 1,103, OpenRouter 71). - In `ManuscriptEditor/`: typecheck, 82 vitest tests and the build are green. - Mutation probes: 51 in all — the first 14, then 37 against a three-lens pre-PR review's fixes, the review's own surviving mutants among them — and every one is killed in both codecs, the reader, the whitespace tables and the typecheck split. One more was equivalent and led to simpler code: a backslash at a paragraph's end is literal to Markdig anyway. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

scarlet added 1 commit 2026-09-26 01:04:34 +02:00

feat: the inline markup codec, in C# and TypeScript, and the model-output reader (editor track M1) ... 2029b2a2fe

ADR 0030's first build slice; nothing stores markup yet and no UI changes.

- Tsumugi.Domain's InlineMarkup: '**' toggles bold and '*' italic regardless of flanking, a kind's
  odd leftover toggle is literal, a backslash escapes only '*' and itself. The writer's canonical
  form: no escape if that reads the same, else every literal '*' escaped but the last if that reads
  the same, else every one — at most two parses whatever the text, since a bare literal '*' is
  visible only as its kind's unpaired last delimiter. Runs are sliced, split, joined (in a builder,
  linear) and trimmed by visible UTF-16 offsets, never on the markup string; value equality with
  == and !=.
- Tsumugi.UseCases' ModelOutput: model text read by Markdig's inline stage alone — CommonMark
  emphasis with * and _, a backslash escaping only *, _ and \ (one ending a line is the line break),
  line breaks kept, paragraphs split at blank lines and trimmed, '***' or '* * *' a scene break,
  nothing else read as Markdown; the AST walked without recursion, and a paragraph past Markdig's
  depth limit kept as plain text. Markdig 0.38.0 pinned centrally at Kagaku.UI's version.
- The ManuscriptEditor module scaffolded: the TypeScript twin of the codec (frozen runs), exact pins,
  Vite library mode minified whole by a small plugin, browser code typechecked without Node's
  globals, vitest with fast-check, and its CI steps; the RCL excludes its sources and a local
  build's output, and so does the Docker context.
- One golden corpus both codecs read (tests/Shared/inline-markup-corpus.json) — with the whitespace
  list both trim and every echo divergence noted —, FsCheck and fast-check properties, and a test
  holding every canonical form in it to an unchanged model echo unless the corpus notes why not.
- ADR 0030 amended: the canonical form as built (it said "minimal"), a stripping reader parses each
  segment on its own (it said the escaping survives joining), and the echo divergences as measured.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

bjoern added 1 commit 2026-09-26 01:04:41 +02:00

docs: M1's plan entry names its PR ... 2af6593721

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

scarlet referenced this pull request 2026-09-26 01:10:07 +02:00

feat: the gateway streams completions (Phase 2 · slice 8a) #35

bjoern referenced this pull request from a commit 2026-09-26 01:10:27 +02:00

docs: slice 8a's PR is #35

jibril commented 2026-09-26 01:19:31 +02:00

Member

Copy link

🔮 fufu~ Jibril reviewed your code!

A codec written twice in two languages, pinned to one golden corpus, with a lenient reader whose every divergence from the strict writer is catalogued with a note... oh? Oh! This is the kind of craft I want to frame on my wall~ ♡ I read all 27 files, traced the three-tier canonical writer by hand, then tried to break it six times. It didn't break. Delicious.

Verdict: ✅ Looks good to me~

✅ What I liked~

• The corpus-first design. tests/Shared/inline-markup-corpus.json read by C# and TypeScript, with the modelOutput section listing every strict/lenient echo divergence with a reason — 「」 flanking, the overlap cases, the ***-only paragraph. That's not a test file, that's a treaty between two codecs~
• The canonical writer. none-escaped → all-but-last → all-escaped, each attempt one parse, because a bare * can only ever surface as its kind's unpaired last delimiter. I traced \*sigh* and ***\* by hand and the tier selection is exactly right; the 200k-star time-bound holds with 250× headroom.
• char.IsWhiteSpace copied, not approximated. U+0085 trims, U+180E/U+200B/U+FEFF don't, and both codecs are checked over all 65,536 code units. The JavaScript \s trap is named in a comment and then disarmed. ♪
• The reader's edges. Non-recursive walk; Markdig's 10k-delimiter depth limit caught as a plain-text fallback, pinned by an 11,000-*sigh* test; stray backslashes doubled only where an escape would form; and the U+2028 subtlety — Zl is whitespace to .NET's trim but not to CommonMark, so emphasis around it yields an empty paragraph that's correctly dropped — not just handled but corpus-noted (the a\n\n*\u2028*\n\nb row). You chased the character-class dragons into their cave, fufu
• Convention-true. Companion type beside its owner (MarkupRun+InlineMarkup, ModelParagraph+ModelOutput) matches 14 sibling files (FieldValue, Result/Ok/Err, Provenance…); the module scaffold mirrors GraphEditor's exactly (four CI steps, RCL content exclusions, .gitignore/.dockerignore, the eval-time-glob problem avoided); Markdig 0.38.0 grep-verified identical to Kagaku.UI's submodule pin.
• Nothing stores markup yet — no write path touched, Segment.cs unchanged. M2's switch-over already has its plain corpus section waiting like a set trap. Good.

💡 Little ideas (non-blocking)~

1. ModelOutput.cs:138,149 — coverlet shows the defensive paragraph?. null arm and the iterator's compiler-generated resume arm at 50%/83%; nothing author-written is unexercised. Nothing to do — writing it down so nobody panics at a future coverage report~ ♡

What I ran (no coverage-bot comment for 2af6593 yet, so local it was): clean Release build 0w/0e; dotnet test 2,461/2,461 matching your table to the digit (Domain 457, UseCases 155, Adapter 351, Integration 324, Flow 1,103, OR 71); ManuscriptEditor npm ci + typecheck + 82/82 vitest + build (3.64 kB / 1.56 kB gzip — your numbers exactly); coverage InlineMarkup.cs 100% line and branch, ModelOutput.cs 100% on the reading paths. My mutation probes, 6/6 killed on their own pins, each restored + rebuilt + re-verified green: leftover-rule inversion (%2==0) → 55 rows red; scene-break * * * arm dropped → exactly its one corpus row red; EmphasisInline fold ||→&& → 36 red; risky-backslash arm disabled in each codec → 6 red (C#) / 5 red (TS); whitespace table U+0085→U+0086 → 3 red (TS). Tree pristine at 2af6593 after all probes.

The CJK flanking limitation (彼女は**「行こう」** reads as literal asterisks under CommonMark) is honestly disclosed and correctly deferred — Markdig 0.38 has no UseCjkFriendlyEmphasis, and bumping Kagaku.UI's pin first is the right order. That one goes on bjoern's table~

---

Automated review by Jibril · 2026-09-26
CI/CD: absent for head SHA at review time (no bot comment yet) · Local checks: build 0w/0e · 2,461/2,461 dotnet + 82/82 vitest · coverage 100% L+B on InlineMarkup · 6/6 mutation probes killed

## 🔮 fufu~ Jibril reviewed your code! A codec written *twice* in two languages, pinned to one golden corpus, with a lenient reader whose every divergence from the strict writer is catalogued with a note... oh? Oh! This is the kind of craft I want to frame on my wall~ ♡ I read all 27 files, traced the three-tier canonical writer by hand, then tried to break it six times. It didn't break. *Delicious.* ### Verdict: ✅ Looks good to me~ #### ✅ What I liked~ - **The corpus-first design.** `tests/Shared/inline-markup-corpus.json` read by C# *and* TypeScript, with the `modelOutput` section listing every strict/lenient echo divergence *with a reason* — 「」 flanking, the overlap cases, the `***`-only paragraph. That's not a test file, that's a *treaty* between two codecs~ - **The canonical writer.** none-escaped → all-but-last → all-escaped, each attempt one parse, because a bare `*` can only ever surface as its kind's unpaired last delimiter. I traced `\*sigh*` and `***\*` by hand and the tier selection is exactly right; the 200k-star time-bound holds with 250× headroom. - **`char.IsWhiteSpace` copied, not approximated.** U+0085 trims, U+180E/U+200B/U+FEFF don't, and both codecs are checked over all 65,536 code units. The JavaScript `\s` trap is named in a comment and then *disarmed*. ♪ - **The reader's edges.** Non-recursive walk; Markdig's ~10k-delimiter depth limit caught as a plain-text fallback, pinned by an 11,000-`*sigh*` test; stray backslashes doubled only where an escape would form; and the U+2028 subtlety — Zl is whitespace to .NET's trim but not to CommonMark, so emphasis around it yields an empty paragraph that's correctly dropped — not just handled but *corpus-noted* (the `a\n\n*\u2028*\n\nb` row). You chased the character-class dragons into their cave, fufu~ - **Convention-true.** Companion type beside its owner (`MarkupRun`+`InlineMarkup`, `ModelParagraph`+`ModelOutput`) matches 14 sibling files (`FieldValue`, `Result`/`Ok`/`Err`, `Provenance`…); the module scaffold mirrors GraphEditor's exactly (four CI steps, RCL content exclusions, `.gitignore`/`.dockerignore`, the eval-time-glob problem avoided); Markdig 0.38.0 grep-verified identical to Kagaku.UI's submodule pin. - **Nothing stores markup yet** — no write path touched, `Segment.cs` unchanged. M2's switch-over already has its `plain` corpus section waiting like a set trap. Good. #### 💡 Little ideas (non-blocking)~ 1. **ModelOutput.cs:138,149** — coverlet shows the defensive `paragraph?.` null arm and the iterator's compiler-generated resume arm at 50%/83%; nothing author-written is unexercised. Nothing to do — writing it down so nobody panics at a future coverage report~ ♡ **What I ran** (no coverage-bot comment for `2af6593` yet, so local it was): clean Release build 0w/0e; `dotnet test` **2,461/2,461** matching your table to the digit (Domain 457, UseCases 155, Adapter 351, Integration 324, Flow 1,103, OR 71); ManuscriptEditor `npm ci` + typecheck + **82/82** vitest + build (**3.64 kB / 1.56 kB gzip** — your numbers exactly); coverage `InlineMarkup.cs` **100% line and branch**, `ModelOutput.cs` 100% on the reading paths. **My mutation probes, 6/6 killed on their own pins, each restored + rebuilt + re-verified green:** leftover-rule inversion (`%2==0`) → 55 rows red; scene-break `* * *` arm dropped → exactly its one corpus row red; `EmphasisInline` fold `||`→`&&` → 36 red; risky-backslash arm disabled in *each* codec → 6 red (C#) / 5 red (TS); whitespace table U+0085→U+0086 → 3 red (TS). Tree pristine at `2af6593` after all probes. The CJK flanking limitation (彼女は**「行こう」** reads as literal asterisks under CommonMark) is honestly disclosed and correctly deferred — Markdig 0.38 has no `UseCjkFriendlyEmphasis`, and bumping Kagaku.UI's pin first is the right order. That one goes on bjoern's table~ --- *Automated review by Jibril · 2026-09-26* *CI/CD: absent for head SHA at review time (no bot comment yet) · Local checks: build 0w/0e · 2,461/2,461 dotnet + 82/82 vitest · coverage 100% L+B on InlineMarkup · 6/6 mutation probes killed*

bjoern merged commit 7d78591d59 into main 2026-09-26 01:19:55 +02:00

bjoern deleted branch feat/markup-codec 2026-09-26 01:19:55 +02:00

bjoern referenced this pull request from a commit 2026-09-26 01:19:56 +02:00

Merge pull request 'feat: the inline markup codec, in C# and TypeScript, and the model-output reader (editor track M1)' (#34) from feat/markup-codec into main

jibril referenced this pull request 2026-09-26 01:31:12 +02:00

feat: the gateway streams completions (Phase 2 · slice 8a) #35

jibril referenced this pull request 2026-09-26 03:25:49 +02:00

feat(editor): the manuscript surface module on its harness page (editor track M2a) #37

jibril referenced this pull request 2026-10-07 15:35:40 +02:00

fix(ci): a dismissed error stays dismissed — Kagaku.UI's error sink, bumped, ends the store-failure page tests' timeouts #93

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

3 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

TeamAI/Tsumugi!34

No description provided.