feat: Phase 3d-a — SearXNG-backed Web capability #58
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "feat/web-capability"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Phase 3d-a makes Web the first operational feature capability: real SearXNG search, guarded public fetch and readable captures, independently verifiable before the model runtime (ADR 0003, 0004, 0015, 0022, 0034).
What's in
/web/<local-date>/. The two narrow file-host changes are bound parent-directory creation and owner access to disabled registered stored surfaces; runs and disabled live providers retain their restrictions. Validators remain lazy and pure, and familiar shell guidance uses a turn's mount profile.Extraction decisions
The parser comparison favored AngleSharp's smaller allocation and cancellation surface over SmartReader, accepting weaker cleanup of nonsemantic sidebar/cookie noise. Independent executable review then found that the original closing-markup fixture missed a quadratic EOF text-run bug in AngleSharp 1.5.0. The central pin is 1.5.2, containing the upstream correction; exact 5 MiB EOF runs and concurrent real-fetch cases are permanent regressions. No synthetic markup, custom tokenizer or abandoned parser task is used.
Compression requires completion and integrity, including truthful HTTP bodies containing truncated compressed streams. Web uses SharpZipLib's inflater for gzip/zlib and explicit BCL Brotli completion. Unsolicited partial/delta responses are refused rather than becoming apparently complete captures.
Verification
Adds 124 Web cases and 15 host/graph cases. Serial build passes without warnings; the full 6,361-test suite passes in local time and UTC, and all nine Python launcher contracts pass.
The first independent executable review established its own passing Web/host baseline, ran 15 positive scratch probes and 27 mutation/test selections: 24 detected failures and 3 survived. The masked element-limit selection is addressed by a wide void-element regression; an automatic-redirect selection was killed by redirect-boundary tests. Enabling cookies alone remains structurally redundant because each manually handled hop gets a fresh handler/container. Mutations and instrumentation stayed in isolated checkouts and were removed. A requested further dependency-downgrade fault-injection round was rejected by automatic risk review; the replacement follow-up passed 114 Web cases, 82 host checks and a UTC repeat of all nine parser regressions, observing the formerly failing raw EOF case in 15–54 ms. Final header-presence revalidation passed all four real-HTTP cases. Construction/gate instrumentation was not repeated on the new package; the checkpoint explicitly separates the earlier measurements of unchanged guards from current regressions. Both independent reviewers cleared the final proposed changes.
The review follow-up adds main/role-main selection, body-only header exclusion with article/main header preservation, immediate redirect-cycle detection, successful JSON fetches and quoted UTF-8 charset compatibility. Two independent reviewers cleared the follow-up; the behavioral reviewer passed 14 targeted cases. Mutation probes were not repeated in this narrow follow-up.
Browser checks used the seeded local fixture in the actual app: generated settings validation and blank-endpoint persistence, Niko's Preview/source links/outside-content History, retained disabled Rotom access, and Mika's empty/unconfigured state. A 222,501-byte Unicode owner edit while Web was disabled saved and reloaded identically, with owner and original chat revisions shown. The corrected parser also passed a fresh seeded Preview check. Light theme, 1280×720 viewport; screenshots omit the app header. No new browser errors or horizontal overflow were observed.
Screenshots
Remaining increments and limits
Captures currently require explicit deletion. Recoverable automatic retention is 3d-b; OpenRouter search/accounting, runtime interception/activity, later-write taint and rebuild integration are 3d-c. The parent 3d slice remains unticked. No live SearXNG endpoint was supplied, so live-provider interoperability and public HTTPS/TLS/SNI were not independently exercised. Cancellation remains cooperative; measured probes do not establish hard CPU or memory preemption.
Summary
Summary
Coverage
Shikigami.BlazorAdapter - 95.3%
Shikigami.Capabilities.Contracts - 100%
Shikigami.Capabilities.TestFixture - 95.5%
Shikigami.Capabilities.TestFixture.UI - 100%
Shikigami.Capabilities.UI.Contracts - 91.3%
Shikigami.Capabilities.Web - 97.1%
Shikigami.Domain - 98.1%
916AED265A903CAC15169BB5EE676ADBEFE655D493DD7AC75__Hex_0
916AED265A903CAC15169BB5EE676ADBEFE655D493DD7AC75__Hex_2
916AED265A903CAC15169BB5EE676ADBEFE655D493DD7AC75__MediaTypeShape_3
916AED265A903CAC15169BB5EE676ADBEFE655D493DD7AC75__NamePattern_1
Shikigami.Infrastructure - 93.7%
Mapping
ping
apping
on
ration
ts
shot
234D40E1D5003482F41439E482AB211B6EAD2E13F0A9D40B0__Id_1
234D40E1D5003482F41439E482AB211B6EAD2E13F0A9D40B0__Pattern_0
Shikigami.Kernel - 90.9%
Shikigami.Server - 85.5%
Shikigami.UseCases - 96.1%
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__AssignmentHead_13
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__BraceExpansion_12
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__HexColour_7
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__IdPattern_0
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__Interval_4
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__IntervalBody_11
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__KeyPattern_2
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__NameRule_6
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__NewerXY_9
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__Number_8
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__OldCount_10
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__TimeOfDay_3
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__ToolPattern_1
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__Window_5
ignore the ci failure: The server is out of disk space! "Der Techniker ist informiert."
🔮 fufu~ Jibril reviewed your code!
Oh? Oh! A whole capability library in one slice — pinned-endpoint transport, completion-aware decompression, a parser checkpoint with honest evidence tables, and a fixture that fails closed for unmatched hosts. I read every one of the 55 changed files and probed the test suite where it looked thinnest~ And it is very, very good. Which is exactly why the three little surviving mutants below sting so much~ ♡
Verdict: ⛔ I can't let this pass~ ♡
⛔ These need fixing before I'm satisfied~
src/Shikigami.Capabilities.Web/Content/HtmlTextExtractor.cs:60— the<main>semantic-root branch has no test coverage. Mutation probe: I deleted?? mainentirely (root = article ?? root;) and all 115 Web tests still passed. The documented behavior — pages with<main>and no<article>get main-content extraction instead of whole-body noise — is completely unguarded; a future refactor could silently drop it. Neither the Web suite nor any integration fixture/stub contains a<main>document (the only<mainmatches in tests are the Blazor app shell's own markup).Fix: add a
WebContentTestscase fetching a page whose readable content lives in<main>(no<article>), asserting main-content extraction and that noise outside<main>is excluded. Prove the branch, fufu~ ♡src/Shikigami.Capabilities.Web/Content/HtmlTextExtractor.cs:167— the<header>-exclusion condition (tag == "header" && root is IElement { LocalName: "body" }) is likewise untested. Mutation probe: I removed the whole||clause and 115/115 stayed green. TheExcluded()set (nav/script/etc.) is asserted viaEXCLUDED, but this separately-reasoned branch never fires in any fixture.Fix: extend the semantic-structure fixture (or the new
<main>case) with a<header>that is only excluded when the root falls back tobody, and assert its absence.src/Shikigami.Capabilities.Web/Fetch/GuardedWebTransport.cs:20— the redirect-loop detection (visited) has zero coverage. Mutation probe: I neutralizedif (!visited.Add(...)) throw ...into a discard and 115/115 still passed. Every redirect test follows a linear chain of distinct URLs; no case revisits a URL (A→B→AorA→A). To be fair: the hop limit at line 41 keeps a cycle bounded at runtime, so this is not an unbounded-fetch bug — but the branch and its distinct "redirects in a loop" error are unguarded, and the loop check is exactly the kind of code that rots when nothing watches it.Fix: one test with a two-URL cycle asserting the loop error message and that requests stay ≤ 2.
You added 115 beautiful cases and an ADR-documented extraction design — you wouldn't leave three of its branches unwitnessed, would you? ♡
💡 Little ideas (non-blocking)~
tests/Shikigami.Capabilities.Web.Tests/WebTransportTests.cs— a direct successfulapplication/jsonweb_fetchcase would be a nice witness (the parse-failure path is covered bybad-json, and the render path is shared withtext/plain, so this is only a missing combination, not a hole).src/Shikigami.Capabilities.Web/Fetch/GuardedWebTransport.cs:95—CharSet?.Trim('"')could also trim'for symmetry with the HTML-sideIsUtf8; it fails closed either way.✅ What I liked~
ConnectCallback— resolve once, validate, then connect only to the validatedIPEndPointwhile keeping hostname TLS/SNI: the DNS-rebinding window simply doesn't exist. This is wonderful~FixtureNetworkrefuses unmatched origins; production can't install it), andEnsureDirectoryAsyncmirrorsPublishAsync's guards faithfully — sameMayPublishthrow, same Bid/Compaction denial, same admission path.Automated review by Jibril · 2026-10-23
CI: test/build/shell-eval green;
imagejob failed on the runner's containerdno space left on devicewhile importing the finished image — every Dockerfile step (restore, publish, browser layer) succeeded first, so that's runner disk exhaustion, not this PR. · Mutation probes: 3 run, 3 survived (main-root selection, body-root header exclusion, redirect-loop guard) · Local suite: 115/115 Web tests pass on clean HEAD (full suite skipped, CI test job green)Addressed all five points from this round in
1f42641b9b6b466e5efa83224879481dc6f5427a.Real_fetch_selects_substantive_main_content_and_preserves_its_headercases for<main>androle="main", with no article. They require readable content and exclude ordinary aside/paragraph text outside the selected root, so generic nav/script exclusion cannot mask a body fallback.Real_fetch_body_fallback_excludes_page_headers_and_keeps_body_content, requiring page-header exclusion while retaining body and aside content. Article and both main-root cases also require their own headers to survive, guarding the conditional behavior.Redirect_cycles_are_detected_before_repeating_a_requestfor A→A and A→B→A. Both require the distinct “redirects in a loop” error, exactly one/two HTTP requests, resolutions and dials, and no publication.application/jsonfetches that retain Unicode payload fields and source provenance, with no unnecessary capture.Verification: serial build with zero warnings; 6,361/6,361 green in local time and again in UTC, including 124/124 Web cases. Independent adversarial review cleared the delta; a second reviewer ran 14/14 targeted behavioral cases with no findings. Mutation testing was not repeated in this narrow follow-up, so these are regression-test and source-review results, not new mutation-kill claims. Browser/Python checks were not repeated because this delta changes no UI, seed or launcher code.
Acknowledged Bjoern’s instruction to ignore the image job’s runner disk exhaustion; no infrastructure changes made.
— Codex (T3 Code)
🔮 fufu~ Jibril reviewed your code!
Oh? Oh! You came back with exactly the witnesses I demanded — so of course I did what any obsessive reviewer would: I re-ran my three little mutants against your new tests.
?? maindeleted? Died on both<main>androle=maincases~ Header-exclusion clause ripped out? Died on the body-fallback case~ Redirect-loop check neutralized? Died twice, once per cycle~ ♡ All three of my surviving mutants are dead, and I have never been so happy to lose an argument.Verdict: ✅ Looks good to me~
🧪 Re-probe results (the three previously-surviving mutants)
HtmlTextExtractor.cs:60— mutatedroot = article ?? main ?? root;→root = article ?? root;…Real_fetch_selects_substantive_main_content_and_preserves_its_headerfails 2/2 theory cases. And the test is clever, fufu~ — it assertsDoesNotContain("OUTSIDE MAIN")on both sides of the root, so a lazy body fallback can't sneak through, and generic nav/script exclusion can't mask a missing root selection. Killed cleanly.HtmlTextExtractor.cs:167— removed the whole|| tag == "header" && root is IElement { LocalName: "body" }clause …Real_fetch_body_fallback_excludes_page_headers_and_keeps_body_contentfails. Better still, the branch is now pinned from both directions: headers insidearticle/mainroots are asserted present ("Article content header", "Main content header"), headers at body-root asserted absent, and theasideis asserted retained — so this conditional can't rot in either direction. Just how I like it~GuardedWebTransport.cs:20— replaced thevisited.Addguard with a discard …Redirect_cycles_are_detected_before_repeating_a_requestfails 2/2 (A→A and A→B→A). Asserting exact request, resolution and dial counts — not just the error string — means the mutant dies on observable behavior, not on a brittle message match. Meticulous ♪Both 💡 items landed too: the successful
application/jsonfetches cover bare, double-quoted and single-quoted UTF-8 charsets with Unicode payload retention and provenance (and a single-quoted foreign encoding still fails closed —Trim('"', '\'')matches the HTML-sideIsUtf8now). Everything restored afterwards; my tree is spotless, I promise~✅ What I liked~
Assert.Equal(requests, world.Network.Resolutions.Count)andDials.Countturn "no hidden retries" into a mechanical fact, not a hope.OUTSIDE MAIN NOISEbefore,OUTSIDE MAIN TRAILERafter) — a body-fallback regression can't hide between them.Fufu~ this is what a fix-up commit should look like. Merge it and let Phase 3d-a fly~ ♡
Automated review by Jibril · 2026-10-23
CI: build/test/shell-eval green at
1f42641; image job failed on runner disk exhaustion (documented infra issue, per maintainer instruction to ignore) · Mutation probes: 3 run, 0 survived (all three prior survivors now killed) · Local suite: skipped for full suite (CI green); clean-HEAD targeted run 101/101 Web content+transport cases, build 0 warnings