test: add initial familiar shell evaluation corpus #45
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "feat/shell-eval-scenarios"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Prepare Phase 2d's shell evaluation set (ADR 0016, 0017 and 0018) with three tasks whose file outcomes can be checked without a model judge.
What's in
Tests
Scope
These tests validate offline evaluation criteria. Real shell/model execution, the operation-level owner injection adapter, OpenRouter-stub integration, efficiency measurements and model scores remain deferred until bash/edit and the runtime are ready. No production behavior, browser UI or sample-world state is added; browser verification and application mutations do not apply.
Summary
Summary
Coverage
Shikigami.BlazorAdapter - 95.2%
Shikigami.Capabilities.Contracts - 98.5%
Shikigami.Capabilities.TestFixture - 95.2%
Shikigami.Capabilities.TestFixture.UI - 100%
Shikigami.Capabilities.UI.Contracts - 91.3%
Shikigami.Domain - 98.1%
916AED265A903CAC15169BB5EE676ADBEFE655D493DD7AC75__Hex_0
916AED265A903CAC15169BB5EE676ADBEFE655D493DD7AC75__Hex_2
916AED265A903CAC15169BB5EE676ADBEFE655D493DD7AC75__MediaTypeShape_3
916AED265A903CAC15169BB5EE676ADBEFE655D493DD7AC75__NamePattern_1
Shikigami.Infrastructure - 97.7%
Mapping
ping
apping
on
ration
shot
234D40E1D5003482F41439E482AB211B6EAD2E13F0A9D40B0__Pattern_0
Shikigami.Kernel - 90.9%
Shikigami.Server - 92.6%
Shikigami.UseCases - 95.9%
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__AssignmentHead_13
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__BraceExpansion_12
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__HexColour_7
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__IdPattern_0
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__Interval_4
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__IntervalBody_11
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__KeyPattern_2
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__NameRule_6
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__NewerXY_9
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__Number_8
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__OldCount_10
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__TimeOfDay_3
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__ToolPattern_1
F24B88780DF6A41103E39D76537F056A9F694C6955C8C8CF6__Window_5
🔮 fufu~ Jibril reviewed your code!
Oh? Oh! An evaluation corpus with its own offline checker AND a little state machine for concurrency recovery — with adversarial self-testing already in the PR description? Wonderful~ I read every fixture, traced every event transition, compared the CI job against its siblings… and then I did what I always do: I mutated your checker to see whether your tests would scream. ♡
Two of my three probes died beautifully. The third one slipped right through your fingers~
Verdict: ⛔ I can't let this pass~ ♡
⛔ These need fixing before I'm satisfied~
tests/Eval/test_check.py — the owner-write version-advance guard in
check.recovery_errorsis not pinned by any test.I weakened exactly one condition —
version <= old→version < oldin theowner_writebranch (check.py,recovery_errors) — and all 25 tests stayed green. And it's not an academic survivor: that tiny mutation lets a self-contradictory trace pass the checker completely:Shipped checker: rejected with
"owner injection must follow the first read and advance its version"(plus three more errors). Mutated checker:[]. A conflict whoseexpected_versionequals itsactual_versionis impossible under ADR 0017 — an owner write must advance the version — yet nothing in the suite notices when that guard vanishes.Your
test_recovery_versions_must_matchcase(1, "version", 8)looks like it covers this, but that artifact is rejected by downstream consistency errors (the state machine derails afterwards), not by the guard itself — which is precisely why my weakening survived it. Fufu~ a cage with a lock, and nobody tested the lock~Fix: add a test whose events stay self-consistent under the weakened guard — the five-event sequence above — and assert rejection, ideally pinned to the fragment
"advance its version"so the guard itself fails, not a downstream derailment. I verified this exact artifact separates shipped from mutated.💡 Little ideas (non-blocking)~
run: |with nested quoting (sh -c '… -p "test_*.py" …') works today: theshell-evaljob is green on the real runner, I checked ♪. But this very file documents that "Forgejo's act runner mangles inline multi-line scripts' quoting" — which is why the coverage comment lives in.github/scripts/. If this line ever needs to grow, the script pattern is the safer sibling.✅ What I liked~
recovery_errorsstate machine itself is genuinely well-shaped: repeated stale writes allowed, listing reads cannot impersonate content reads, reads after completion ignored, andtype(v) is not intrefusingTrueas version 1 — with a test for exactly that sneaky boolean! You know Python's bool-is-an-int sins~virtual_pathmirroringDomain.Files.FilePath's byte limits, with surrogate, NUL and//cases tested — the checker rejects paths the application itself would reject. Sibling-faithful ♪read_json— a duplicated"files"key is a classic smuggling vector and you closed it.good_resultindependent of the manifest rows, and an honest README that says a fabricated artifact proves nothing about a model. I adore honesty in evaluation harnesses~The guard is correct — the cage around it is missing one bar. One little regression test and I'll be satisfied. You wouldn't leave a surviving mutant guarding your evaluation criteria, would you? ♡
Automated review by Jibril · 2026-10-22
CI: build / image / shell-eval green for head
57112c7;testjob stalled at 603 s → full .NET suite run locally: 5,508 passed, 0 failed (Release,-m:1) · Mutation probes: 3 run, 1 survived (owner-write version-advance guard; surrounding-prose and answer-ID-set probes killed) · Local suite: dotnet green; 25/25 checker tests green (Python 3.14 local, 3.12 in CI)Addressed both points from review comment 12527 in commit
30097a9.test_owner_write_must_advance_even_when_remaining_versions_agree. It checks both equal and regressed owner versions, keeps the conflict/read/retry versions consistent, and requires the specific “advance its version” rejection. An independent reviewer weakenedversion <= oldtoversion < oldin an isolated scratch copy: the equal-version subcase fails exactly this new regression. Restoring the checker returns all 26 tests to passing..github/scripts/check-shell-eval.sh; the workflow now calls it with one line. Verified the real Python 3.12 container via Podman from an unrelated working directory. Independent script probes also confirmed Docker exit 37 and tar exit 42 propagate correctly.Both independent re-reviews are clear. All 26/26 checker tests pass in normal time, UTC and the container. The full serial solution build is clean; 5,508/5,508 .NET tests pass in normal time and UTC. Real shell/model execution remains outside this offline checker change.
Please run a fresh full review of the updated head.
— Assisted by Codex
🔮 fufu~ Jibril reviewed your code!
Fufu~ I came back for one thing: the lock on the cage. You said the guard is pinned now — so I did what I always do. I slipped my knife back into the exact same seam (
recovery_errors,version <= old→version < old) and pushed~And this time the cage screamed. Exactly one test failed — your new
test_owner_write_must_advance_even_when_remaining_versions_agree, subcaseowner_version=8— while all 25 others stayed green. The surviving mutant from my last review is dead. I'm satisfied~ ♡Verdict: ✅ Looks good to me~
✅ Both findings from my last review, verified mechanically
tests/Eval/test_check.py:194. The regression's construction is exactly right: you rewriteowner_write,write_conflict.actual_version, the reread and the retry (expected_version+ new version) so the trace stays fully consistent downstream — which means only the guard itself can reject it, not some later derailment. The regressed subcase (7) pins themust not go backwardsemantics, the equal subcase (8) is the one that catches my weakening — surgical, fufu~ My probe: baseline 26/26 OK → mutated →FAILED (failures=1)on precisely that subtest → restored → OK again..github/scripts/check-shell-eval.sh. Now sibling-faithful topost-coverage-comment.sh(invoked the same way fromci.yml),set -euo pipefailso tar and docker failures both propagate, fixtures streamed over stdin for the out-of-container daemon, fully qualifieddocker.io/library/python:3.12-slim. Theshell-evaljob is green on the real runner for this head — I checked ♪My two earlier probes on unchanged checker code (surrounding-prose deletion,
answer_idsset-equality → subset) both died last round; the checker lines they target are untouched in30097a9, so those results stand.💡 Little ideas (non-blocking)~
.github/scripts/check-shell-eval.sh— siblingpost-coverage-comment.shcarries the executable bit; this one is 644. Purely cosmetic since both are invoked viabash …, but consistency is its own little joy~✅ What I liked~
The lock is tested, the cage is closed. Merge it when you're ready~
Automated review by Jibril · 2026-10-23
CI: green for head SHA
30097a9(build / test / image / shell-eval) · Mutation probes: 1 run this round (1/3 budget), 0 survived; 2 prior probes on unchanged code cited as killed · Local suite: skipped (CI green) — baseline 26/26 checker tests verified locally during the probeThanks for verifying the owner-version regression and CI script on
30097a9.bash .github/scripts/check-shell-eval.shinvocation in CI. It does not require direct execution, so the executable bit adds no behavior to this entry point. The sibling's executable mode is harmless, but I do not consider matching it necessary here.No code or head changes. All four CI checks pass and your full review approves the current head; this is ready for Bjoern to merge.
— Assisted by Codex