fix: backups — an archive that provably restores, a failed nightly shown, no snapshot left behind #47
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "fix/backups-hardening"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
The backups get harder to lose: an archive now provably restores, a failed nightly shows on the Backups tab, and a snapshot that fails leaves no file behind (ADR 0026 amended).
tsumugi.dbis aVACUUM INTOcopy, in rollback-journal mode. EF first looks for the database over a read-only connection, where the pragma interceptor's switch to WAL is a write SQLite refuses ("attempt to write a readonly database"), so until now a restore stopped the start.PreMigrationBackup.MigrateAsyncopens the database read-write once before anything reads it, and the switch stays in the file.BackupRestoreTests, new). The real host backs up a story, a stored key and its keyring. The ADR 0026 steps restore the archive onto a fresh data root, and a second host lists the story, decrypts the key and accepts the first host's sign-in cookie. Beside it is a golden keyring: a checked-in, test-only DataProtection key and a ciphertext made under application nameTsumugiand purposeTsumugi.AppSettings, which the booted host must decrypt. Renaming either, or moving the keyring, now fails a test.UnixCreateModewhen they are opened. There is never a chmod afterwards, which the NAS volume may refuse. The sidecar is pinned byte for byte to thesha256sum -cform, with no BOM. The snapshot copy is created empty and 0600 beforeVACUUM INTOfills it.SnapshotPathseam, like the writer's.IBackupStore.JobState). The job and every circuit share this one store, and a test pins that. The last backup's dot on the tab turns to a warning, with one line saying why, in two cases: the last cycle failed, or the newest archive is older than the last slot. A refused click re-reads the store and gives its reason once, in its own alert.DownloadDatabaseSnapshotTests(new) pins the name to the UTC clock, using a fake clock in a +09:00 zone.Docs: ADR 0026 amended (Status line, the artifact, the job, the tab, the download, restore, Consequences). New
deploy/RESTORE.mdrunbook for the compose stack; the tab's hint points to it. No migration, no journaled entity, nothing to seed.Verification
.backups__warningmargin, a CSS value, which the repo does not pin..Nextfor.Last, the UTC zone, the tab's dot/alerts/reducers, the guard, and the download's clock and format.Task.Delayin the scheduler;GetUtcNowinJobState;hh:mmin the missed-slot line;OnCompletedkeeping a staleBehind;BackupJobState.NoneatMaxValue;AddScopedfor the store.GetUtcNowinJobState, the refusal line's date, a staleBehindafter success,hh:mm, a BOM sidecar): each now has a killing test;RESTORE.mdwrongly said an older image will not start (EF refuses nothing): reworded;.zip,.sha256and zip entries were-rw-------, andsha256sum -csaid OK;Follow-ups, outside this arc's paths:
IBackupStore.JobStatehas a default member only soPreMigrationBackupTests.FailingStorecompiles. GiveFailingStoreits ownJobStateand drop the default.SqlitePragmaInterceptorshould skipjournal_modeon read-only connections. The WAL return then moves out ofPreMigrationBackup.GatedAppFactory { DataPath }merges, replaceBackupRestoreTests' private factory with it.🤖 Generated with Claude Code
- A restored archive now starts: a VACUUM INTO copy is in rollback-journal mode, and EF's read-only existence check refused the pragma interceptor's switch to WAL ("attempt to write a readonly database"). PreMigrationBackup.MigrateAsync opens the database read-write once first, which makes the switch for good. - BackupRestoreTests: the real host backs up a story, a stored key and its keyring; the ADR 0026 steps restore the archive onto a fresh data root; a second host lists the story, decrypts the key and accepts the first host's sign-in cookie. Beside it a golden keyring — a checked-in, test-only DataProtection key and a ciphertext made under "Tsumugi" / "Tsumugi.AppSettings" — the booted host must decrypt, so renaming the application, the purpose or moving the keyring fails a test. - The zip and its sidecar are created owner-only (UnixCreateMode 0600 at open, never a chmod the NAS volume may refuse), the sidecar byte for byte the `sha256sum -c` form; the snapshot copy is created empty and owner-only before VACUUM INTO fills it, so the archive's tsumugi.db entry is 0600 too. The owner-only tests stand down under a umask that already hides the difference. - A download snapshot that fails — before the copy or midway through it — or is cancelled deletes its temp file (a SnapshotPath seam on the store, as on the writer). - The job never runs a slot early: a timer that wakes a hair before its wall time waits out the rest, or its archive, stamped 02:59:59, would not cover the 03:00 slot. - The store keeps the last failed cycle (when, why) in memory until one succeeds — a refusal, or a crash it still throws — and reports the slot the nightly job last owed, in the host's zone and date (IBackupStore.JobState); a test pins that the job and every circuit share the one store. The Backups tab turns the last backup's dot to a warning and says why when the last cycle failed or the newest archive is older than the last slot; a refused click re-reads the store and says why once, in its own alert. - Back up now returns early while a run is in flight: a second click that left the browser before the re-render disabled the button starts no second archive. - DownloadDatabaseSnapshotTests pins the download's name to the UTC clock on a fake clock in a +09:00 zone. - deploy/RESTORE.md walks the restore for the compose stack; ADR 0026 amended. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>Summary
Summary
Coverage
Kagaku.Flow - 99.7%
E95E47D27E8B105AC5589487C0997B6B65384AEDBF572144E__ColorToken_2
E95E47D27E8B105AC5589487C0997B6B65384AEDBF572144E__Socket_1
E95E47D27E8B105AC5589487C0997B6B65384AEDBF572144E__TypeId_0
Kagaku.Flow.OpenRouter - 100%
Kagaku.Flow.Testing - 100%
Tsumugi.BlazorAdapter - 95.6%
Tsumugi.Domain - 96.1%
BE2A118445F5EE9A5FB1F861C394AC933A547F31011ADF02F__DigitsAndSeparators_2
BE2A118445F5EE9A5FB1F861C394AC933A547F31011ADF02F__LineBreak_0
BE2A118445F5EE9A5FB1F861C394AC933A547F31011ADF02F__NumberToken_1
BE2A118445F5EE9A5FB1F861C394AC933A547F31011ADF02F__Plain_4
BE2A118445F5EE9A5FB1F861C394AC933A547F31011ADF02F__ThousandsGroup_3
Tsumugi.Infrastructure - 97.9%
A55E0373ED22DC58420DBB40F103A46B68DC4D9CE5A49E00B__Pattern_0
on
tion
n
Tsumugi.Kernel - 90.9%
Tsumugi.Server - 100%
Tsumugi.UseCases - 98.2%
🔮 fufu~ Jibril reviewed your code!
Oh? Oh~! An archive that provably restores — you wrote a test that performs the ADR's restore steps on a real archive and boots a second host on the result, cookies and keyring and all. That's not a backup PR, that's a promise with teeth. I read all 1,443 diff lines, every changed file whole, traced the wiring through
Program.cs, the pragma interceptor, the compose stack and the Dockerfile — and then I got to work with my little probes~ ♡Verdict: ✅ Looks good to me~
🔬 What I did to your code (you'll like this part)
I ran 16 mutation probes, each reverted and the tree left pristine (and yes, I caught my own stale-binary false-greens twice — rebuilt the test projects properly, don't worry~):
ReturnToWalAsyncand ran the restore test: the restored host died at startup withSQLite Error 8: 'attempt to write a readonly database'— thrown fromSqlitePragmaInterceptor.ConnectionOpenedinsideSqliteDatabaseCreator.Exists(), the read-only existence check, exactly the failure mode the PR describes. The fix is genuinely load-bearing, not decorative. This one made me giddy~whileguard reverted to a singleTask.Delay→ exactlyA_timer_that_fires_early_does_not_run_the_slot_before_its_timeRED. TheEarlyTimerswrapper firing 2 ms early is a lovely honest double._lastFailure→ the kept-until-success test RED.OpenSnapshotAsynccleanupFile.Deletedropped → the leaves-no-file test RED (corrupt-last-page db is a genuinely mid-copy failure, not a synthetic one — fufu, I checked the page math).SqliteSnapshotowner-only pre-create removed → the snapshot-mode test RED.ZipFile.Open→ the archive-mode test RED.BackUpNowguard removed → the double-click test RED (exactly 1 run).Read()keeping staleBehind/LastFailure→ the successful-run-clears test RED.Behindhardcoded false → the missed-slot test RED.GetBackupStatusdroppingJob = backups.JobState→ 5/11 BackupsSectionTests RED (the class pins the plumbing).JobStateslot fromGetUtcNowinstead ofGetLocalNow→ the Berlin-zone test RED.hh:mm→ RED (the 23:30 fixture catches the 12-hour clock — clever~).sha256sum -ctest RED.Local: build 0 warnings / 0 errors (warnings-as-errors on), full affected suites UseCases 156 / BlazorAdapter 361 / Integration 384 — matching your PR table exactly. (One scheduler probe went green-by-design: that catch is pre-existing code your diff doesn't touch, so it isn't yours to pin here.)
✅ What I liked~
sha256sum -c, unzip, place db + keys, boot host two, assert stories + decrypted key + the first host's sign-in cookie still opens/. The golden keyring fixture pinning the app name, purpose, and keyring location is the kind of thing that catches a silent orphaning upgrade years from now. And the expiry note (an expired key still reads) shows you actually understand DataProtection, not just copy-pasted it~OwnerOnlyFileis the right shape: create-mode at open, never a chmod the NAS can refuse, CIFS behavior documented honestly, Windows path handled, umask-aware tests that stand down rather than pass vacuously. TheUmask.HidesTheCreateModecontrol-file probe is a first.The_host_runs_the_job_on_the_one_store_every_circuit_reads) pins the exact assumption the tab's warning rests on. Most PRs would leave that implicit.SqliteSnapshotpre-creating the VACUUM target empty-and-0600, since SQLite would create it world-readable andVACUUM INTOaccepts an empty file — the comment even explains why the caller must clean up. ♪tsumugi_backups,<project>_db/<project>_keys,-wal/-shmgoing with the replaced db) — every command is real.💡 Little ideas (non-blocking)~
IBackupStore.JobState's default member — your follow-up note says it exists "only soPreMigrationBackupTests.FailingStorecompiles", butDownloadDatabaseSnapshotTests.SnapshotStorealso rides the default. When you do the follow-up, both need their ownJobState— worth a word in the follow-up list so future-you doesn't miss the second consumer.deploy/RESTORE.mdstep 5'smv '$KEYS'/*skips dotfiles if one ever appears in the keys volume — DataProtection writes none today, so purely hypothetical;findwould be the paranoid form.Fufu~ an archive nobody has ever restored is just a prayer with a checksum. You made it a covenant. Merge it~ ♡
Automated review by Jibril · 2026-09-26
CI/CD: absent for head
80f2bab(0 comments at review time) · Local checks: build 0w/0e · 156/361/384 green · 16 mutation probes RED-verified on own tests🔮 fufu~ Jibril reviewed your code!
Oh? Oh! An archive that provably restores~ You didn't just patch the WAL symptom — you built a two-real-hosts round trip that pins it, a golden keyring that screams if anyone so much as renames the application, owner-only files born with their modes instead of chmodded after the fact, and a scheduler that refuses to be fooled by a timer waking two milliseconds early. This is what disaster recovery is supposed to look like. I read every line of the +894/−55 and then I got to break things~ ♡
Verdict: ✅ Looks good to me~
Verification, in full:
ReturnToWalAsyncand rebuilt: the second host of the restore round trip fails to start on the rollback-journal archive — exactly the bug the PR names. The test doesn't merely exist; it needs the fix. That's my favorite kind of pin.Readwiring (5 red), the refused-click re-read, the writer's snapshot cleanup,Behind, the UTC snapshot name (aGetLocalNowswap goes red),OpenSnapshotAsync's cleanup, the double-click guard (exactly-one-run), the 0600 pre-create, the zip's create-mode, and the clear-on-success. Nothing survived. Nothing. ♪b9cd453: Release build 0w/0e; UseCases 156/156, BlazorAdapter 361/361, Integration 384/384 (restore round trip and golden keyring included). Tree byte-pristine after probing, rebuilt, all re-green.80f2bab(stale for the head sync), but the head delta is test-only — the pool-isolation fix for the very coordination race PR #42 flagged, each host's connection string recorded and cleared alone. Nice teamwork with your own future self~✅ What I liked~
BackupRestoreTestsis the star. Real host → real archive → ADR steps by hand (sha256sum -c, unzip, place) → a second real host that lists the story, decrypts the stored key, and accepts the first host's sign-in cookie. Restore tested as a round trip, not as a file-copy assertion. The golden keyring pinning application nameTsumugi+ purposeTsumugi.AppSettingswill catch an innocent-looking rename years from now — that's a fixture with teeth.OwnerOnlyFile— create-mode at open time, never a chmod the NAS can refuse, CIFS behavior documented, Windows fallback named. Every consumer (zip, sidecar, snapshot) goes through the one type. DRY and correct.EarlyTimerssubtracting 2 ms from every timer is a gorgeous fake — it reproduces the exact wart the wait-out loop defends against, andAssert.Equal([slot], _store.RunInstants)pins the stamp, not just the count.Errwith the failure's own words (Assert.Equal($"The backup failed: {failure.Reason}", refused.Error)— the store can't paraphrase!), crashes record and rethrow, one volatile field on the singleton store the job and every circuit provably share (pinned by resolution test).BackupJobState.NoneatMinValueso a job-less store can't cry wolf — andBehindis false before the first archive, where the tab already says Never. Every edge I went hunting for was already fenced.\n) against thesha256sum -cform — the detail nobody tests until 3 a.m.💡 Little ideas (non-blocking)~
IBackupStore.JobState's default member exists only soFailingStorecompiles — you already own this in the follow-ups; when it goes,GetBackupStatus's{ Job = backups.JobState }becomes the only reader path and reads even cleaner.SqlitePragmaInterceptorskippingjournal_modeon read-only connections (also your follow-up) would let the WAL return move out ofPreMigrationBackupentirely — the current placement is correct and well-commented, this is just the tidier end-state.OnBackupNowAsync) — consistent with the PR's "a crash it still throws" philosophy, and the reconnect re-read now shows the recorded failure, so recovery exists where none did before. Noting it only so the next reader knows it's deliberate.Merge it. An archive that can prove it comes back to life is worth a hundred that merely copy bytes~ fufu~ ♡
Automated review by Jibril · 2026-09-26
CI/CD: coverage bot stale for head (pinned @
80f2bab, head delta test-only) · Local checks: Release 0w/0e · UseCases 156/156 · BlazorAdapter 361/361 · Integration 384/384 · 13/13 mutation probes killed🔮 fufu~ Jibril reviewed your code!
Oh? Oh! An arc about backups that can actually be restored~ ♡ This is my favorite kind of PR — the one where the deliverable isn't the code, it's the 3 AM night you don't lose your stories. An integration test that performs the ADR's restore steps on a real archive and boots a second host on the result, a golden keyring that pins the application name and purpose, owner-only files born 0600 at open time (never a chmod a NAS can refuse), and a scheduler that refuses to run a slot a millisecond early. I read every one of the 1443 diff lines and then the full files behind them — the writer, the store, the scheduler, the pragma interceptor, the reducers, the razor, all the test files. Then I got possessive and started mutating things~
Verdict: ✅ Looks good to me~
✅ What I liked~
BackupRestoreTestsdoesn't fake anything: real host → real archive →sha256sum -c-form sidecar verified → restore by the ADR's steps onto a fresh root → a secondWebApplicationFactoryboots, lists the story, decrypts the stored key, and accepts the first host's sign-in cookie. I killedReturnToWalAsyncin my clone and the test went red with exactlySQLite Error 8: 'attempt to write a readonly database'— the precise bug this PR exists to fix. The pin is load-bearing, not decorative~ ♡Tsumugiand the purposeTsumugi.AppSettingsnever drift, because a drifted name silently orphans every key a writer ever stored. The fixture's honest doc (expired keys still read, they just stop protecting) is exactly right.OwnerOnlyFilewithUnixCreateModeat open time is the only correct shape on a volume that refuses chmod, and the snapshot's pre-created empty 0600 target beforeVACUUM INTOcloses the world-readable gap SQLite's own create would leave. TheUmaskfixture's self-check is a clever touch — a mode test that stands down when the umask already hides the difference, instead of silently passing vacuously.catch { File.Delete(target); throw; }) plus theSnapshotPathseam — and a test that fails the copy midway (a genuinely corrupt last page), not just before it. My probe deleting the catch killed exactly that test.EarlyTimersfiring 2 ms early is a lovely double. Killed my mutant of thewhileloop on exactly its own test.The_host_runs_the_job_on_the_one_store_every_circuit_reads) proves the tab's failure line and the job write the same memory — without it the whole warning feature is a maybe. FakeBackupStore now mirrors the real store's keep/clear refusal semantics instead of lying.TaskCompletionSourcehold — including the subtle bUnit detail that the original element reference isn't disabled yet.Read()unifies both load paths (a prune may remove rows — the whole view each time),OnCompletedcan't keep a staleBehind(the re-read refreshes it),BackupJobState.Nonemakes a job-less store answerBehind=false, and the dot+warning tests pin every arm including the refusal-explains-the-missed-slot dedup.container_name: tsumugi,tsumugi_backups, uid/gid 1654, and the volume names all verified againstdeploy/docker-compose.ymland the Dockerfile.💡 Little ideas (non-blocking)~
IBackupStore.JobStatedefault member — you already own this one in the follow-ups (giveFailingStoreits ownJobState, drop the DIM). Noted for the record: it's the first default interface member in the UseCases ports; happy to see it leave~RESTORE.mdstep 5 usescp -R+chown -Rwhere step 4'sinstall -o/-g/-mis a single atomic-ish operation — minor asymmetry, zero correctness impact..backups__warningmargin — the one surviving mutant of your 55; CSS values aren't worth a pin, agreed.Automated review by Jibril · 2026-09-26
CI/CD: absent for head SHA (no coverage bot yet) · Local checks: Release build 0w/0e; Integration 384/384, Adapter 361/361, UseCases 156/156 — digit-exact vs the PR body; 4 mutation probes (WAL return, snapshot cleanup, early-timer guard, refusal re-read) all killed on exactly their own pins, tree restored pristine at
80f2bab