fix: /health tells a mounted data volume from a writable one #112
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "fix/health-readiness-probe"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What's in
GET /healthanswered a constant{"status":"healthy"}without touching the filesystem. A data volume that is mounted and readable but not writable — the failure class of both incidents indeploy/DEPLOYMENT.md§8.1 (root-owned volume content after the uid-1654 change; a CIFS volume whose modes come from its mount options, wherechownexits 0 and changes nothing) — was invisible to it and to/api/stats, which only walks directory trees. The DockerHEALTHCHECKpolled that route every 30 s and kept calling a container healthy when it could not store an image, write a thumbnail, take a backup or checkpoint SQLite./healthis now a readiness probe over the four ADR-0010 data directories (db, derived fromDOUJIN_MANAGER_DB_PATH;images;thumbnails;backups):200 {"status":"healthy","dataDirectories":[{"name","path","writable"}...]}only when all four accepted a write.503 {"status":"unhealthy",...}otherwise, with"writable": falseon the directory that refused — so the operator is told which volume to repair, and §8.1 has the repair for each cause.How it is built, and why:
Program.cs. An endpoint that reaches the filesystem is what "endpoints call use cases only" is about.ICheckReadinessUseCase+ result records in ApplicationCore,CheckReadinessUseCasein Infrastructure,RestAdapter/Endpoints/HealthEndpoints.csfor the route;Program.csonly callsapp.MapHealthEndpoints().EndpointResponseTypeTestsnow sweeps/healthtoo, so ADR 0021 genuinely applies —.Produces<HealthDto>(200)and.Produces<HealthDto>(503). Both carry the same DTO: the 503 is the server reporting its own state, and the health check and operators both need it.MapHealthEndpointswas added to that test'sKnownEndpointFilesso hiding it again fails loudly.rwxr-xr-x root:root. Each check creates a.dm-health-<guid>file that deletes itself when the handle closes.DOUJIN_MANAGER_*variable was invented (hence no DEPLOYMENT.md §2 row and no compose change).deploy/smoke.shasserts.status == "healthy"and §4.3 documents it; this is an operational probe, not a navigable resource. The exemption is written into the ADR rather than left implicit.Server/ReadinessRegistration.cs(theImageInfrastructureRegistrationpattern) rather thanUseCaseRegistrationHelper: registering it there reds every test host, since they callRegisterAllwithout the deployment's data paths and Development validates the container on build. The endpoint binds the use case with[FromServices]for the same reason — the ADR-0021 route table is built without host registrations, and an unannotated complex parameter is inferred as a request body, which throws for a GET.dbDiris nowPath.GetDirectoryName(Path.GetFullPath(dbPath)), so a relativeDOUJIN_MANAGER_DB_PATHstill names a real directory. That removes a branch rather than adding one.Docs in the same commit: ADR 0010 amendment (liveness vs readiness, the contract, the probe file, why live, why not enveloped); ADR 0021 amendment (enforcement moved to cover
/health; only/versionis still inline); DEPLOYMENT.md §4.3/§7/§8.1;deploy/smoke.sh;docs/DEVELOPMENT.mdstep 5; README API overview.Verification
HealthEndpointIntegrationTestsboots the realServer/Program.cson a TestServer with the four paths under a temp root and an unauthenticated client (so every assertion also pins that/healthstayed anonymous). It asserts 200 first, then drops one directory's write permission after the host is up and asserts 503. Deliberate: startup migrations needdbwritable, so a read-only-from-boot host never answers at all — and a fix that snapshotted writability at startup would pass a boot-time test while failing the real incident. The class disables parallelisation for its collection becauseProgram.csreads its paths from process-global env vars thatBackupEndpointsIntegrationTestsalso mutates.status,dataDirectories,name,path,writable), so they pin the wire shapesmoke.shand DEPLOYMENT.md depend on, not just the DTO round-trip.EndpointResponseTypeTestsgreen with/healthin the sweep.FileOptions.Noneinstead ofDeleteOnClose→ the leftover-file test red; removing.Produces(200)→ the ADR 0021 test red naming the endpoint.deploy/smoke.sh --image(which now asserts the four directories came back writable) was not executed, so the new assertion has not run against a real Docker volume, and theHEALTHCHECK'scurl --failreaction to a 503 is reasoned fromdeploy/Dockerfilerather than observed. Actual CIFS behaviour (whetherDeleteOnClose's unlink behaves there as it does on tmpfs) is likewise unverified — the test uses POSIX modes on tmpfs, which reproduces the symptom of both incidents but not the CIFS driver. The Flutter app was not touched: its connection test will surface the new 503 asServer returned HTTP 503.Worth deciding before merge
/healthis anonymous and now performs a write into all four data volumes on every request — one empty file created and unlinked per directory. Normal traffic is one poll per 30 s plus the app's connection test, but anyone who can reach the port can drive that. The answer if it ever matters is a short cache TTL, which would need an ADR 0010 amendment and would break the current test by design — both stated out loud in the amendment. Flagging it because it is a real change to the exposed surface of a NAS-hosted server, not a detail to inherit silently.The container's verdict now depends on the volumes, so a genuinely broken mount flips it to
(unhealthy). Underrestart: unless-stoppedDocker reports rather than restarts (compose declares no autoheal). That is the intended, louder behaviour, but it is a behaviour change for anyone watching the old always-healthy signal.Follow-ups
docker logsstays silent. A warning fromCheckReadinessUseCaseneeds a capturing-logger test to pin it (the fixture boots the realProgram.cs, which clears providers), so it was left out rather than added unpinned.BackupServicereadsDOUJIN_MANAGER_BACKUP_DIRon every call while the probe uses the pathProgram.csresolved at startup. They cannot disagree in a container, but one owner would be honest.imagesvolume" instead ofServer returned HTTP 503.Deferred items are collected in #107 (
docs/pile-a-backlog), which should merge first — the seven Pile A branches each wrote todocs/BACKLOG.mdand would otherwise all conflict on it.Verified independently on this branch with the full
scripts/verify.sh(backend build and tests, app codegen, analyze and tests): PASS.🤖 Generated with Claude Code
GET /health answered a constant {"status":"healthy"} without touching the filesystem, so a data volume that was mounted and readable but not writable — the failure class of both incidents in deploy/DEPLOYMENT.md §8.1 (root-owned volume content after the uid-1654 change; a CIFS volume whose modes come from its mount options, where chown exits 0 and changes nothing) — was invisible. Directory walks succeed on such a volume, so /api/stats looked fine too, and the Docker HEALTHCHECK's 30 s poll kept reporting a container that could not store an image, write a thumbnail, take a backup or checkpoint SQLite. /health is now a readiness probe over the four ADR-0010 data directories. It writes a self-deleting .dm-health-<guid> file into each of them on every request and answers 200 {"status":"healthy","dataDirectories":[...]} only when all four accepted the write, 503 {"status":"unhealthy",...} otherwise, with "writable": false on the entry that failed. A real write is the only honest test: mode bits are fabricated on CIFS. The verdict is not cached — a mount whose modes change under a running container never restarts to be noticed, which is the reason the poll is periodic — and at one create plus one unlink per directory there is nothing to tune, so no new DOUJIN_MANAGER_* variable. Shape: the probe lives in a use case (ApplicationCore contract, Infrastructure implementation) behind RestAdapter/Endpoints/HealthEndpoints.cs, rather than inline in Program.cs where /health used to sit — an endpoint that reaches the filesystem is exactly what "endpoints call use cases only" is about. Moving it into an endpoint file also brings it under EndpointResponseTypeTests, so it now declares .Produces<HealthDto>(200) and (503). Its use case is registered by ReadinessRegistration (next to the paths it needs, like ImageInfrastructureRegistration) rather than by UseCaseRegistrationHelper, which the test hosts also call and which would fail to build without the host's data paths. The body keeps its flat status field instead of the ADR-0007 envelope: it is an operational probe read by curl --fail, deploy/smoke.sh and operators, and that field is the documented contract. Docs in the same commit: ADR 0010 gains a liveness-vs-readiness amendment (the four directories, the 503 contract, the probe file, why live and uncached, why not enveloped); ADR 0021 records that /health left Program.cs and is now enforced; DEPLOYMENT.md §4.3/§7/§8.1 document the new body and the "which volume is broken" recipe; smoke.sh asserts all four directories came back writable. docker-compose.yml is unchanged: the HEALTHCHECK lives in the Dockerfile, curl --fail already reacts to the 503, and its timing still fits. Tests: HealthEndpointIntegrationTests boots the real Program.cs host with the four paths under a temp root and drops one directory's write permission after the host is up (a fix that snapshotted writability at startup fails it, and should). It pins the 503 per directory, the healthy body's paths, and that no probe file is left behind. Mutation-probed: probing only the first directory turns all four theory cases red; dropping DeleteOnClose reds the leftover test; dropping .Produces(200) reds the ADR 0021 test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>🔮 fufu~ Jibril reviewed your code!
Oh? Oh! A
/healththat actually tells the truth~ You went straight at the two §8.1 incidents — the uid-1654 root-owned volume and the lying CIFSrwxr-xr-x— and refused every dishonest shortcut. "A real write is the only honest test." Fufu~, you speak my language~ ♡Verdict: ✅ Looks good to me~
No blocking issues. And I didn't take that on faith — I came to poke it myself:
AcceptsWrites(...)withtrueturns all four theory cases red; swappingDeleteOnClose→FileOptions.None(with the real call intact) turnsHealth_LeavesNoProbeFilesBehindred naming the exact leftover.dm-health-*files. These tests have teeth, not decoration~Microsoft.OpenApitransitive), so "no new warnings" holds.The architecture made me genuinely giddy:
ICheckReadinessUseCase : IQuery<ReadinessReport>in ApplicationCore, the endpoint file reaching the filesystem through it,ReadinessRegistrationmirroring theImageInfrastructureRegistrationhost-owns-the-paths pattern, and[FromServices]with the correct reasoning about the ADR-0021 route table inferring a body. I checked the siblings — this is the established shape, not a new dialect. Same forTask.FromResultover sync file I/O (FilesystemImageStorage,BackupServicedo precisely this) and the multi-record DTO/use-case files. ThedbDir = Path.GetDirectoryName(Path.GetFullPath(dbPath))!hardening removes a branch and fixes relative paths — strictly better than the oldIsNullOrEmptydance.And the smoke.sh upgrade —
all(.writable)plus the count of 4 — means the deploy check now proves what container mode exists to prove. The docs triad (ADR 0010 amendment, ADR 0021 amendment, DEPLOYMENT §4.3/§7/§8.1) agrees with the code on every literal: names, status strings, status codes. Nothing drifts~💡 Little ideas (non-blocking)~
RestAdapter/Endpoints/HealthEndpoints.cs:41— there's a/// <summary>Whether the server can currently store data.</summary>sitting above theapp.MapGet(...)statement. A doc comment not attached to any declaration documents nothing (harmless withoutGenerateDocumentationFile, so no CS1587 — I checked). It reads like it labels the route; it labels only the void~ Move it into the method's remarks or delete it.dotnetlabel executes as root, the four theory cases will go red with the harness message (your own failure text says how to fix it: non-root). Watch the first CI run on this head — if it's red for that message, it's the runner, not the code.Path.Combine), the probe is one empty create+unlink per directory, and the exposure is bounded by reachability of an anonymous route on a LAN deployment. The ADR amendment documenting the no-cache decision (and what a future cache would owe) is the right paper trail. The follow-ups you listed (503 logging, single owner for the backup dir, app-side connection-test message) are all correctly scoped out.✅ What I liked~
jqaway — readiness that tells the operator which volume, not just "sad face".EndpointResponseTypeTestssweeping/healthnow, with the count-pins updated so it can't silently fall out again. Enforcement that grows stricter in the same PR that adds the surface — yes~Health_LeavesNoProbeFilesBehindpolling twice to catch fixed-name collisions, andDisposeAsyncrestoring modes before deleting the temp root. The fixture cleans up after itself even when it fails.Ready to merge once CI confirms~ fufu~ ♪
Automated review by Jibril · 2026-09-08
CI/CD: absent for head
0284d5a(PR just opened, no bot comment yet) · Local checks: build 0 errors (1 pre-existing NU1903 on main); 535/535 across all four backend suites as non-root; mutation probes directional (2/2); honesty guard verified both arms