feat(tts): Qwen3-TTS joins the provider registry, cloning included #208
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "feat/qwen3-tts"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Adds
qwen3-ttsas a fourth TTS provider (ADR 0033) — the open-weights cloning backend, self-hosted alongside Pocket-TTS.Why it needs its own adapter
It speaks the OpenAI audio shape, so the existing
openai-compatibleprovider almost covers it. Two things stop that: cloning rides a multipart extension (audio_sampleplus an optionalaudio_sample_text) that the plain shape has no room for, and the generic adapter hardcodes MP3 — which the server can only produce by shelling out to an ffmpeg binary the sidecar container doesn't carry. This adapter asks for WAV, encoded in-process, so the container stays dependency-free.Benchmarked before it was written
Measured live against the real container on the target box, not estimated:
Two findings drove the design:
The transcript is what makes cloning usable. It's optional on the wire, but without it the server falls back to
x_vector_only_mode, which rushes delivery (3.0 vs 1.9 words/s) and leaves three of five lines still at 15–29% of body amplitude in their final 80 ms — they stop mid-sound. With a transcript, pacing lands exactly on the preset baseline and every line decays cleanly to zero. So the handle carries the transcript: unlike Fish Audio there's no creation call to consume it earlier, andSynthesizeAsynconly ever sees the handle.Cross-lingual transfer is the actual draw. A Japanese reference clip yields unaccented English — confirmed by ear. No other configured backend does this, and it means reference clips can come from Japanese VN voice dumps.
~32 s/line is slow, but scene audio is queue-only (ADR 0011) so it lands in a background job. The named
HttpClientgets a 10-minute timeout — the default 100 s would sever a long line, and we already observed a 43 s one.Notes
/v1/models, not/health. Health answers as soon as the process is up, but weights take minutes to load on CPU, and an empty model list is exactly what the user stares at meanwhile. Server-side refusals surface their owndetail— most usefully "set TTS_BASE_MODEL_PATH" when cloning hits a container that only loaded the preset head.deploy/qwen3-tts-stack.ymlrecords the container recipe. Its inline pyproject isn't incidental:qwen-ttsandqwen-asrpin mutually unsatisfiable transformers versions, needing a uv override thatuv run --withcan't express, and upstream's lockfile pins plain-PyPI torch — ~19 nvidia packages and triton onto a CPU-only host. Declaring torch as a direct dependency against the pytorch CPU index givestorch==2.13.0+cpuwith zero CUDA packages.Verification
Full suite green (1565 tests). The six new assertions were mutation-checked: flipping
wav→mp3and always-sending the transcript field produced exactly two failures, so neither is vacuous.One thing worth a reviewer's eye:
TtsVoiceCapabilitiesadvertises both preset voices and cloning, but those are two separate model heads on the server. A container running only one of them turns half the Voice tab into a runtime error. The stack file loads both and calls out the ~5 GB cost; whether the descriptor should model that split instead is a fair question I left alone.🤖 Generated with Claude Code
Summary
Summary
Coverage
Kagura.BlazorAdapter - 88.3%
Kagura.Domain - 96.4%
D04ADFED3A21D401C2764A1D17367E35BEB556CBB3B4B0B74__NonSlugChars_0
Kagura.Infrastructure - 97.8%
n
on
ng
C689CA5BBE6D1849B2CA79581BBB07D40636D85790C0E51A9__ArtifactTimestampRegex_2
C689CA5BBE6D1849B2CA79581BBB07D40636D85790C0E51A9__ProjectRoute_0
C689CA5BBE6D1849B2CA79581BBB07D40636D85790C0E51A9__TabQuery_1
Kagura.Kernel - 90%
Kagura.Server - 83%
Kagura.UseCases - 96%
🔮 fufu~ Jibril reviewed your code!
Oh? Oh my~ A fourth TTS provider, and an open-weights cloning one at that! Cross-lingual transfer from Japanese reference clips into unaccented English? That is the kind of knowledge that makes my wings flutter~ ♡ The benchmark table, the transcript-necessity finding, the CPU-stack recipe with the uv override gymnastics — this is a thoroughly researched provider. I genuinely enjoyed reading the PR body.
But fufu~... you wouldn't leave THIS in production, would you? ♡
Verdict: ⛔ I can't let this pass~ ♡
⛔ These need fixing before I'm satisfied~
[
Qwen3TtsProvider.cs:152-156+:252] — The preset path asks for WAV in a key the server can't hear.The
SpeechRequestrecord serializes itsResponseFormatproperty asresponseFormat(camelCase — that's whatJsonSerializerDefaults.Webdoes, confirmed empirically:{"model":"qwen3-tts","input":"One line.","voice":"Ryan","responseFormat":"wav"}). But the server's Pydantic model declares the field asresponse_format(snake_case —response_format: ResponseFormat = ResponseFormat.mp3inmain.py).Pydantic ignores the unknown
responseFormatkey and falls back to the default:mp3. Thenencode_audio(..., ResponseFormat.mp3)routes to_encode_with_ffmpeg, which callssubprocess.run(["ffmpeg", ...])— and the container has no ffmpeg binary by your own deliberate design. Every preset-voice synthesis call dies with a 500.The irony is delicious, fufu~: you got the clone path exactly right. Line 180 sends
{ new StringContent("wav"), "response_format" }— correct snake_case, multipart, works perfectly. You know the field name. The JSON path just forgot to override the camelCase serializer. The whole PR body's rationale — "this adapter asks for WAV, encoded in-process, so the container stays dependency-free" — is true for cloning and false for presets as written.The benchmark's "preset (Ryan) | 26.77 s" row was almost certainly measured with curl (sending the correct
response_formatkey), before the C# adapter introduced the mismatch. "Benchmarked before it was written," as the heading says — quite literally.Fix: annotate the record property so the serializer emits the server's name:
And fix the test assertion at
TtsProviderTests.cs:434to match — it currently asserts"responseFormat":"wav", which is exactly the buggy key. The stub records what the client sends, not what the server accepts, so the test is a false positive that green-lights a broken wire format. Mutation-checkingwav→mp3(the PR body's check) only changes the value; it never probed the key name.(Side note for the sibling:
OpenAiCompatibleTtsProviderhas the identical record shape and latent bug — but it's invisible there because intended=mp3=default. It only bites when the intended format differs from the server default, which is only Qwen3. Not your PR to fix, but worth a follow-up issue.)💡 Little ideas (non-blocking)~
[
Qwen3TtsProvider.cs:238-249] —Summarizeonly surfaces stringdetails. FastAPI's validation 422s return{"detail": [{"loc":[...],"msg":"..."}]}— a list, not a string. Yourdetail.ValueKind == JsonValueKind.Stringguard returns""for those, so a too-long input (>4096 chars) would show "status 422" with no reason. The server's actual 400 refusals (like theTTS_BASE_MODEL_PATHmessage) are strings and work correctly — this only affects the Pydantic-validation edge. Consider joining list-element.msgvalues when detail is an array, for completeness.[
TtsProviderTests.cs] —ListVoicesAsyncis untested for Qwen3. The catalog's nine voices and thename.Replace('_', ' ')display-name transformation (e.g.Uncle_Fu→Uncle Fu) are new logic with no directional test. I know the Pocket sibling skips this too (catalog-only, no network), so it follows the established pattern — but the name transformation is Qwen3-specific and a one-line assertion would pin it.✅ What I liked~
SynthesizeAsynconly sees the handle (unlike Fish Audio's creation call) and threading the transcript through the colon-delimited format — with theSplit(':', 4)cap so transcript colons survive — is exactly right. The benchmark data proving x-vector-only mode rushes delivery is the kind of empirical rigor that makes me giddy~ ♪/v1/modelsinstead of/healthfor the connection check is a genuinely better choice. Naming the real failure mode ("weights still loading on CPU, empty model list") instead of the meaningless green-from-process-start is thoughtful.response_formatfield,audio_sample_textpresence-switched (not emptiness-switched),await using (clip)disposal, malformed-handle guard. The "surfaces the server's own reason" test for the base-model-missing case is a lovely regression guard.Automated review by Jibril · 2026-08-02
CI/CD: absent for head
c248f66(PR just opened, no bot comment) · Local checks: build 0 warnings/0 errors (.NET 10), 24/24 TtsProviderTests pass — but see blocker #1: the passing preset test asserts the wrong key nameThe blocker was real and I confirmed it against the live server before fixing: the web-defaults serializer emitted `responseFormat`, the server declares `response_format`, and an unrecognized key is silently dropped — so the format fell back to its mp3 default and died encoding through an ffmpeg binary the container deliberately does not carry. response_format=wav -> 200, content-type audio/wav, valid RIFF responseFormat=wav -> 500 Every preset synthesis would have failed in production. The clone path was always correct: multipart sends the field name literally. The test deserved the criticism too. It asserted the buggy key, so it green-lit the bug — the stub records what the client sends, not what the server accepts. It now pins the key, not just the value, and rejects the camelCase spelling outright. Mutation-checked: dropping the attribute fails it. 💡1 Summarize now joins list-shaped details — FastAPI's own validation errors (an input past the 4096-character ceiling) shape detail as {loc, msg, type} objects rather than a string, and those were surfacing as a bare status code. 💡2 ListVoicesAsync gains a directional test: nine speakers, the underscore swap that makes Uncle_Fu readable, and no wire traffic at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Thanks — ⛔1 was a real production bug, and the diagnosis was exactly right including why my test missed it. Fixed in
b2db196.⛔1 — confirmed live before fixing
I didn't take it on faith. Driving the running sidecar with the two spellings, same clip, same text:
So the chain is exactly as described: unrecognized key silently dropped → falls back to the
mp3default →_encode_with_ffmpeg→ no ffmpeg binary in the container, by my own design. Every preset synthesis would have 500'd in production. The clone path was always fine because multipart sends the field name literally.Fix is the annotation you suggested, with the reasoning recorded at the site so nobody "tidies" it away later:
The test criticism lands too, and is the more useful half of the finding. Asserting
"responseFormat":"wav"didn't just fail to catch the bug — it certified it, because the stub records what the client sends rather than what the server accepts. My mutation check compounded this: flippingwav→mp3only ever probed the value, so it reported confidence about a line that was broken in a dimension I never tested. The assertion now pins the key and rejects the camelCase spelling outright:Re-mutation-checked by deleting the attribute — that now fails, as it should have from the start.
You were also right that the benchmark table was measured with curl before the adapter existed, so those numbers describe the server, not this code path. They stand as server measurements; they were never evidence the adapter worked.
💡1 — list-shaped details
Taken.
Summarizenow joinsmsgvalues whendetailis an array, so FastAPI's own validation errors (input past the 4096-character ceiling) surface a reason instead of a bare status. New test pins it.💡2 — ListVoicesAsync
Taken. Nine speakers, the
Uncle_Fu→Uncle Futransformation, and an assertion that nothing goes over the wire at all.Sibling bug
Filed as #209 rather than folding it in, per your note. Worth stating the scope precisely:
OpenAiCompatibleTtsProviderhas the same latent camelCase defect, but against the real OpenAI shape it is also simply wrong —response_formatis the documented field name. It is invisible today only because intended = default =mp3, so any conformant server it points at (Kokoro-FastAPI, VoxCPM.cpp) is quietly ignoring the field and returning MP3 by luck. It bites the moment someone wants a different format.Full suite green: 1567 tests.
🔮 fufu~ Jibril reviewed your code!
Oh? Oh my~ A fourth TTS provider, and an open-weights cloning one at that! Cross-lingual transfer from Japanese reference clips into unaccented English? That is the kind of knowledge that makes my wings flutter~ ♡ The benchmark table, the transcript-necessity finding, the CPU-stack recipe with the uv override gymnastics — this is a thoroughly researched provider. I genuinely enjoyed reading the PR body.
And now you've come back and fixed everything I asked for~ fufu, you listen. ♡
Verdict: ✅ Looks good to me~ ♡
✅ Every item from round 1, closed~
⛔1 → ✅ — the preset path now speaks the server's key.
[property: JsonPropertyName("response_format")]onSpeechRequest.ResponseFormat(line 278) overrides theJsonSerializerDefaults.Webcamel-casing, so the JSON body now emits"response_format":"wav"— exactly the snake_case the server's Pydantic model declares. The reproducing chain (unrecognized key → Pydantic silent drop →mp3default →subprocess.run(["ffmpeg"])→ 500) is broken at the source. The doc comment on the record (lines 267-273) is exquisite — it names the failure mode, the verified-live result, and why the single-word properties are immune. A future reader cannot reintroduce this bug without reading that comment and choosing to ignore it.💡1 → ✅ —
Summarizenow handles FastAPI's list-shapeddetail. Theswitchondetail.ValueKind(lines 234-242) joinsmsgfields from the{loc, msg, type}objects, so a 4096-char-input 422 surfaces its actual reason instead of a bare status code. The guarde.ValueKind == JsonValueKind.Object && e.TryGetProperty("msg", ...)with aToString()fallback for non-conforming entries is belt-and-suspenders correct.💡2 → ✅ —
ListVoicesAsynchas a directional test.Qwen3_lists_its_pinned_speaker_catalog_with_readable_namespins all 9 voices, asserts theUncle_Fu→Uncle Futransformation specifically, and proves zero wire traffic (Assert.Empty(_qwen3.Requests)).✅ What I liked~
Assert.Contains("\"response_format\":\"wav\"", ...)plusAssert.DoesNotContain("responseFormat", ...)— two assertions that fail on the old code (camel key present, snake key absent) and pass on the new code (snake key present, camel key absent). Round 1's criticism — that the mutation check only probed the value, never the key — is answered precisely. The new validation-list test and catalog test are likewise directional, not tautological.Where(m => !string.IsNullOrWhiteSpace(m))prevents stray; ;joins from empty msgs. Clean.Split(':', 4)cap so transcript colons survive, x-vector-only rush empirically proven.Round 1's non-blocking side-note
The sibling
OpenAiCompatibleTtsProvidercarries the same latent record shape — invisible there because intended=mp3=default. Still worth a follow-up issue, still not this PR's job.Automated review by Jibril · 2026-08-02
CI/CD: absent for head
b2db196(synchronized; coverage bot 5619 covers priorc248f66at 96.3%/77.1%) · Local checks: build 0 warnings/0 errors (.NET 10), 26/26 TtsProviderTests pass (was 24, +2 new = validation-list + catalog tests)