steveandClaude Fable 5
b3a172a053
- audio.StemSeparator/StemSeparationProvider: Demucs zip transport via POST /upstream/<id>/v1/stems (Mode two -> two_stems=vocals; model + format fields); bounded zip unpack, entry name -> stem, ext -> MIME. - SFXModel reuses musicgen against the sync /upstream/<id>/v1/sfx route (JSON prompt/seconds/steps/cfg_scale/seed -> WAV); musicgen.Request gains CFGScale. - audio.SpeechEnhancer/SpeechEnhancementProvider: POST /upstream/<id>/v1/enhance -> WAV (result reuses SpeechResult). - SpeechRequest.ReferenceAudio/ReferenceMIME (+WithReferenceAudio): llamaswap switches to the chatterbox clone route POST /upstream/<id>/v1/audio/speech/upload (input + voice_file), wav MIME fallback. - TranscriptionRequest.Translate (+WithTranslate): translate=true form field, language=auto forced when no explicit hint (whisper.cpp default en would skip translation). - httptest contract tests (zip unpack, clone-route switch, translate + auto-language injection); ADR-0024 (index row deferred — MJ-A backfills the ADR index table and parallel edits would conflict). Co-Authored-By: Claude Fable 5 <[email protected]>
66 lines
3.7 KiB
Markdown
66 lines
3.7 KiB
Markdown
# ADR-0024: Wave-3 audio surfaces (stems, SFX, speech enhance, voice clone, translate)
|
|
|
|
Status: Accepted (2026-07-16)
|
|
|
|
## Context
|
|
|
|
The llama-swap host is gaining wave-3 audio capabilities (spec: mort
|
|
docs/specs/2026-07-16-llamaswap-wave3.md): Demucs stem separation and
|
|
DeepFilterNet speech enhancement (both on the CPU `audioutils` shim), Stable
|
|
Audio Open sound effects (`sfxgen`), plus two upgrades to existing models —
|
|
chatterbox's stateless voice-clone route and whisper.cpp's per-request
|
|
translate flag (both verified against the live images 2026-07-16).
|
|
|
|
## Decision
|
|
|
|
1. **`audio.StemSeparator` / `StemSeparationProvider`** —
|
|
`StemSeparationRequest{Audio, MIME, Filename, Mode, Model, Format}` →
|
|
`StemSeparationResult{Stems []Stem{Name, Audio, MIME}}`.
|
|
- `Mode` is the caller-facing split: `"two"` (vocals + accompaniment,
|
|
sent as Demucs' `two_stems=vocals`) or `"four"`; `""` = backend
|
|
default. `Model` selects the Demucs variant (`htdemucs`/`htdemucs_ft`),
|
|
mirroring `BackgroundRemovalRequest.Net`.
|
|
- **The wire format is a ZIP** (`POST /upstream/<id>/v1/stems`): four WAV
|
|
stems would blow any JSON-of-base64 budget. Entry name → stem name,
|
|
extension → MIME; entries may sit under a per-model directory. Unpacking
|
|
is bounded per entry (zip-bomb guard) and a non-zip 2xx body fails loud.
|
|
2. **`SFXModel` reuses `musicgen`** — a sound effect is a short audio clip
|
|
from a text prompt; only the provider method differs. The sfxgen route
|
|
(`POST /upstream/<id>/v1/sfx`, JSON `{prompt, seconds, steps?, cfg_scale?,
|
|
seed?}`) is SYNCHRONOUS (WAV body), unlike ACE-Step's job queue.
|
|
`musicgen.Request` gains `CFGScale *float64` for it; lyrics and non-wav
|
|
formats are rejected (the model can't honor them). The ~11s model ceiling
|
|
is the backend's to enforce.
|
|
3. **`audio.SpeechEnhancer` / `SpeechEnhancementProvider`** —
|
|
`EnhancementRequest{Audio, MIME, Filename}` →
|
|
`POST /upstream/<id>/v1/enhance` → WAV. The result reuses `SpeechResult`;
|
|
audio-in/audio-out needs no new result type.
|
|
4. **Voice cloning is a `SpeechRequest` field, not a new surface** —
|
|
`ReferenceAudio []byte` + `ReferenceMIME`. When set, the llamaswap
|
|
speech model switches from JSON `/v1/audio/speech` to multipart
|
|
`POST /upstream/<id>/v1/audio/speech/upload` (fields `input` +
|
|
`voice_file`; verified live: stateless per-request cloning, no voice
|
|
library). Voice/format/speed still ride when set; the clone route's MIME
|
|
fallback is wav (not the JSON route's mp3). Backends without cloning must
|
|
reject a reference-carrying request rather than silently ignore it.
|
|
5. **Translation is a `TranscriptionRequest` bool** — `Translate` maps to
|
|
whisper.cpp's `translate=true` form field (server.cpp:534). Because that
|
|
server's language DEFAULT is `en` (which silently skips translation), the
|
|
provider forces `language=auto` when translating without an explicit
|
|
language hint; an explicit hint wins.
|
|
6. **Binary success bodies are validated before wrapping** (ADR-0020 rule):
|
|
sfx/enhance require positive evidence of audio-ness (declared audio/*
|
|
Content-Type or sniffed RIFF/WAVE), stems require a parseable zip with
|
|
at least one stem.
|
|
|
|
## Consequences
|
|
|
|
- `audio` grows from three surfaces to five; the SFX surface adds zero new
|
|
types (musicgen reuse) — one format→MIME table and one multipart builder
|
|
keep serving every audio endpoint.
|
|
- The clone-route switch means one `SpeechModel` can answer over two wire
|
|
shapes; tests pin both routes so a regression can't silently drop cloning.
|
|
- `two_stems` is hardwired to vocals: "isolate X vs the rest" for other
|
|
sources is a backend capability not exposed in v1 (add a field when a
|
|
caller needs it, not before).
|