Merge pull request 'feat: wave-3 audio surfaces — stems, SFX, speech enhance, voice clone, translate (ADR-0024)' (#18) from feat/wave3-audio-surfaces into main
CI / Tidy (push) Successful in 9m22s
CI / Build & Test (push) Successful in 9m44s

This commit was merged in pull request #18.
This commit is contained in:
2026-07-16 23:24:42 +00:00
12 changed files with 1275 additions and 5 deletions
+65
View File
@@ -0,0 +1,65 @@
# ADR-0024: Wave-3 audio surfaces (stems, SFX, speech enhance, voice clone, translate)
Status: Accepted (2026-07-16)
## Context
The llama-swap host is gaining wave-3 audio capabilities (spec: mort
docs/specs/2026-07-16-llamaswap-wave3.md): Demucs stem separation and
DeepFilterNet speech enhancement (both on the CPU `audioutils` shim), Stable
Audio Open sound effects (`sfxgen`), plus two upgrades to existing models —
chatterbox's stateless voice-clone route and whisper.cpp's per-request
translate flag (both verified against the live images 2026-07-16).
## Decision
1. **`audio.StemSeparator` / `StemSeparationProvider`** —
`StemSeparationRequest{Audio, MIME, Filename, Mode, Model, Format}`
`StemSeparationResult{Stems []Stem{Name, Audio, MIME}}`.
- `Mode` is the caller-facing split: `"two"` (vocals + accompaniment,
sent as Demucs' `two_stems=vocals`) or `"four"`; `""` = backend
default. `Model` selects the Demucs variant (`htdemucs`/`htdemucs_ft`),
mirroring `BackgroundRemovalRequest.Net`.
- **The wire format is a ZIP** (`POST /upstream/<id>/v1/stems`): four WAV
stems would blow any JSON-of-base64 budget. Entry name → stem name,
extension → MIME; entries may sit under a per-model directory. Unpacking
is bounded per entry (zip-bomb guard) and a non-zip 2xx body fails loud.
2. **`SFXModel` reuses `musicgen`** — a sound effect is a short audio clip
from a text prompt; only the provider method differs. The sfxgen route
(`POST /upstream/<id>/v1/sfx`, JSON `{prompt, seconds, steps?, cfg_scale?,
seed?}`) is SYNCHRONOUS (WAV body), unlike ACE-Step's job queue.
`musicgen.Request` gains `CFGScale *float64` for it; lyrics and non-wav
formats are rejected (the model can't honor them). The ~11s model ceiling
is the backend's to enforce.
3. **`audio.SpeechEnhancer` / `SpeechEnhancementProvider`** —
`EnhancementRequest{Audio, MIME, Filename}`
`POST /upstream/<id>/v1/enhance` → WAV. The result reuses `SpeechResult`;
audio-in/audio-out needs no new result type.
4. **Voice cloning is a `SpeechRequest` field, not a new surface**
`ReferenceAudio []byte` + `ReferenceMIME`. When set, the llamaswap
speech model switches from JSON `/v1/audio/speech` to multipart
`POST /upstream/<id>/v1/audio/speech/upload` (fields `input` +
`voice_file`; verified live: stateless per-request cloning, no voice
library). Voice/format/speed still ride when set; the clone route's MIME
fallback is wav (not the JSON route's mp3). Backends without cloning must
reject a reference-carrying request rather than silently ignore it.
5. **Translation is a `TranscriptionRequest` bool**`Translate` maps to
whisper.cpp's `translate=true` form field (server.cpp:534). Because that
server's language DEFAULT is `en` (which silently skips translation), the
provider forces `language=auto` when translating without an explicit
language hint; an explicit hint wins.
6. **Binary success bodies are validated before wrapping** (ADR-0020 rule):
sfx/enhance require positive evidence of audio-ness (declared audio/*
Content-Type or sniffed RIFF/WAVE), stems require a parseable zip with
at least one stem.
## Consequences
- `audio` grows from three surfaces to five; the SFX surface adds zero new
types (musicgen reuse) — one format→MIME table and one multipart builder
keep serving every audio endpoint.
- The clone-route switch means one `SpeechModel` can answer over two wire
shapes; tests pin both routes so a regression can't silently drop cloning.
- `two_stems` is hardwired to vocals: "isolate X vs the rest" for other
sources is a backend capability not exposed in v1 (add a field when a
caller needs it, not before).