majordomo/docs/adr/0024-audio-wave3-surfaces.md

ADR-0024: Wave-3 audio surfaces (stems, SFX, speech enhance, voice clone, translate)

Status: Accepted (2026-07-16)

Context

The llama-swap host is gaining wave-3 audio capabilities (spec: mort docs/specs/2026-07-16-llamaswap-wave3.md): Demucs stem separation and DeepFilterNet speech enhancement (both on the CPU audioutils shim), Stable Audio Open sound effects (sfxgen), plus two upgrades to existing models — chatterbox's stateless voice-clone route and whisper.cpp's per-request translate flag (both verified against the live images 2026-07-16).

Decision

1. audio.StemSeparator / StemSeparationProvider — StemSeparationRequest{Audio, MIME, Filename, Mode, Model, Format} → StemSeparationResult{Stems []Stem{Name, Audio, MIME}}.
   • Mode is the caller-facing split: "two" (vocals + accompaniment, sent as Demucs' two_stems=vocals) or "four"; "" = backend default. Model selects the Demucs variant (htdemucs/htdemucs_ft), mirroring BackgroundRemovalRequest.Net.
   • The wire format is a ZIP (POST /upstream/<id>/v1/stems): four WAV stems would blow any JSON-of-base64 budget. Entry name → stem name, extension → MIME; entries may sit under a per-model directory. Unpacking is bounded per entry (zip-bomb guard) and a non-zip 2xx body fails loud.
2. SFXModel reuses musicgen — a sound effect is a short audio clip from a text prompt; only the provider method differs. The sfxgen route (POST /upstream/<id>/v1/sfx, JSON {prompt, seconds, steps?, cfg_scale?, seed?}) is SYNCHRONOUS (WAV body), unlike ACE-Step's job queue. musicgen.Request gains CFGScale *float64 for it; lyrics and non-wav formats are rejected (the model can't honor them). The ~11s model ceiling is the backend's to enforce.
3. audio.SpeechEnhancer / SpeechEnhancementProvider — EnhancementRequest{Audio, MIME, Filename} → POST /upstream/<id>/v1/enhance → WAV. The result reuses SpeechResult; audio-in/audio-out needs no new result type.
4. Voice cloning is a SpeechRequest field, not a new surface — ReferenceAudio []byte + ReferenceMIME. When set, the llamaswap speech model switches from JSON /v1/audio/speech to multipart POST /upstream/<id>/v1/audio/speech/upload (fields input + voice_file; verified live: stateless per-request cloning, no voice library). Voice/format/speed still ride when set; the clone route's MIME fallback is wav (not the JSON route's mp3). Backends without cloning must reject a reference-carrying request rather than silently ignore it.
5. Translation is a TranscriptionRequest bool — Translate maps to whisper.cpp's translate=true form field (server.cpp:534). Because that server's language DEFAULT is en (which silently skips translation), the provider forces language=auto when translating without an explicit language hint; an explicit hint wins.
6. Binary success bodies are validated before wrapping (ADR-0020 rule): sfx/enhance require positive evidence of audio-ness (declared audio/* Content-Type or sniffed RIFF/WAVE), stems require a parseable zip with at least one stem.

Consequences

• audio grows from three surfaces to five; the SFX surface adds zero new types (musicgen reuse) — one format→MIME table and one multipart builder keep serving every audio endpoint.
• The clone-route switch means one SpeechModel can answer over two wire shapes; tests pin both routes so a regression can't silently drop cloning.
• two_stems is hardwired to vocals: "isolate X vs the rest" for other sources is a backend capability not exposed in v1 (add a field when a caller needs it, not before).