Merge pull request 'feat: wave-3 audio surfaces — stems, SFX, speech enhance, voice clone, translate (ADR-0024)' (#18) from feat/wave3-audio-surfaces into main
This commit was merged in pull request #18.
This commit is contained in:
@@ -0,0 +1,65 @@
|
||||
# ADR-0024: Wave-3 audio surfaces (stems, SFX, speech enhance, voice clone, translate)
|
||||
|
||||
Status: Accepted (2026-07-16)
|
||||
|
||||
## Context
|
||||
|
||||
The llama-swap host is gaining wave-3 audio capabilities (spec: mort
|
||||
docs/specs/2026-07-16-llamaswap-wave3.md): Demucs stem separation and
|
||||
DeepFilterNet speech enhancement (both on the CPU `audioutils` shim), Stable
|
||||
Audio Open sound effects (`sfxgen`), plus two upgrades to existing models —
|
||||
chatterbox's stateless voice-clone route and whisper.cpp's per-request
|
||||
translate flag (both verified against the live images 2026-07-16).
|
||||
|
||||
## Decision
|
||||
|
||||
1. **`audio.StemSeparator` / `StemSeparationProvider`** —
|
||||
`StemSeparationRequest{Audio, MIME, Filename, Mode, Model, Format}` →
|
||||
`StemSeparationResult{Stems []Stem{Name, Audio, MIME}}`.
|
||||
- `Mode` is the caller-facing split: `"two"` (vocals + accompaniment,
|
||||
sent as Demucs' `two_stems=vocals`) or `"four"`; `""` = backend
|
||||
default. `Model` selects the Demucs variant (`htdemucs`/`htdemucs_ft`),
|
||||
mirroring `BackgroundRemovalRequest.Net`.
|
||||
- **The wire format is a ZIP** (`POST /upstream/<id>/v1/stems`): four WAV
|
||||
stems would blow any JSON-of-base64 budget. Entry name → stem name,
|
||||
extension → MIME; entries may sit under a per-model directory. Unpacking
|
||||
is bounded per entry (zip-bomb guard) and a non-zip 2xx body fails loud.
|
||||
2. **`SFXModel` reuses `musicgen`** — a sound effect is a short audio clip
|
||||
from a text prompt; only the provider method differs. The sfxgen route
|
||||
(`POST /upstream/<id>/v1/sfx`, JSON `{prompt, seconds, steps?, cfg_scale?,
|
||||
seed?}`) is SYNCHRONOUS (WAV body), unlike ACE-Step's job queue.
|
||||
`musicgen.Request` gains `CFGScale *float64` for it; lyrics and non-wav
|
||||
formats are rejected (the model can't honor them). The ~11s model ceiling
|
||||
is the backend's to enforce.
|
||||
3. **`audio.SpeechEnhancer` / `SpeechEnhancementProvider`** —
|
||||
`EnhancementRequest{Audio, MIME, Filename}` →
|
||||
`POST /upstream/<id>/v1/enhance` → WAV. The result reuses `SpeechResult`;
|
||||
audio-in/audio-out needs no new result type.
|
||||
4. **Voice cloning is a `SpeechRequest` field, not a new surface** —
|
||||
`ReferenceAudio []byte` + `ReferenceMIME`. When set, the llamaswap
|
||||
speech model switches from JSON `/v1/audio/speech` to multipart
|
||||
`POST /upstream/<id>/v1/audio/speech/upload` (fields `input` +
|
||||
`voice_file`; verified live: stateless per-request cloning, no voice
|
||||
library). Voice/format/speed still ride when set; the clone route's MIME
|
||||
fallback is wav (not the JSON route's mp3). Backends without cloning must
|
||||
reject a reference-carrying request rather than silently ignore it.
|
||||
5. **Translation is a `TranscriptionRequest` bool** — `Translate` maps to
|
||||
whisper.cpp's `translate=true` form field (server.cpp:534). Because that
|
||||
server's language DEFAULT is `en` (which silently skips translation), the
|
||||
provider forces `language=auto` when translating without an explicit
|
||||
language hint; an explicit hint wins.
|
||||
6. **Binary success bodies are validated before wrapping** (ADR-0020 rule):
|
||||
sfx/enhance require positive evidence of audio-ness (declared audio/*
|
||||
Content-Type or sniffed RIFF/WAVE), stems require a parseable zip with
|
||||
at least one stem.
|
||||
|
||||
## Consequences
|
||||
|
||||
- `audio` grows from three surfaces to five; the SFX surface adds zero new
|
||||
types (musicgen reuse) — one format→MIME table and one multipart builder
|
||||
keep serving every audio endpoint.
|
||||
- The clone-route switch means one `SpeechModel` can answer over two wire
|
||||
shapes; tests pin both routes so a regression can't silently drop cloning.
|
||||
- `two_stems` is hardwired to vocals: "isolate X vs the rest" for other
|
||||
sources is a backend capability not exposed in v1 (add a field when a
|
||||
caller needs it, not before).
|
||||
Reference in New Issue
Block a user