feat: wave-3 audio surfaces — stems, sfx, speech enhance, voice clone, translate (ADR-0024)
CI / Tidy (pull_request) Successful in 9m27s
CI / Build & Test (pull_request) Successful in 10m37s
Gadfly review (reusable) / review (pull_request) Successful in 41m14s
Adversarial Review (Gadfly) / review (pull_request) Successful in 41m14s

- audio.StemSeparator/StemSeparationProvider: Demucs zip transport via
  POST /upstream/<id>/v1/stems (Mode two -> two_stems=vocals; model +
  format fields); bounded zip unpack, entry name -> stem, ext -> MIME.
- SFXModel reuses musicgen against the sync /upstream/<id>/v1/sfx route
  (JSON prompt/seconds/steps/cfg_scale/seed -> WAV); musicgen.Request
  gains CFGScale.
- audio.SpeechEnhancer/SpeechEnhancementProvider:
  POST /upstream/<id>/v1/enhance -> WAV (result reuses SpeechResult).
- SpeechRequest.ReferenceAudio/ReferenceMIME (+WithReferenceAudio):
  llamaswap switches to the chatterbox clone route
  POST /upstream/<id>/v1/audio/speech/upload (input + voice_file),
  wav MIME fallback.
- TranscriptionRequest.Translate (+WithTranslate): translate=true form
  field, language=auto forced when no explicit hint (whisper.cpp default
  en would skip translation).
- httptest contract tests (zip unpack, clone-route switch, translate +
  auto-language injection); ADR-0024 (index row deferred — MJ-A backfills
  the ADR index table and parallel edits would conflict).

Co-Authored-By: Claude Fable 5 <[email protected]>
This commit is contained in:
2026-07-16 17:01:11 -04:00
co-authored by Claude Fable 5
parent cd43009672
commit b3a172a053
11 changed files with 1182 additions and 2 deletions
+28
View File
@@ -30,6 +30,17 @@ type SpeechRequest struct {
// Speed is the playback-rate multiplier; 0 = backend default (1.0).
Speed float64
// ReferenceAudio is a short voice sample for zero-shot voice cloning
// (chatterbox style): when set, the model speaks Input in the sampled
// voice instead of a named Voice. nil = normal synthesis. Backends
// without cloning support must reject a reference-carrying request
// rather than silently ignoring it.
ReferenceAudio []byte
// ReferenceMIME is the reference audio's MIME type (e.g. "audio/wav");
// "" = let the backend sniff it.
ReferenceMIME string
}
// SpeechResult is the canonical synthesis result: raw audio bytes plus the
@@ -59,6 +70,11 @@ func WithFormat(f string) SpeechOption { return func(r *SpeechRequest) { r.Forma
// WithSpeed sets the playback-rate multiplier.
func WithSpeed(s float64) SpeechOption { return func(r *SpeechRequest) { r.Speed = s } }
// WithReferenceAudio provides a voice sample for zero-shot voice cloning.
func WithReferenceAudio(data []byte, mime string) SpeechOption {
return func(r *SpeechRequest) { r.ReferenceAudio, r.ReferenceMIME = data, mime }
}
// Apply returns a copy of the request with all options applied. Providers
// call this once at the top of Speak.
func (r SpeechRequest) Apply(opts ...SpeechOption) SpeechRequest {
@@ -121,6 +137,13 @@ type TranscriptionRequest struct {
// Prompt is optional context or vocabulary to bias decoding; "" = none.
Prompt string
// Translate requests an English translation of the speech instead of a
// same-language transcript (whisper's translate task). When set and no
// Language is given, providers must force source-language auto-detection
// — a backend whose default language is "en" would otherwise skip
// translation entirely.
Translate bool
}
// TranscriptionResult is the canonical transcription result.
@@ -145,6 +168,11 @@ func WithPrompt(p string) TranscriptionOption {
return func(r *TranscriptionRequest) { r.Prompt = p }
}
// WithTranslate requests an English translation instead of a transcript.
func WithTranslate() TranscriptionOption {
return func(r *TranscriptionRequest) { r.Translate = true }
}
// Apply returns a copy of the request with all options applied.
func (r TranscriptionRequest) Apply(opts ...TranscriptionOption) TranscriptionRequest {
for _, opt := range opts {