feat: wave-3 audio surfaces — stems, SFX, speech enhance, voice clone, translate (ADR-0024) #18

Merged
steve merged 2 commits from feat/wave3-audio-surfaces into main 2026-07-16 23:24:42 +00:00
7 changed files with 128 additions and 38 deletions
Showing only changes of commit 31bccf6e19 - Show all commits
+38 -14
View File
@@ -83,12 +83,12 @@ func (m *speechModel) Speak(ctx context.Context, req audio.SpeechRequest, opts .
// speakWithReference performs zero-shot voice cloning via the upstream // speakWithReference performs zero-shot voice cloning via the upstream
// passthrough clone route. The reference sample rides as the `voice_file` // passthrough clone route. The reference sample rides as the `voice_file`
// part and the text as an `input` field; voice/format/speed still go on the // part and the text as an `input` field; voice/format/speed still go on the
// wire when set (upstreams ignore fields they don't understand). The route's // wire when set (upstreams ignore fields they don't understand). The result
// default container is WAV, so MIME resolution falls back to audio/wav (not // MIME comes from the response header or content sniffing (audioResultMIME,
// the JSON route's mp3) when neither the response header nor the requested // same validation as the sfx/enhance surfaces) — a JSON soft error or an
// format decides. // HTML proxy page must never be wrapped up as audio bytes.
func (m *speechModel) speakWithReference(ctx context.Context, req audio.SpeechRequest) (*audio.SpeechResult, error) { func (m *speechModel) speakWithReference(ctx context.Context, req audio.SpeechRequest) (*audio.SpeechResult, error) {
path, err := upstreamPath(m.id, "/v1/audio/speech/upload") upPath, err := upstreamPath(m.id, "/v1/audio/speech/upload")
Review

Local var named path here vs upPath in stems.go/enhance.go — minor naming drift

maintainability · flagged by 1 model

  • Minor local-naming drift for the upstream path variable. stems.go:57/enhance.go name it upPath (deliberately, to dodge the path package import in stems.go), while speakWithReference (provider/llamaswap/audio.go:91) names it path (audio.go doesn't import the path package, so both compile). Purely cosmetic consistency, not a bug. Trivial.

🪰 Gadfly · advisory

⚪ **Local var named `path` here vs `upPath` in stems.go/enhance.go — minor naming drift** _maintainability · flagged by 1 model_ - **Minor local-naming drift for the upstream path variable.** `stems.go:57`/`enhance.go` name it `upPath` (deliberately, to dodge the `path` package import in `stems.go`), while `speakWithReference` (`provider/llamaswap/audio.go:91`) names it `path` (audio.go doesn't import the `path` package, so both compile). Purely cosmetic consistency, not a bug. Trivial. <sub>🪰 Gadfly · advisory</sub>
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -107,18 +107,17 @@ func (m *speechModel) speakWithReference(ctx context.Context, req audio.SpeechRe
if err != nil { if err != nil {
return nil, err return nil, err
} }
audioBytes, contentType, err := m.p.doRaw(ctx, http.MethodPost, path, m.id, formType, body, maxResponseBytes) audioBytes, contentType, err := m.p.doRaw(ctx, http.MethodPost, upPath, m.id, formType, body, maxAudioResponseBytes)
Review

🔴 speech clone route does not validate response is audio before wrapping in SpeechResult

error-handling, security · flagged by 2 models

  • provider/llamaswap/audio.go:113-119speakWithReference resolves a MIME type and returns the raw response body as a successful SpeechResult without ever checking that the bytes are actually audio. The PR's own ADR-0024 point 6 states "Binary success bodies are validated before wrapping," and the new sfx.go and enhance.go both enforce this with audioResultMIME; the clone route does not. A misconfigured upstream returning an HTML error page or JSON soft-error will be returned to…

🪰 Gadfly · advisory

🔴 **speech clone route does not validate response is audio before wrapping in SpeechResult** _error-handling, security · flagged by 2 models_ - **`provider/llamaswap/audio.go:113-119`** — `speakWithReference` resolves a MIME type and returns the raw response body as a successful `SpeechResult` without ever checking that the bytes are actually audio. The PR's own ADR-0024 point 6 states "Binary success bodies are validated before wrapping," and the new `sfx.go` and `enhance.go` both enforce this with `audioResultMIME`; the clone route does not. A misconfigured upstream returning an HTML error page or JSON soft-error will be returned to… <sub>🪰 Gadfly · advisory</sub>
if err != nil { if err != nil {
return nil, err return nil, err
} }
if len(audioBytes) == 0 { if len(audioBytes) == 0 {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id, Message: "speech clone response contained no audio"} return nil, &llm.APIError{Provider: m.p.name, Model: m.id, Message: "speech clone response contained no audio"}
} }
mimeType := "audio/wav" mimeType := audioResultMIME(contentType, audioBytes)
Review

🟠 speakWithReference lacks audio content sniffing validation present in enhance/sfx endpoints

error-handling, security · flagged by 2 models

  • speakWithReference lacks audio-content validation that the other new binary endpoints enforce (provider/llamaswap/audio.go:117-123). The enhance and sfx endpoints both use audioResultMIME, which falls back to http.DetectContentType sniffing and returns "" (→ error) when the response body is not recognizably audio. speakWithReference has no such sniffing fallback: if the upstream returns a non-audio Content-Type (e.g. text/html from a proxy error page) and req.Format i…

🪰 Gadfly · advisory

🟠 **speakWithReference lacks audio content sniffing validation present in enhance/sfx endpoints** _error-handling, security · flagged by 2 models_ - **`speakWithReference` lacks audio-content validation that the other new binary endpoints enforce** (`provider/llamaswap/audio.go:117-123`). The `enhance` and `sfx` endpoints both use `audioResultMIME`, which falls back to `http.DetectContentType` sniffing and returns `""` (→ error) when the response body is not recognizably audio. `speakWithReference` has no such sniffing fallback: if the upstream returns a non-audio `Content-Type` (e.g. `text/html` from a proxy error page) and `req.Format` i… <sub>🪰 Gadfly · advisory</sub>
if mt := mimeFromContentType(contentType, "audio/"); mt != "" { if mimeType == "" {
mimeType = mt return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
} else if req.Format != "" { Message: fmt.Sprintf("speech clone response is not audio (Content-Type %q): %s", contentType, truncateForError(audioBytes))}
mimeType = speechMIME("", req.Format)
} }
return &audio.SpeechResult{Audio: audioBytes, MIME: mimeType}, nil return &audio.SpeechResult{Audio: audioBytes, MIME: mimeType}, nil
} }
@@ -144,6 +143,28 @@ func speechMIME(contentType, format string) string {
} }
} }
// audioResultMIME resolves an audio body's MIME type: the response
// Content-Type when it is a concrete audio type, else content sniffing
// (RIFF/WAVE, Ogg and friends), else "" — the caller treats undetectable as
// an upstream error, mirroring videoMIME. Sniffed "audio/wave" is normalized
// to the conventional "audio/wav", and Ogg's container type
// "application/ogg" to "audio/ogg". Shared by the clone, sfx, and enhance
// surfaces, which all answer raw audio bodies.
func audioResultMIME(contentType string, data []byte) string {
if mt := mimeFromContentType(contentType, "audio/"); mt != "" {
return mt
}
switch mt := http.DetectContentType(data); {
case mt == "audio/wave":
return "audio/wav"
case mt == "application/ogg":
return "audio/ogg"
case strings.HasPrefix(mt, "audio/"):
return mt
}
return ""
}
// TranscriptionModel implements audio.TranscriptionProvider, binding a // TranscriptionModel implements audio.TranscriptionProvider, binding a
// speech-to-text model served by llama-swap (routed to a whisper.cpp-style // speech-to-text model served by llama-swap (routed to a whisper.cpp-style
// upstream). // upstream).
@@ -241,10 +262,13 @@ func transcriptionFilename(filename, mimeType string) string {
} }
// sanitizeFilename strips characters that would corrupt or inject into the // sanitizeFilename strips characters that would corrupt or inject into the
// multipart Content-Disposition header. Quotes and backslashes are escaped // multipart Content-Disposition header, or smuggle directory structure to
// by mime/multipart itself; CR/LF are not — they must go. // the receiving side. Quotes and backslashes are escaped by mime/multipart
// itself, but a file-writing shim decodes them right back — so CR/LF/NUL
// and both path separators are dropped: an upload-metadata filename must
// never traverse ("../x", "a/b", "C:\x").
func sanitizeFilename(name string) string { func sanitizeFilename(name string) string {
name = strings.NewReplacer("\r", "", "\n", "").Replace(name) name = strings.NewReplacer("\r", "", "\n", "", "\x00", "", "/", "", "\\", "").Replace(name)
return strings.TrimSpace(name) return strings.TrimSpace(name)
} }
+40 -4
View File
@@ -2,12 +2,14 @@ package llamaswap
import ( import (
"context" "context"
"errors"
"io" "io"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"testing" "testing"
"gitea.stevedudenhoeffer.com/steve/majordomo/audio" "gitea.stevedudenhoeffer.com/steve/majordomo/audio"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
) )
func TestSpeakWithReferenceUsesCloneRoute(t *testing.T) { func TestSpeakWithReferenceUsesCloneRoute(t *testing.T) {
1
@@ -54,7 +56,7 @@ func TestSpeakWithReferenceUsesCloneRoute(t *testing.T) {
} }
} }
func TestSpeakWithReferenceDefaultsToWavMIME(t *testing.T) { func TestSpeakWithReferenceSniffsHeaderlessWav(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if err := r.ParseMultipartForm(1 << 20); err != nil { if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Fatalf("parse multipart: %v", err) t.Fatalf("parse multipart: %v", err)
@@ -65,7 +67,7 @@ func TestSpeakWithReferenceDefaultsToWavMIME(t *testing.T) {
} }
} }
w.Header()["Content-Type"] = nil // no declared type at all w.Header()["Content-Type"] = nil // no declared type at all
_, _ = w.Write([]byte("RAWAUDIO")) _, _ = w.Write(wavFixture())
})) }))
defer srv.Close() defer srv.Close()
@@ -77,9 +79,43 @@ func TestSpeakWithReferenceDefaultsToWavMIME(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("Speak: %v", err) t.Fatalf("Speak: %v", err)
} }
// The clone route's default container is wav, not the JSON route's mp3. // Headerless RIFF sniffs audio/wave, normalized to the conventional wav.
if res.MIME != "audio/wav" { if res.MIME != "audio/wav" {
t.Errorf("MIME = %q, want audio/wav fallback", res.MIME) t.Errorf("MIME = %q, want sniffed audio/wav", res.MIME)
}
}
func TestSpeakWithReferenceRejectsNonAudioResponse(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// A FastAPI-style 2xx soft error must not be wrapped up as audio.
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"detail":"reference audio too short"}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
sm, _ := p.SpeechModel("chatterbox")
_, err := sm.Speak(context.Background(),
audio.SpeechRequest{Input: "hi"},
audio.WithReferenceAudio([]byte("REF"), "audio/wav"))
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want APIError for non-audio clone body", err)
}
}
func TestSanitizeFilenameStripsPathAndControlBytes(t *testing.T) {
for in, want := range map[string]string{
"song.mp3": "song.mp3",
"../../etc/passwd": "....etcpasswd",
"a\r\nContent-Type: evil": "aContent-Type: evil",
"..\\..\\boot.ini": "....boot.ini",
"nul\x00byte.wav": "nulbyte.wav",
" / ": "",
} {
if got := sanitizeFilename(in); got != want {
t.Errorf("sanitizeFilename(%q) = %q, want %q", in, got, want)
}
} }
} }
+1 -1
View File
@@ -45,7 +45,7 @@ func (m *speechEnhancerModel) Enhance(ctx context.Context, req audio.Enhancement
if err != nil { if err != nil {
return nil, err return nil, err
} }
raw, respType, err := m.p.doRaw(ctx, http.MethodPost, path, m.id, contentType, body, maxResponseBytes) raw, respType, err := m.p.doRaw(ctx, http.MethodPost, path, m.id, contentType, body, maxAudioResponseBytes)
Review

🟡 Enhance uses 64 MB JSON cap (maxResponseBytes) for WAV responses that can exceed it for recordings >10 min

correctness · flagged by 1 model

🪰 Gadfly · advisory

🟡 **Enhance uses 64 MB JSON cap (maxResponseBytes) for WAV responses that can exceed it for recordings >10 min** _correctness · flagged by 1 model_ <sub>🪰 Gadfly · advisory</sub>
if err != nil { if err != nil {
return nil, err return nil, err
} }
1
+6
View File
@@ -53,6 +53,12 @@ const maxResponseBytes = 64 << 20
// can't allocate without limit. // can't allocate without limit.
const maxVideoResponseBytes = 512 << 20 const maxVideoResponseBytes = 512 << 20
// maxAudioResponseBytes caps bodies that ARE a single encoded audio clip
// (voice clone, speech enhancement, sfx): a long uncompressed WAV
// legitimately passes the 64MB JSON cap. Still bounded so a buggy upstream
// can't allocate without limit.
const maxAudioResponseBytes = 256 << 20
// Provider is a llama-swap client. It satisfies llm.Provider (chat, delegated // Provider is a llama-swap client. It satisfies llm.Provider (chat, delegated
// to provider/openai) and imagegen.Provider (image generation), and exposes // to provider/openai) and imagegen.Provider (image generation), and exposes
// llama-swap's management endpoints as concrete methods. // llama-swap's management endpoints as concrete methods.
+1 -19
View File
@@ -84,7 +84,7 @@ func (m *sfxModel) Generate(ctx context.Context, req musicgen.Request, opts ...m
if err != nil { if err != nil {
return nil, fmt.Errorf("llama-swap: encode sfx request: %w", err) return nil, fmt.Errorf("llama-swap: encode sfx request: %w", err)
} }
raw, contentType, err := m.p.doRaw(ctx, http.MethodPost, path, m.id, "application/json", bytes.NewReader(encoded), maxResponseBytes) raw, contentType, err := m.p.doRaw(ctx, http.MethodPost, path, m.id, "application/json", bytes.NewReader(encoded), maxAudioResponseBytes)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -98,21 +98,3 @@ func (m *sfxModel) Generate(ctx context.Context, req musicgen.Request, opts ...m
} }
return &musicgen.Result{Audio: musicgen.Audio{Data: raw, MIME: mimeType}}, nil return &musicgen.Result{Audio: musicgen.Audio{Data: raw, MIME: mimeType}}, nil
} }
// audioResultMIME resolves an audio body's MIME type: the response
// Content-Type when it is a concrete audio type, else content sniffing
// (RIFF/WAVE and friends), else "" — the caller treats undetectable as an
// upstream error, mirroring videoMIME. Sniffed "audio/wave" is normalized
// to the conventional "audio/wav".
func audioResultMIME(contentType string, data []byte) string {
if mt := mimeFromContentType(contentType, "audio/"); mt != "" {
return mt
}
if mt := http.DetectContentType(data); strings.HasPrefix(mt, "audio/") {
if mt == "audio/wave" {
return "audio/wav"
}
return mt
}
return ""
}
+19
View File
1
@@ -31,6 +31,15 @@ const maxStemsResponseBytes = 512 << 20
// A single WAV stem of even a very long song stays far under this. // A single WAV stem of even a very long song stays far under this.
const maxStemEntryBytes = 256 << 20 const maxStemEntryBytes = 256 << 20
// maxStemEntries caps how many stem entries are unpacked: Demucs emits at
// most six, so anything past a small multiple of that is a hostile or
// broken archive, not a result.
const maxStemEntries = 16
// maxStemsTotalBytes caps the AGGREGATE decompressed size across entries —
// the per-entry bound alone would still let a many-entry bomb multiply up.
const maxStemsTotalBytes = 1 << 30
// StemSeparatorModel implements audio.StemSeparationProvider. The id selects // StemSeparatorModel implements audio.StemSeparationProvider. The id selects
// which upstream llama-swap loads. // which upstream llama-swap loads.
func (p *Provider) StemSeparatorModel(id string, opts ...audio.StemSeparatorModelOption) (audio.StemSeparator, error) { func (p *Provider) StemSeparatorModel(id string, opts ...audio.StemSeparatorModelOption) (audio.StemSeparator, error) {
2
@@ -95,15 +104,25 @@ func (m *stemSeparatorModel) SeparateStems(ctx context.Context, req audio.StemSe
Message: fmt.Sprintf("stems response is not a zip (Content-Type %q): %s", respType, truncateForError(raw))} Message: fmt.Sprintf("stems response is not a zip (Content-Type %q): %s", respType, truncateForError(raw))}
} }
res := &audio.StemSeparationResult{} res := &audio.StemSeparationResult{}
var totalBytes int64
for _, f := range zr.File { for _, f := range zr.File {
if f.FileInfo().IsDir() { if f.FileInfo().IsDir() {
continue continue
} }
if len(res.Stems) >= maxStemEntries {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: fmt.Sprintf("stems zip holds more than %d entries", maxStemEntries)}
}
data, err := readZipEntry(f) data, err := readZipEntry(f)
if err != nil { if err != nil {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id, return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: fmt.Sprintf("stems zip entry %q: %v", f.Name, err)} Message: fmt.Sprintf("stems zip entry %q: %v", f.Name, err)}
} }
totalBytes += int64(len(data))
if totalBytes > maxStemsTotalBytes {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: fmt.Sprintf("stems zip decompresses past %d bytes", int64(maxStemsTotalBytes))}
}
// Entry name → stem name; extension → MIME. Entries may sit in a // Entry name → stem name; extension → MIME. Entries may sit in a
// per-model directory ("htdemucs/vocals.mp3"), so use the base name. // per-model directory ("htdemucs/vocals.mp3"), so use the base name.
base := path.Base(f.Name) base := path.Base(f.Name)
@@ -6,8 +6,10 @@ import (
"context" "context"
"encoding/json" "encoding/json"
"errors" "errors"
"fmt"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"strings"
"testing" "testing"
"gitea.stevedudenhoeffer.com/steve/majordomo/audio" "gitea.stevedudenhoeffer.com/steve/majordomo/audio"
@@ -169,6 +171,27 @@ func TestSeparateStemsRejectsEmptyZip(t *testing.T) {
} }
} }
func TestSeparateStemsRejectsTooManyEntries(t *testing.T) {
Review

wavFixture test helper defined in stems_sfx_enhance_test.go but used by clone_translate_test.go — poor test helper locality

maintainability · flagged by 1 model

  • wavFixture() defined in stems_sfx_enhance_test.go but consumed by clone_translate_test.go (provider/llamaswap/stems_sfx_enhance_test.go:174, provider/llamaswap/clone_translate_test.go:31): Same locality problem in tests — a shared test helper lives in a file whose name suggests it belongs to stems/sfx/enhance tests only. Since both files are in package llamaswap it compiles, but a maintainer adding a new test that needs a WAV fixture won't find it without a grep. Move it to a d…

🪰 Gadfly · advisory

⚪ **wavFixture test helper defined in stems_sfx_enhance_test.go but used by clone_translate_test.go — poor test helper locality** _maintainability · flagged by 1 model_ - **`wavFixture()` defined in `stems_sfx_enhance_test.go` but consumed by `clone_translate_test.go`** (`provider/llamaswap/stems_sfx_enhance_test.go:174`, `provider/llamaswap/clone_translate_test.go:31`): Same locality problem in tests — a shared test helper lives in a file whose name suggests it belongs to stems/sfx/enhance tests only. Since both files are in `package llamaswap` it compiles, but a maintainer adding a new test that needs a WAV fixture won't find it without a grep. Move it to a d… <sub>🪰 Gadfly · advisory</sub>
entries := map[string]string{}
for i := 0; i <= maxStemEntries; i++ {
entries[fmt.Sprintf("stem%02d.wav", i)] = "X"
}
zipBody := stemsZipFixture(t, entries)
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/zip")
_, _ = w.Write(zipBody)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
ss, _ := p.StemSeparatorModel("audioutils")
_, err := ss.SeparateStems(context.Background(), audio.StemSeparationRequest{Audio: []byte("SONG")})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) || !strings.Contains(apiErr.Message, "entries") {
t.Fatalf("err = %v, want APIError for over-long stems zip", err)
}
}
// wavFixture is a minimal RIFF/WAVE header so http.DetectContentType sniffs // wavFixture is a minimal RIFF/WAVE header so http.DetectContentType sniffs
// audio/wave. // audio/wave.
func wavFixture() []byte { func wavFixture() []byte {