feat: audio surfaces (TTS + transcription), imagegen.Editor, llamaswap health probe
CI / Tidy (pull_request) Successful in 10m9s
CI / Build & Test (pull_request) Successful in 11m16s
Adversarial Review (Gadfly) / review (pull_request) Successful in 14m51s

- New leaf package `audio` (ADR-0017): SpeechModel/SpeechProvider and
  TranscriptionModel/TranscriptionProvider with imagegen conventions
  (zero value = backend default, functional options + Apply, bytes
  in/out, never URLs). Root re-exports added.
- imagegen.Editor (ADR-0018): optional image-to-image interface —
  EditRequest carries the generation knobs plus Init image and
  denoising Strength; separate interface so existing Models keep
  compiling.
- provider/llamaswap implements all of it: POST /v1/audio/speech (JSON,
  raw-audio response, MIME from Content-Type with format fallback),
  POST /v1/audio/transcriptions (multipart, response_format=json),
  ListVoices (GET /v1/audio/voices?model=, tolerant of string-list and
  object-list shapes), POST /sdapi/v1/img2img (txt2img wire +
  init_images/denoising_strength, shared image decode), and Health(ctx)
  (GET /health) — a cheap liveness probe for often-offline hosts.
- Hermetic httptest coverage for every new wire shape and validation
  path; README sections + support-matrix footnote updated in the same
  commit (also corrects the stale /v1/images/generations claim — the
  image path has been SDAPI since the seed fix).

First consumer: mort's llamaswap media tool cluster (status / image /
TTS / STT agent tools against the netherstorm host).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXQxVhXBw8PwFAtsVrXSmj
This commit is contained in:
2026-07-11 23:24:14 -04:00
parent 6abb399f5b
commit 434d721b99
16 changed files with 1317 additions and 8 deletions
+45 -5
View File
@@ -175,7 +175,9 @@ LLM_LS=llama-swaps://token@swap.example.com # https → TLS-fronted instance
[llama-swap](https://github.com/mostlygeek/llama-swap) is a model-swapping proxy
over llama.cpp. Its chat API is OpenAI-compatible (majordomo reuses the openai
client), and the `*llamaswap.Provider` adds management methods
(`ListModels`/`Running`/`Unload`) plus image generation (see below). A cold
(`ListModels`/`Running`/`Unload`/`ListVoices`), a cheap liveness probe
(`Health`, GET /health — the proxy answers without touching a model), image
generation and editing, and speech synthesis/transcription (see below). A cold
model swap can take many seconds — bound calls with a context deadline, not a
client timeout.
@@ -211,8 +213,9 @@ resp, err := m.Generate(ctx, majordomo.Request{
Text-to-image is a separate contract (`imagegen`) from chat, because it shares
none of the message/tool/stream machinery. Generated images come back as
`llm.ImagePart`, so they drop straight back into a chat turn. The first backend
is llama-swap (OpenAI `/v1/images/generations` a stable-diffusion.cpp
upstream).
is llama-swap (the A1111-style `/sdapi/v1/txt2img` on a stable-diffusion.cpp
upstream — chosen over OpenAI `/v1/images/generations` because that route
ignores `seed` there).
```go
ls := llamaswap.New(llamaswap.WithBaseURL("http://box.local:8080"))
@@ -224,9 +227,45 @@ res, err := im.Generate(ctx, imagegen.Request{Prompt: "a red bicycle"},
// majordomo.UserParts(majordomo.Text("describe this"), res.Images[0])
```
Image-to-image editing is the optional `imagegen.Editor` interface (ADR-0018)
— llama-swap's image models implement it via `/sdapi/v1/img2img`:
```go
ed := im.(imagegen.Editor)
res, err := ed.Edit(ctx, imagegen.EditRequest{
Prompt: "make it night",
Init: res.Images[0], // any llm.ImagePart
}, imagegen.WithEditStrength(0.6)) // 0..1: how far to depart from Init
```
`*llamaswap.Provider` also exposes management methods: `ListModels` (what
llama-swap can serve), `Running` (what's loaded), and `Unload` (free a model).
## Speech: synthesis + transcription
Text-to-speech and speech-to-text live in the `audio` package (ADR-0017),
mirroring imagegen: small `SpeechModel`/`TranscriptionModel` contracts,
zero values mean backend defaults, bytes in/out (never URLs). First backend:
llama-swap (OpenAI `/v1/audio/speech` + `/v1/audio/transcriptions` routed to
kokoro/whisper.cpp-style upstreams).
```go
ls := llamaswap.New(llamaswap.WithBaseURL("http://box.local:8080"))
sm, _ := ls.SpeechModel("kokoro")
speech, err := sm.Speak(ctx, audio.SpeechRequest{Input: "hello world"},
audio.WithVoice("af_heart"), audio.WithFormat("mp3"))
// speech.Audio ([]byte) + speech.MIME ("audio/mpeg")
tm, _ := ls.TranscriptionModel("whisper-large-v3-turbo")
tr, err := tm.Transcribe(ctx, audio.TranscriptionRequest{
Audio: speech.Audio, MIME: speech.MIME,
})
// tr.Text
voices, err := ls.ListVoices(ctx, "kokoro") // []string of voice ids
```
## Tool calls
```go
@@ -358,8 +397,9 @@ response as a single delta plus final event.
² llama-swap's chat is OpenAI-compatible and reuses the openai client, so these
capabilities are present at the client level; whether a given call succeeds
depends on the llama.cpp model llama-swap loads. llama-swap also provides
**image generation** (a separate `imagegen` axis, not shown above) and
management methods on `*llamaswap.Provider`.
**image generation + editing** (`imagegen`), **speech synthesis +
transcription** (`audio`) — separate axes, not shown above — plus a `Health`
probe and management methods on `*llamaswap.Provider`.
Notes: Ollama has no native tool_choice — `"none"` drops the tools;
`"required"`/named choices are best-effort ignored there. Ollama Cloud