feat: audio surfaces (TTS + transcription), imagegen.Editor, llamaswap health probe
- New leaf package `audio` (ADR-0017): SpeechModel/SpeechProvider and TranscriptionModel/TranscriptionProvider with imagegen conventions (zero value = backend default, functional options + Apply, bytes in/out, never URLs). Root re-exports added. - imagegen.Editor (ADR-0018): optional image-to-image interface — EditRequest carries the generation knobs plus Init image and denoising Strength; separate interface so existing Models keep compiling. - provider/llamaswap implements all of it: POST /v1/audio/speech (JSON, raw-audio response, MIME from Content-Type with format fallback), POST /v1/audio/transcriptions (multipart, response_format=json), ListVoices (GET /v1/audio/voices?model=, tolerant of string-list and object-list shapes), POST /sdapi/v1/img2img (txt2img wire + init_images/denoising_strength, shared image decode), and Health(ctx) (GET /health) — a cheap liveness probe for often-offline hosts. - Hermetic httptest coverage for every new wire shape and validation path; README sections + support-matrix footnote updated in the same commit (also corrects the stale /v1/images/generations claim — the image path has been SDAPI since the seed fix). First consumer: mort's llamaswap media tool cluster (status / image / TTS / STT agent tools against the netherstorm host). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXQxVhXBw8PwFAtsVrXSmj
This commit is contained in:
@@ -175,7 +175,9 @@ LLM_LS=llama-swaps://token@swap.example.com # https → TLS-fronted instance
|
||||
[llama-swap](https://github.com/mostlygeek/llama-swap) is a model-swapping proxy
|
||||
over llama.cpp. Its chat API is OpenAI-compatible (majordomo reuses the openai
|
||||
client), and the `*llamaswap.Provider` adds management methods
|
||||
(`ListModels`/`Running`/`Unload`) plus image generation (see below). A cold
|
||||
(`ListModels`/`Running`/`Unload`/`ListVoices`), a cheap liveness probe
|
||||
(`Health`, GET /health — the proxy answers without touching a model), image
|
||||
generation and editing, and speech synthesis/transcription (see below). A cold
|
||||
model swap can take many seconds — bound calls with a context deadline, not a
|
||||
client timeout.
|
||||
|
||||
@@ -211,8 +213,9 @@ resp, err := m.Generate(ctx, majordomo.Request{
|
||||
Text-to-image is a separate contract (`imagegen`) from chat, because it shares
|
||||
none of the message/tool/stream machinery. Generated images come back as
|
||||
`llm.ImagePart`, so they drop straight back into a chat turn. The first backend
|
||||
is llama-swap (OpenAI `/v1/images/generations` → a stable-diffusion.cpp
|
||||
upstream).
|
||||
is llama-swap (the A1111-style `/sdapi/v1/txt2img` on a stable-diffusion.cpp
|
||||
upstream — chosen over OpenAI `/v1/images/generations` because that route
|
||||
ignores `seed` there).
|
||||
|
||||
```go
|
||||
ls := llamaswap.New(llamaswap.WithBaseURL("http://box.local:8080"))
|
||||
@@ -224,9 +227,45 @@ res, err := im.Generate(ctx, imagegen.Request{Prompt: "a red bicycle"},
|
||||
// majordomo.UserParts(majordomo.Text("describe this"), res.Images[0])
|
||||
```
|
||||
|
||||
Image-to-image editing is the optional `imagegen.Editor` interface (ADR-0018)
|
||||
— llama-swap's image models implement it via `/sdapi/v1/img2img`:
|
||||
|
||||
```go
|
||||
ed := im.(imagegen.Editor)
|
||||
res, err := ed.Edit(ctx, imagegen.EditRequest{
|
||||
Prompt: "make it night",
|
||||
Init: res.Images[0], // any llm.ImagePart
|
||||
}, imagegen.WithEditStrength(0.6)) // 0..1: how far to depart from Init
|
||||
```
|
||||
|
||||
`*llamaswap.Provider` also exposes management methods: `ListModels` (what
|
||||
llama-swap can serve), `Running` (what's loaded), and `Unload` (free a model).
|
||||
|
||||
## Speech: synthesis + transcription
|
||||
|
||||
Text-to-speech and speech-to-text live in the `audio` package (ADR-0017),
|
||||
mirroring imagegen: small `SpeechModel`/`TranscriptionModel` contracts,
|
||||
zero values mean backend defaults, bytes in/out (never URLs). First backend:
|
||||
llama-swap (OpenAI `/v1/audio/speech` + `/v1/audio/transcriptions` routed to
|
||||
kokoro/whisper.cpp-style upstreams).
|
||||
|
||||
```go
|
||||
ls := llamaswap.New(llamaswap.WithBaseURL("http://box.local:8080"))
|
||||
|
||||
sm, _ := ls.SpeechModel("kokoro")
|
||||
speech, err := sm.Speak(ctx, audio.SpeechRequest{Input: "hello world"},
|
||||
audio.WithVoice("af_heart"), audio.WithFormat("mp3"))
|
||||
// speech.Audio ([]byte) + speech.MIME ("audio/mpeg")
|
||||
|
||||
tm, _ := ls.TranscriptionModel("whisper-large-v3-turbo")
|
||||
tr, err := tm.Transcribe(ctx, audio.TranscriptionRequest{
|
||||
Audio: speech.Audio, MIME: speech.MIME,
|
||||
})
|
||||
// tr.Text
|
||||
|
||||
voices, err := ls.ListVoices(ctx, "kokoro") // []string of voice ids
|
||||
```
|
||||
|
||||
## Tool calls
|
||||
|
||||
```go
|
||||
@@ -358,8 +397,9 @@ response as a single delta plus final event.
|
||||
² llama-swap's chat is OpenAI-compatible and reuses the openai client, so these
|
||||
capabilities are present at the client level; whether a given call succeeds
|
||||
depends on the llama.cpp model llama-swap loads. llama-swap also provides
|
||||
**image generation** (a separate `imagegen` axis, not shown above) and
|
||||
management methods on `*llamaswap.Provider`.
|
||||
**image generation + editing** (`imagegen`), **speech synthesis +
|
||||
transcription** (`audio`) — separate axes, not shown above — plus a `Health`
|
||||
probe and management methods on `*llamaswap.Provider`.
|
||||
|
||||
Notes: Ollama has no native tool_choice — `"none"` drops the tools;
|
||||
`"required"`/named choices are best-effort ignored there. Ollama Cloud
|
||||
|
||||
Reference in New Issue
Block a user