feat: audio surfaces (TTS + transcription), imagegen.Editor, llamaswap health probe
CI / Tidy (pull_request) Successful in 10m9s
CI / Build & Test (pull_request) Successful in 11m16s
Adversarial Review (Gadfly) / review (pull_request) Successful in 14m51s

- New leaf package `audio` (ADR-0017): SpeechModel/SpeechProvider and
  TranscriptionModel/TranscriptionProvider with imagegen conventions
  (zero value = backend default, functional options + Apply, bytes
  in/out, never URLs). Root re-exports added.
- imagegen.Editor (ADR-0018): optional image-to-image interface —
  EditRequest carries the generation knobs plus Init image and
  denoising Strength; separate interface so existing Models keep
  compiling.
- provider/llamaswap implements all of it: POST /v1/audio/speech (JSON,
  raw-audio response, MIME from Content-Type with format fallback),
  POST /v1/audio/transcriptions (multipart, response_format=json),
  ListVoices (GET /v1/audio/voices?model=, tolerant of string-list and
  object-list shapes), POST /sdapi/v1/img2img (txt2img wire +
  init_images/denoising_strength, shared image decode), and Health(ctx)
  (GET /health) — a cheap liveness probe for often-offline hosts.
- Hermetic httptest coverage for every new wire shape and validation
  path; README sections + support-matrix footnote updated in the same
  commit (also corrects the stale /v1/images/generations claim — the
  image path has been SDAPI since the seed fix).

First consumer: mort's llamaswap media tool cluster (status / image /
TTS / STT agent tools against the netherstorm host).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXQxVhXBw8PwFAtsVrXSmj
This commit is contained in:
2026-07-11 23:24:14 -04:00
parent 6abb399f5b
commit 434d721b99
16 changed files with 1317 additions and 8 deletions
+53
View File
@@ -0,0 +1,53 @@
# ADR-0017: audio — canonical speech synthesis + transcription interfaces
**Status:** Accepted — 2026-07-11
## Context
mort is growing agent tools that speak (TTS) and transcribe audio through a
llama-swap host whose upstreams expose the OpenAI `/v1/audio/speech` and
`/v1/audio/transcriptions` endpoints (kokoro, chatterbox, whisper.cpp). Like
image generation before it (ADR-0016), speech shares none of the chat
contract's message/tool/stream machinery, and majordomo had no speech surface
— an earlier migration doc explicitly scoped transcription out of the llm
package. The same reasoning that produced `imagegen` applies.
## Decision
- One new canonical **leaf package `audio`** holding both directions —
synthesis and transcription — rather than two packages; they are one
modality and will share future types (voice metadata, audio formats).
Root re-exports mirror imagegen (`SpeechModel`, `SpeechProvider`,
`SpeechRequest`, `SpeechResult`, `TranscriptionModel`, ...).
- Minimal v1 surface, imagegen conventions throughout (zero value = backend
default, functional options + `Apply`, `Raw any` escape hatch):
- `SpeechRequest{ Input; Voice; Format; Speed }`
`SpeechResult{ Audio []byte; MIME string; Raw }`;
`SpeechModel.Speak(ctx, req, ...opts)`;
`SpeechProvider.SpeechModel(id, ...)`.
- `TranscriptionRequest{ Audio []byte; MIME; Filename; Language; Prompt }`
`TranscriptionResult{ Text string; Raw }`;
`TranscriptionModel.Transcribe(ctx, req, ...opts)`;
`TranscriptionProvider.TranscriptionModel(id, ...)`.
- **Bytes in/out, never URLs** — mirrors `llm.ImagePart`'s bytes-only
contract; fetching is the caller's concern.
- Providers are split (`SpeechProvider` vs `TranscriptionProvider`) so a
backend can implement either half; llamaswap implements both.
- First implementation: `provider/llamaswap``/v1/audio/speech` (JSON body,
raw audio response; MIME from Content-Type with a format-based fallback),
`/v1/audio/transcriptions` (multipart, `response_format=json`), plus
`ListVoices(ctx, model)` (GET `/v1/audio/voices?model=`, tolerant of the
string-list and object-list shapes upstreams use) as a llamaswap management
method, not part of the canonical contract.
- Out of scope for v1 (designed-for, deferred): streaming synthesis,
word-level timestamps/segments, translation, voice cloning inputs, and
registry-level DSN resolution for audio models.
## Consequences
- Speech is provider-agnostic from day one; an OpenAI or Google speech
backend implements the same interfaces.
- `SpeechResult.Audio` is a plain byte slice, so results flow into any file
store or attachment pipeline without a majordomo dependency.
- The `audio` package name is the modality, not the direction; if music/sfx
generation ever lands it has a home.
+36
View File
@@ -0,0 +1,36 @@
# ADR-0018: imagegen.Editor — image-to-image as a separate optional interface
**Status:** Accepted — 2026-07-11
## Context
ADR-0016 shipped text-to-image and explicitly deferred img2img. mort's new
llama-swap media tools need "edit this image under this prompt"
(image-to-image with a denoising strength). Two shape questions: does Edit
belong on `imagegen.Model`, and which llama-swap endpoint carries it —
OpenAI-style `/v1/images/edits` (multipart) or A1111-style `/sdapi/v1/img2img`
(JSON)?
## Decision
- **`Editor` is a separate, optional interface** (`Edit(ctx, EditRequest,
...EditOption) (*Result, error)`), not a new method on `Model`. Existing
`Model` implementations keep compiling; callers type-assert
(`m.(imagegen.Editor)`) or require the capability explicitly.
- `EditRequest` = the generation knobs (prompt, N, size, steps, cfg, negative
prompt, sampler, seed) plus `Init Image` (required) and `Strength *float64`
(denoising strength in [0,1]; nil = backend default). Same option/Apply
conventions; result type is the shared `Result`.
- llama-swap implements it via **`/sdapi/v1/img2img`**, not
`/v1/images/edits`: the same sd-server build that ignores `seed` on the
OpenAI images route (the reason txt2img went SDAPI in ADR-0016's
implementation) applies; the JSON shape is txt2img's plus
`init_images: ["<b64>"]` + `denoising_strength`, so it reuses `doJSON`
verbatim, where the OpenAI route is multipart.
## Consequences
- Backends that can't edit simply don't implement `Editor`; no stub methods.
- The init image travels base64-inline in JSON (~33% overhead) — acceptable at
chat-image sizes; a future backend needing multipart can still satisfy the
same interface.
+2
View File
@@ -20,3 +20,5 @@ One decision per file, append-only; supersede rather than rewrite.
| [0014](0014-conversion-driven-extensions.md) | Conversion-driven extensions (resolvers, typed tools, hooks, ops controls) | Accepted |
| [0015](0015-llama-swap-provider.md) | llama-swap provider — reuse openai for chat, tailored management + image | Accepted |
| [0016](0016-imagegen-interface.md) | imagegen — a canonical text-to-image interface | Accepted |
| [0017](0017-audio-interfaces.md) | audio — canonical speech synthesis + transcription interfaces | Accepted |
| [0018](0018-imagegen-editor.md) | imagegen.Editor — image-to-image as a separate optional interface | Accepted |