steve
434d721b99
- New leaf package `audio` (ADR-0017): SpeechModel/SpeechProvider and TranscriptionModel/TranscriptionProvider with imagegen conventions (zero value = backend default, functional options + Apply, bytes in/out, never URLs). Root re-exports added. - imagegen.Editor (ADR-0018): optional image-to-image interface — EditRequest carries the generation knobs plus Init image and denoising Strength; separate interface so existing Models keep compiling. - provider/llamaswap implements all of it: POST /v1/audio/speech (JSON, raw-audio response, MIME from Content-Type with format fallback), POST /v1/audio/transcriptions (multipart, response_format=json), ListVoices (GET /v1/audio/voices?model=, tolerant of string-list and object-list shapes), POST /sdapi/v1/img2img (txt2img wire + init_images/denoising_strength, shared image decode), and Health(ctx) (GET /health) — a cheap liveness probe for often-offline hosts. - Hermetic httptest coverage for every new wire shape and validation path; README sections + support-matrix footnote updated in the same commit (also corrects the stale /v1/images/generations claim — the image path has been SDAPI since the seed fix). First consumer: mort's llamaswap media tool cluster (status / image / TTS / STT agent tools against the netherstorm host). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXQxVhXBw8PwFAtsVrXSmj
2.8 KiB
2.8 KiB
ADR-0017: audio — canonical speech synthesis + transcription interfaces
Status: Accepted — 2026-07-11
Context
mort is growing agent tools that speak (TTS) and transcribe audio through a
llama-swap host whose upstreams expose the OpenAI /v1/audio/speech and
/v1/audio/transcriptions endpoints (kokoro, chatterbox, whisper.cpp). Like
image generation before it (ADR-0016), speech shares none of the chat
contract's message/tool/stream machinery, and majordomo had no speech surface
— an earlier migration doc explicitly scoped transcription out of the llm
package. The same reasoning that produced imagegen applies.
Decision
- One new canonical leaf package
audioholding both directions — synthesis and transcription — rather than two packages; they are one modality and will share future types (voice metadata, audio formats). Root re-exports mirror imagegen (SpeechModel,SpeechProvider,SpeechRequest,SpeechResult,TranscriptionModel, ...). - Minimal v1 surface, imagegen conventions throughout (zero value = backend
default, functional options +
Apply,Raw anyescape hatch):SpeechRequest{ Input; Voice; Format; Speed }→SpeechResult{ Audio []byte; MIME string; Raw };SpeechModel.Speak(ctx, req, ...opts);SpeechProvider.SpeechModel(id, ...).TranscriptionRequest{ Audio []byte; MIME; Filename; Language; Prompt }→TranscriptionResult{ Text string; Raw };TranscriptionModel.Transcribe(ctx, req, ...opts);TranscriptionProvider.TranscriptionModel(id, ...).
- Bytes in/out, never URLs — mirrors
llm.ImagePart's bytes-only contract; fetching is the caller's concern. - Providers are split (
SpeechProvidervsTranscriptionProvider) so a backend can implement either half; llamaswap implements both. - First implementation:
provider/llamaswap—/v1/audio/speech(JSON body, raw audio response; MIME from Content-Type with a format-based fallback),/v1/audio/transcriptions(multipart,response_format=json), plusListVoices(ctx, model)(GET/v1/audio/voices?model=, tolerant of the string-list and object-list shapes upstreams use) as a llamaswap management method, not part of the canonical contract. - Out of scope for v1 (designed-for, deferred): streaming synthesis, word-level timestamps/segments, translation, voice cloning inputs, and registry-level DSN resolution for audio models.
Consequences
- Speech is provider-agnostic from day one; an OpenAI or Google speech backend implements the same interfaces.
SpeechResult.Audiois a plain byte slice, so results flow into any file store or attachment pipeline without a majordomo dependency.- The
audiopackage name is the modality, not the direction; if music/sfx generation ever lands it has a home.