feat(videogen): canonical video-generation surface + llama-swap client
Adversarial Review (Gadfly) / review (pull_request) Successful in 8m43s
CI / Tidy (pull_request) Successful in 9m28s
CI / Build & Test (pull_request) Successful in 10m11s

New videogen/ contract package (ADR-0019): Request/Result/Model/Provider
with the imagegen conventions. Text-to-video and image-to-video are one
surface (Request.InitImage, nil = t2v) since hybrid checkpoints like
Wan 2.2 TI2V serve both from one model; Result carries a single clip.

provider/llamaswap gains VideoModel(id) targeting the blocking
POST {base}/v1/videos/sync (multipart, model-routed by the fork's new
video routes): vLLM-Omni parameter names, OpenAI-style input_reference
file part, optional fields stay off the wire so per-model launch-flag
defaults apply. CLAUDE.md package map picks up audio/ (missed in #12)
and videogen/.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXQxVhXBw8PwFAtsVrXSmj
This commit is contained in:
2026-07-12 09:31:08 -04:00
parent 5ed18abb94
commit 4629c6af0f
6 changed files with 640 additions and 1 deletions
+49
View File
@@ -0,0 +1,49 @@
# ADR-0019: videogen — canonical video-generation surface
**Status:** Accepted — 2026-07-12
## Context
mort is adding local video generation (Wan 2.2 / LTX-class models on a 24GB
GPU) behind the same llama-swap instance that serves imagegen and audio. Like
those modalities, video generation shares none of the chat machinery, so it
needs its own small contract package (the ADR-0016/0017 pattern). Three shape
questions: one interface for text-to-video and image-to-video or two, batch or
single result, and which wire endpoint the llama-swap provider targets.
## Decision
- **New `videogen/` package** with the established conventions: `Request` /
`Result` / `Option` + `Apply`, `Model` / `ModelOption` / `Provider`, zero
values mean backend default, `Image = llm.ImagePart`.
- **Text-to-video and image-to-video are one surface.** `Request.InitImage
*Image` (nil = pure text-to-video) instead of an imagegen-style separate
`Editor` interface: hybrid checkpoints (Wan 2.2 TI2V) serve both modes from
the same model and endpoint, so a second interface would duplicate the
request shape for no dispatch benefit.
- **`Result` carries a single `Video`, not a batch.** The blocking sync
endpoint answers with the encoded clip as the response body — one request,
one clip. Batching multi-minute generations behind one HTTP request is the
wrong shape; if batch ever matters it arrives with an async job surface,
not by widening this one.
- **provider/llamaswap targets `POST {base}/v1/videos/sync`**
(multipart/form-data, llama-swap routes by the `model` form field; the
response body is the video). This is vLLM-Omni's blocking videos endpoint,
and the steve/llama-swap fork dispatches it as a model route. Parameter
names follow vLLM-Omni (`num_frames`, `fps`, `num_inference_steps`,
`guidance_scale`); the conditioning frame is an `input_reference` file part
per OpenAI's videos API. Optional fields stay off the wire so per-model
launch-flag defaults apply — the imagegen convention.
- **No polling in v1.** The async `POST /v1/videos` + `GET /v1/videos/{id}`
job flow is deliberately not wrapped: callers (mort's skill tools) already
run synchronous-with-generous-timeout and bound the call with a context
deadline. An async `Job` surface is a compatible later addition.
## Consequences
- A `videogen.Video` is its own type (`Data []byte`, `MIME string`); there is
no `llm.VideoPart`, and no chat-side video-input support is implied.
- Duration is expressed as `NumFrames` + `FPS` (the diffusion-native knobs),
not seconds; callers wanting seconds convert at their edge.
- Any upstream exposing the same `/v1/videos/sync` shape (e.g. a ComfyUI
shim) works unchanged; the contract does not name an engine.