steve
4629c6af0f
New videogen/ contract package (ADR-0019): Request/Result/Model/Provider
with the imagegen conventions. Text-to-video and image-to-video are one
surface (Request.InitImage, nil = t2v) since hybrid checkpoints like
Wan 2.2 TI2V serve both from one model; Result carries a single clip.
provider/llamaswap gains VideoModel(id) targeting the blocking
POST {base}/v1/videos/sync (multipart, model-routed by the fork's new
video routes): vLLM-Omni parameter names, OpenAI-style input_reference
file part, optional fields stay off the wire so per-model launch-flag
defaults apply. CLAUDE.md package map picks up audio/ (missed in #12)
and videogen/.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXQxVhXBw8PwFAtsVrXSmj
50 lines
2.7 KiB
Markdown
50 lines
2.7 KiB
Markdown
# ADR-0019: videogen — canonical video-generation surface
|
|
|
|
**Status:** Accepted — 2026-07-12
|
|
|
|
## Context
|
|
|
|
mort is adding local video generation (Wan 2.2 / LTX-class models on a 24GB
|
|
GPU) behind the same llama-swap instance that serves imagegen and audio. Like
|
|
those modalities, video generation shares none of the chat machinery, so it
|
|
needs its own small contract package (the ADR-0016/0017 pattern). Three shape
|
|
questions: one interface for text-to-video and image-to-video or two, batch or
|
|
single result, and which wire endpoint the llama-swap provider targets.
|
|
|
|
## Decision
|
|
|
|
- **New `videogen/` package** with the established conventions: `Request` /
|
|
`Result` / `Option` + `Apply`, `Model` / `ModelOption` / `Provider`, zero
|
|
values mean backend default, `Image = llm.ImagePart`.
|
|
- **Text-to-video and image-to-video are one surface.** `Request.InitImage
|
|
*Image` (nil = pure text-to-video) instead of an imagegen-style separate
|
|
`Editor` interface: hybrid checkpoints (Wan 2.2 TI2V) serve both modes from
|
|
the same model and endpoint, so a second interface would duplicate the
|
|
request shape for no dispatch benefit.
|
|
- **`Result` carries a single `Video`, not a batch.** The blocking sync
|
|
endpoint answers with the encoded clip as the response body — one request,
|
|
one clip. Batching multi-minute generations behind one HTTP request is the
|
|
wrong shape; if batch ever matters it arrives with an async job surface,
|
|
not by widening this one.
|
|
- **provider/llamaswap targets `POST {base}/v1/videos/sync`**
|
|
(multipart/form-data, llama-swap routes by the `model` form field; the
|
|
response body is the video). This is vLLM-Omni's blocking videos endpoint,
|
|
and the steve/llama-swap fork dispatches it as a model route. Parameter
|
|
names follow vLLM-Omni (`num_frames`, `fps`, `num_inference_steps`,
|
|
`guidance_scale`); the conditioning frame is an `input_reference` file part
|
|
per OpenAI's videos API. Optional fields stay off the wire so per-model
|
|
launch-flag defaults apply — the imagegen convention.
|
|
- **No polling in v1.** The async `POST /v1/videos` + `GET /v1/videos/{id}`
|
|
job flow is deliberately not wrapped: callers (mort's skill tools) already
|
|
run synchronous-with-generous-timeout and bound the call with a context
|
|
deadline. An async `Job` surface is a compatible later addition.
|
|
|
|
## Consequences
|
|
|
|
- A `videogen.Video` is its own type (`Data []byte`, `MIME string`); there is
|
|
no `llm.VideoPart`, and no chat-side video-input support is implied.
|
|
- Duration is expressed as `NumFrames` + `FPS` (the diffusion-native knobs),
|
|
not seconds; callers wanting seconds convert at their edge.
|
|
- Any upstream exposing the same `/v1/videos/sync` shape (e.g. a ComfyUI
|
|
shim) works unchanged; the contract does not name an engine.
|