feat(videogen): canonical video-generation surface + llama-swap client
New videogen/ contract package (ADR-0019): Request/Result/Model/Provider
with the imagegen conventions. Text-to-video and image-to-video are one
surface (Request.InitImage, nil = t2v) since hybrid checkpoints like
Wan 2.2 TI2V serve both from one model; Result carries a single clip.
provider/llamaswap gains VideoModel(id) targeting the blocking
POST {base}/v1/videos/sync (multipart, model-routed by the fork's new
video routes): vLLM-Omni parameter names, OpenAI-style input_reference
file part, optional fields stay off the wire so per-model launch-flag
defaults apply. CLAUDE.md package map picks up audio/ (missed in #12)
and videogen/.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXQxVhXBw8PwFAtsVrXSmj
This commit is contained in:
@@ -0,0 +1,49 @@
|
||||
# ADR-0019: videogen — canonical video-generation surface
|
||||
|
||||
**Status:** Accepted — 2026-07-12
|
||||
|
||||
## Context
|
||||
|
||||
mort is adding local video generation (Wan 2.2 / LTX-class models on a 24GB
|
||||
GPU) behind the same llama-swap instance that serves imagegen and audio. Like
|
||||
those modalities, video generation shares none of the chat machinery, so it
|
||||
needs its own small contract package (the ADR-0016/0017 pattern). Three shape
|
||||
questions: one interface for text-to-video and image-to-video or two, batch or
|
||||
single result, and which wire endpoint the llama-swap provider targets.
|
||||
|
||||
## Decision
|
||||
|
||||
- **New `videogen/` package** with the established conventions: `Request` /
|
||||
`Result` / `Option` + `Apply`, `Model` / `ModelOption` / `Provider`, zero
|
||||
values mean backend default, `Image = llm.ImagePart`.
|
||||
- **Text-to-video and image-to-video are one surface.** `Request.InitImage
|
||||
*Image` (nil = pure text-to-video) instead of an imagegen-style separate
|
||||
`Editor` interface: hybrid checkpoints (Wan 2.2 TI2V) serve both modes from
|
||||
the same model and endpoint, so a second interface would duplicate the
|
||||
request shape for no dispatch benefit.
|
||||
- **`Result` carries a single `Video`, not a batch.** The blocking sync
|
||||
endpoint answers with the encoded clip as the response body — one request,
|
||||
one clip. Batching multi-minute generations behind one HTTP request is the
|
||||
wrong shape; if batch ever matters it arrives with an async job surface,
|
||||
not by widening this one.
|
||||
- **provider/llamaswap targets `POST {base}/v1/videos/sync`**
|
||||
(multipart/form-data, llama-swap routes by the `model` form field; the
|
||||
response body is the video). This is vLLM-Omni's blocking videos endpoint,
|
||||
and the steve/llama-swap fork dispatches it as a model route. Parameter
|
||||
names follow vLLM-Omni (`num_frames`, `fps`, `num_inference_steps`,
|
||||
`guidance_scale`); the conditioning frame is an `input_reference` file part
|
||||
per OpenAI's videos API. Optional fields stay off the wire so per-model
|
||||
launch-flag defaults apply — the imagegen convention.
|
||||
- **No polling in v1.** The async `POST /v1/videos` + `GET /v1/videos/{id}`
|
||||
job flow is deliberately not wrapped: callers (mort's skill tools) already
|
||||
run synchronous-with-generous-timeout and bound the call with a context
|
||||
deadline. An async `Job` surface is a compatible later addition.
|
||||
|
||||
## Consequences
|
||||
|
||||
- A `videogen.Video` is its own type (`Data []byte`, `MIME string`); there is
|
||||
no `llm.VideoPart`, and no chat-side video-input support is implied.
|
||||
- Duration is expressed as `NumFrames` + `FPS` (the diffusion-native knobs),
|
||||
not seconds; callers wanting seconds convert at their edge.
|
||||
- Any upstream exposing the same `/v1/videos/sync` shape (e.g. a ComfyUI
|
||||
shim) works unchanged; the contract does not name an engine.
|
||||
Reference in New Issue
Block a user