steve
4629c6af0f
New videogen/ contract package (ADR-0019): Request/Result/Model/Provider
with the imagegen conventions. Text-to-video and image-to-video are one
surface (Request.InitImage, nil = t2v) since hybrid checkpoints like
Wan 2.2 TI2V serve both from one model; Result carries a single clip.
provider/llamaswap gains VideoModel(id) targeting the blocking
POST {base}/v1/videos/sync (multipart, model-routed by the fork's new
video routes): vLLM-Omni parameter names, OpenAI-style input_reference
file part, optional fields stay off the wire so per-model launch-flag
defaults apply. CLAUDE.md package map picks up audio/ (missed in #12)
and videogen/.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXQxVhXBw8PwFAtsVrXSmj
2.7 KiB
2.7 KiB
ADR-0019: videogen — canonical video-generation surface
Status: Accepted — 2026-07-12
Context
mort is adding local video generation (Wan 2.2 / LTX-class models on a 24GB GPU) behind the same llama-swap instance that serves imagegen and audio. Like those modalities, video generation shares none of the chat machinery, so it needs its own small contract package (the ADR-0016/0017 pattern). Three shape questions: one interface for text-to-video and image-to-video or two, batch or single result, and which wire endpoint the llama-swap provider targets.
Decision
- New
videogen/package with the established conventions:Request/Result/Option+Apply,Model/ModelOption/Provider, zero values mean backend default,Image = llm.ImagePart. - Text-to-video and image-to-video are one surface.
Request.InitImage *Image(nil = pure text-to-video) instead of an imagegen-style separateEditorinterface: hybrid checkpoints (Wan 2.2 TI2V) serve both modes from the same model and endpoint, so a second interface would duplicate the request shape for no dispatch benefit. Resultcarries a singleVideo, not a batch. The blocking sync endpoint answers with the encoded clip as the response body — one request, one clip. Batching multi-minute generations behind one HTTP request is the wrong shape; if batch ever matters it arrives with an async job surface, not by widening this one.- provider/llamaswap targets
POST {base}/v1/videos/sync(multipart/form-data, llama-swap routes by themodelform field; the response body is the video). This is vLLM-Omni's blocking videos endpoint, and the steve/llama-swap fork dispatches it as a model route. Parameter names follow vLLM-Omni (num_frames,fps,num_inference_steps,guidance_scale); the conditioning frame is aninput_referencefile part per OpenAI's videos API. Optional fields stay off the wire so per-model launch-flag defaults apply — the imagegen convention. - No polling in v1. The async
POST /v1/videos+GET /v1/videos/{id}job flow is deliberately not wrapped: callers (mort's skill tools) already run synchronous-with-generous-timeout and bound the call with a context deadline. An asyncJobsurface is a compatible later addition.
Consequences
- A
videogen.Videois its own type (Data []byte,MIME string); there is nollm.VideoPart, and no chat-side video-input support is implied. - Duration is expressed as
NumFrames+FPS(the diffusion-native knobs), not seconds; callers wanting seconds convert at their edge. - Any upstream exposing the same
/v1/videos/syncshape (e.g. a ComfyUI shim) works unchanged; the contract does not name an engine.