steveandClaude Fable 5
5f175ecf82
- videogen.LipSyncer/LipsyncProvider: SadTalker talking heads via
POST /upstream/<id>/v1/talking_head (multipart image+audio parts,
still/enhance/preprocess flags) -> mp4.
- videogen.VideoBackgroundRemover/VideoBackgroundRemovalProvider:
POST /upstream/<id>/v1/video/matte (output greenscreen_mp4|alpha_webm).
- videogen.VideoUpscaler/VideoUpscaleProvider:
POST /upstream/<id>/v1/video/upscale (scale 2|4).
- videogen.Chainer/ChainerProvider: async long-video chain-job client —
SubmitChain (JSON POST /v1/video/chain, init_image_b64), ChainStatus
(GET /v1/jobs/{id}, tolerant segment-id decode), ChainResult,
ChainSegmentResult (partial delivery after mid-chain failure); hostile
job-id path rejection.
- Shared singleVideoResult validation (positive video evidence) + a
videoInputFilename hint helper; httptest contract tests per surface;
ADR-0025 (index row deferred — MJ-A backfills the ADR index table and
parallel edits would conflict).
Co-Authored-By: Claude Fable 5 <[email protected]>
3.6 KiB
3.6 KiB
ADR-0025: Wave-3 video surfaces (lipsync, video matte, video upscale, chain jobs)
Status: Accepted (2026-07-16)
Context
The llama-swap host is gaining wave-3 video capabilities (spec: mort docs/specs/2026-07-16-llamaswap-wave3.md): SadTalker talking heads, Robust Video Matting and per-frame Real-ESRGAN on the mediautils shim, and a videoutils orchestrator that generates LONG videos as a chain of i2v segments (generate → extract last frame → continue → concat → optional RIFE smoothing), exposed as an async job API following the ACE-Step precedent.
Decision
- Three new optional
videogeninterfaces, ADR-0016→0024 conventions:videogen.LipSyncer/LipsyncProvider—LipsyncRequest{Image, Audio, AudioMIME, AudioFilename, Still, Enhance, Preprocess}→POST /upstream/<id>/v1/talking_head(multipartimage+audiofile parts + optionalstill/enhance/preprocessfields) → mp4. Sync and minutes-slow (Hunyuan precedent); context deadline is the budget.videogen.VideoBackgroundRemover/VideoBackgroundRemovalProvider—VideoBackgroundRemovalRequest{Video, MIME, Filename, Output}→POST /upstream/<id>/v1/video/matte.Output∈greenscreen_mp4(universally playable) |alpha_webm(true transparency); "" = backend default.videogen.VideoUpscaler/VideoUpscaleProvider—VideoUpscaleRequest{Video, MIME, Filename, Scale}(2 or 4) →POST /upstream/<id>/v1/video/upscale→ mp4.
- The chain client is deliberately ASYNC (
videogen.Chainer/ChainerProvider), unlike musicgen's blocking Generate (ADR-0021): a chain holds the GPU through multiple model swaps for many minutes, and the caller must be able to poll progress AND fetch completed segments after a mid-chain failure — partial delivery is mandatory (never discard multi-minute GPU output), which a blocking one-call contract cannot express.SubmitChain(ctx, ChainRequest{Segments[{Prompt,Seconds}], InitImage, SmoothJoins, Size}) (jobID, error)— JSONPOST /upstream/<id>/v1/video/chain, init image as base64init_image_b64(JSON submit, not multipart, per the pinned host contract).ChainStatus(ctx, jobID) (*ChainJob{Status, Segment, Total, SegmentIDs, Raw})—GET /v1/jobs/{id}; polling doubles as a liveness signal for the shim's idle TTL. Segment entries are tolerated as strings or{id|segment_id}objects.ChainResult(ctx, jobID)/ChainSegmentResult(ctx, jobID, n)—GET /v1/jobs/{id}/resultand/v1/jobs/{id}/segments/{n}.- Job ids are echoed server input: job paths reject ids carrying path
structure (
/?#,..), the upstreamPath smuggling rule.
- Binary success bodies are validated before wrapping (ADR-0020 rule):
all four clip-returning calls go through a shared
singleVideoResult(positive evidence of video-ness — declared video/* Content-Type or sniffed mp4/webm magic — so a JSON status page or proxy error can never become "the clip").
Consequences
- videogen grows from two surfaces (Model, Interpolator) to six; the chain client is the package's first async surface — the job-API shape deferred in ADR-0019/0021 now exists where the workload actually demands it.
- The wire shapes are pinned by the netherstorm videoutils/mediautils/ SadTalker image builds; host smoke tests are the drift defence.
- mort's long-video tool owns the poll loop, timeout budget, and partial-result envelope; majordomo only guarantees the artifacts stay fetchable.