Merge pull request 'feat: wave-3 video surfaces — lipsync, video matte, video upscale, chain jobs (ADR-0025)' (#19) from feat/wave3-video-surfaces into main
CI / Tidy (push) Successful in 9m24s
CI / Build & Test (push) Successful in 9m46s

This commit was merged in pull request #19.
This commit is contained in:
2026-07-16 23:25:18 +00:00
12 changed files with 1507 additions and 24 deletions
+67
View File
@@ -0,0 +1,67 @@
# ADR-0025: Wave-3 video surfaces (lipsync, video matte, video upscale, chain jobs)
Status: Accepted (2026-07-16)
## Context
The llama-swap host is gaining wave-3 video capabilities (spec: mort
docs/specs/2026-07-16-llamaswap-wave3.md): SadTalker talking heads, Robust
Video Matting and per-frame Real-ESRGAN on the mediautils shim, and a
videoutils orchestrator that generates LONG videos as a chain of i2v
segments (generate → extract last frame → continue → concat → optional RIFE
smoothing), exposed as an async job API following the ACE-Step precedent.
## Decision
1. **Three new optional `videogen` interfaces**, ADR-0016→0024 conventions:
- `videogen.LipSyncer` / `LipsyncProvider` — `LipsyncRequest{Image,
Audio, AudioMIME, AudioFilename, Still, Enhance, Preprocess}` →
`POST /upstream/<id>/v1/talking_head` (multipart `image` + `audio`
file parts + optional `still`/`enhance`/`preprocess` fields) → mp4.
Sync and minutes-slow (Hunyuan precedent); context deadline is the
budget.
- `videogen.VideoBackgroundRemover` / `VideoBackgroundRemovalProvider` —
`VideoBackgroundRemovalRequest{Video, MIME, Filename, Output}` →
`POST /upstream/<id>/v1/video/matte`. `Output` ∈
`greenscreen_mp4` (universally playable) | `alpha_webm` (true
transparency); "" = backend default.
- `videogen.VideoUpscaler` / `VideoUpscaleProvider` —
`VideoUpscaleRequest{Video, MIME, Filename, Scale}` (2 or 4) →
`POST /upstream/<id>/v1/video/upscale` → mp4.
2. **The chain client is deliberately ASYNC** (`videogen.Chainer` /
`ChainerProvider`), unlike musicgen's blocking Generate (ADR-0021): a
chain holds the GPU through multiple model swaps for many minutes, and
the caller must be able to poll progress AND fetch completed segments
after a mid-chain failure — partial delivery is mandatory (never discard
multi-minute GPU output), which a blocking one-call contract cannot
express.
- `SubmitChain(ctx, ChainRequest{Segments[{Prompt,Seconds}], InitImage,
SmoothJoins, Size}) (jobID, error)` — JSON
`POST /upstream/<id>/v1/video/chain`, init image as base64
`init_image_b64` (JSON submit, not multipart, per the pinned host
contract).
- `ChainStatus(ctx, jobID) (*ChainJob{Status, Segment, Total,
SegmentIDs, Raw})` — `GET /v1/jobs/{id}`; polling doubles as a
liveness signal for the shim's idle TTL. Segment entries are tolerated
as strings or `{id|segment_id}` objects.
- `ChainResult(ctx, jobID)` / `ChainSegmentResult(ctx, jobID, n)` —
`GET /v1/jobs/{id}/result` and `/v1/jobs/{id}/segments/{n}`.
- Job ids are echoed server input: job paths reject ids carrying path
structure (`/?#`, `..`), the upstreamPath smuggling rule.
3. **Binary success bodies are validated before wrapping** (ADR-0020 rule):
all four clip-returning calls go through a shared `singleVideoResult`
(positive evidence of video-ness — declared video/* Content-Type or
sniffed mp4/webm magic — so a JSON status page or proxy error can never
become "the clip").
## Consequences
- videogen grows from two surfaces (Model, Interpolator) to six; the
chain client is the package's first async surface — the job-API shape
deferred in ADR-0019/0021 now exists where the workload actually
demands it.
- The wire shapes are pinned by the netherstorm videoutils/mediautils/
SadTalker image builds; host smoke tests are the drift defence.
- mort's long-video tool owns the poll loop, timeout budget, and
partial-result envelope; majordomo only guarantees the artifacts stay
fetchable.