Merge pull request 'feat: wave-3 video surfaces — lipsync, video matte, video upscale, chain jobs (ADR-0025)' (#19) from feat/wave3-video-surfaces into main
This commit was merged in pull request #19.
This commit is contained in:
@@ -0,0 +1,67 @@
|
||||
# ADR-0025: Wave-3 video surfaces (lipsync, video matte, video upscale, chain jobs)
|
||||
|
||||
Status: Accepted (2026-07-16)
|
||||
|
||||
## Context
|
||||
|
||||
The llama-swap host is gaining wave-3 video capabilities (spec: mort
|
||||
docs/specs/2026-07-16-llamaswap-wave3.md): SadTalker talking heads, Robust
|
||||
Video Matting and per-frame Real-ESRGAN on the mediautils shim, and a
|
||||
videoutils orchestrator that generates LONG videos as a chain of i2v
|
||||
segments (generate → extract last frame → continue → concat → optional RIFE
|
||||
smoothing), exposed as an async job API following the ACE-Step precedent.
|
||||
|
||||
## Decision
|
||||
|
||||
1. **Three new optional `videogen` interfaces**, ADR-0016→0024 conventions:
|
||||
- `videogen.LipSyncer` / `LipsyncProvider` — `LipsyncRequest{Image,
|
||||
Audio, AudioMIME, AudioFilename, Still, Enhance, Preprocess}` →
|
||||
`POST /upstream/<id>/v1/talking_head` (multipart `image` + `audio`
|
||||
file parts + optional `still`/`enhance`/`preprocess` fields) → mp4.
|
||||
Sync and minutes-slow (Hunyuan precedent); context deadline is the
|
||||
budget.
|
||||
- `videogen.VideoBackgroundRemover` / `VideoBackgroundRemovalProvider` —
|
||||
`VideoBackgroundRemovalRequest{Video, MIME, Filename, Output}` →
|
||||
`POST /upstream/<id>/v1/video/matte`. `Output` ∈
|
||||
`greenscreen_mp4` (universally playable) | `alpha_webm` (true
|
||||
transparency); "" = backend default.
|
||||
- `videogen.VideoUpscaler` / `VideoUpscaleProvider` —
|
||||
`VideoUpscaleRequest{Video, MIME, Filename, Scale}` (2 or 4) →
|
||||
`POST /upstream/<id>/v1/video/upscale` → mp4.
|
||||
2. **The chain client is deliberately ASYNC** (`videogen.Chainer` /
|
||||
`ChainerProvider`), unlike musicgen's blocking Generate (ADR-0021): a
|
||||
chain holds the GPU through multiple model swaps for many minutes, and
|
||||
the caller must be able to poll progress AND fetch completed segments
|
||||
after a mid-chain failure — partial delivery is mandatory (never discard
|
||||
multi-minute GPU output), which a blocking one-call contract cannot
|
||||
express.
|
||||
- `SubmitChain(ctx, ChainRequest{Segments[{Prompt,Seconds}], InitImage,
|
||||
SmoothJoins, Size}) (jobID, error)` — JSON
|
||||
`POST /upstream/<id>/v1/video/chain`, init image as base64
|
||||
`init_image_b64` (JSON submit, not multipart, per the pinned host
|
||||
contract).
|
||||
- `ChainStatus(ctx, jobID) (*ChainJob{Status, Segment, Total,
|
||||
SegmentIDs, Raw})` — `GET /v1/jobs/{id}`; polling doubles as a
|
||||
liveness signal for the shim's idle TTL. Segment entries are tolerated
|
||||
as strings or `{id|segment_id}` objects.
|
||||
- `ChainResult(ctx, jobID)` / `ChainSegmentResult(ctx, jobID, n)` —
|
||||
`GET /v1/jobs/{id}/result` and `/v1/jobs/{id}/segments/{n}`.
|
||||
- Job ids are echoed server input: job paths reject ids carrying path
|
||||
structure (`/?#`, `..`), the upstreamPath smuggling rule.
|
||||
3. **Binary success bodies are validated before wrapping** (ADR-0020 rule):
|
||||
all four clip-returning calls go through a shared `singleVideoResult`
|
||||
(positive evidence of video-ness — declared video/* Content-Type or
|
||||
sniffed mp4/webm magic — so a JSON status page or proxy error can never
|
||||
become "the clip").
|
||||
|
||||
## Consequences
|
||||
|
||||
- videogen grows from two surfaces (Model, Interpolator) to six; the
|
||||
chain client is the package's first async surface — the job-API shape
|
||||
deferred in ADR-0019/0021 now exists where the workload actually
|
||||
demands it.
|
||||
- The wire shapes are pinned by the netherstorm videoutils/mediautils/
|
||||
SadTalker image builds; host smoke tests are the drift defence.
|
||||
- mort's long-video tool owns the poll loop, timeout budget, and
|
||||
partial-result envelope; majordomo only guarantees the artifacts stay
|
||||
fetchable.
|
||||
Reference in New Issue
Block a user