steveandClaude Fable 5
5f175ecf82
- videogen.LipSyncer/LipsyncProvider: SadTalker talking heads via
POST /upstream/<id>/v1/talking_head (multipart image+audio parts,
still/enhance/preprocess flags) -> mp4.
- videogen.VideoBackgroundRemover/VideoBackgroundRemovalProvider:
POST /upstream/<id>/v1/video/matte (output greenscreen_mp4|alpha_webm).
- videogen.VideoUpscaler/VideoUpscaleProvider:
POST /upstream/<id>/v1/video/upscale (scale 2|4).
- videogen.Chainer/ChainerProvider: async long-video chain-job client —
SubmitChain (JSON POST /v1/video/chain, init_image_b64), ChainStatus
(GET /v1/jobs/{id}, tolerant segment-id decode), ChainResult,
ChainSegmentResult (partial delivery after mid-chain failure); hostile
job-id path rejection.
- Shared singleVideoResult validation (positive video evidence) + a
videoInputFilename hint helper; httptest contract tests per surface;
ADR-0025 (index row deferred — MJ-A backfills the ADR index table and
parallel edits would conflict).
Co-Authored-By: Claude Fable 5 <[email protected]>
68 lines
3.6 KiB
Markdown
68 lines
3.6 KiB
Markdown
# ADR-0025: Wave-3 video surfaces (lipsync, video matte, video upscale, chain jobs)
|
|
|
|
Status: Accepted (2026-07-16)
|
|
|
|
## Context
|
|
|
|
The llama-swap host is gaining wave-3 video capabilities (spec: mort
|
|
docs/specs/2026-07-16-llamaswap-wave3.md): SadTalker talking heads, Robust
|
|
Video Matting and per-frame Real-ESRGAN on the mediautils shim, and a
|
|
videoutils orchestrator that generates LONG videos as a chain of i2v
|
|
segments (generate → extract last frame → continue → concat → optional RIFE
|
|
smoothing), exposed as an async job API following the ACE-Step precedent.
|
|
|
|
## Decision
|
|
|
|
1. **Three new optional `videogen` interfaces**, ADR-0016→0024 conventions:
|
|
- `videogen.LipSyncer` / `LipsyncProvider` — `LipsyncRequest{Image,
|
|
Audio, AudioMIME, AudioFilename, Still, Enhance, Preprocess}` →
|
|
`POST /upstream/<id>/v1/talking_head` (multipart `image` + `audio`
|
|
file parts + optional `still`/`enhance`/`preprocess` fields) → mp4.
|
|
Sync and minutes-slow (Hunyuan precedent); context deadline is the
|
|
budget.
|
|
- `videogen.VideoBackgroundRemover` / `VideoBackgroundRemovalProvider` —
|
|
`VideoBackgroundRemovalRequest{Video, MIME, Filename, Output}` →
|
|
`POST /upstream/<id>/v1/video/matte`. `Output` ∈
|
|
`greenscreen_mp4` (universally playable) | `alpha_webm` (true
|
|
transparency); "" = backend default.
|
|
- `videogen.VideoUpscaler` / `VideoUpscaleProvider` —
|
|
`VideoUpscaleRequest{Video, MIME, Filename, Scale}` (2 or 4) →
|
|
`POST /upstream/<id>/v1/video/upscale` → mp4.
|
|
2. **The chain client is deliberately ASYNC** (`videogen.Chainer` /
|
|
`ChainerProvider`), unlike musicgen's blocking Generate (ADR-0021): a
|
|
chain holds the GPU through multiple model swaps for many minutes, and
|
|
the caller must be able to poll progress AND fetch completed segments
|
|
after a mid-chain failure — partial delivery is mandatory (never discard
|
|
multi-minute GPU output), which a blocking one-call contract cannot
|
|
express.
|
|
- `SubmitChain(ctx, ChainRequest{Segments[{Prompt,Seconds}], InitImage,
|
|
SmoothJoins, Size}) (jobID, error)` — JSON
|
|
`POST /upstream/<id>/v1/video/chain`, init image as base64
|
|
`init_image_b64` (JSON submit, not multipart, per the pinned host
|
|
contract).
|
|
- `ChainStatus(ctx, jobID) (*ChainJob{Status, Segment, Total,
|
|
SegmentIDs, Raw})` — `GET /v1/jobs/{id}`; polling doubles as a
|
|
liveness signal for the shim's idle TTL. Segment entries are tolerated
|
|
as strings or `{id|segment_id}` objects.
|
|
- `ChainResult(ctx, jobID)` / `ChainSegmentResult(ctx, jobID, n)` —
|
|
`GET /v1/jobs/{id}/result` and `/v1/jobs/{id}/segments/{n}`.
|
|
- Job ids are echoed server input: job paths reject ids carrying path
|
|
structure (`/?#`, `..`), the upstreamPath smuggling rule.
|
|
3. **Binary success bodies are validated before wrapping** (ADR-0020 rule):
|
|
all four clip-returning calls go through a shared `singleVideoResult`
|
|
(positive evidence of video-ness — declared video/* Content-Type or
|
|
sniffed mp4/webm magic — so a JSON status page or proxy error can never
|
|
become "the clip").
|
|
|
|
## Consequences
|
|
|
|
- videogen grows from two surfaces (Model, Interpolator) to six; the
|
|
chain client is the package's first async surface — the job-API shape
|
|
deferred in ADR-0019/0021 now exists where the workload actually
|
|
demands it.
|
|
- The wire shapes are pinned by the netherstorm videoutils/mediautils/
|
|
SadTalker image builds; host smoke tests are the drift defence.
|
|
- mort's long-video tool owns the poll loop, timeout budget, and
|
|
partial-result envelope; majordomo only guarantees the artifacts stay
|
|
fetchable.
|