docs(videogen): gadfly — README FL2V section, and stop pointing at a note that does not exist
- README documented only t2v/i2v. Now a table of the four keyframe
combinations, plus the undetectable-support caveat, which is the one thing a
caller cannot work out for itself.
- The LastImage doc comment said "see the note on LastImage support in
provider/llamaswap" — there was no such note. A pointer to something that
does not exist is worse than no pointer; the comment is now self-contained.
- Generate's doc described only input_reference; it now names
input_reference_last and explains why an unsupporting backend returns a clip
rather than an error.
The 2/4 finding (writeImagePart reusing the "frame" base for both parts) was
already fixed in dbc9689 — from the receiving end, where the consequence is
concrete rather than stylistic: ComfyUI stages uploads by FILENAME with
overwrite=true, so a shared name means the second clobbers the first and both
keyframes resolve to one image.
Not taken: initImageFilename's name is no longer misleading (writeImagePart
stopped calling it), and it is still used by lipsync.go so it is not dead.
The empty-LastImage test stays standalone — it mirrors the existing standalone
empty-InitImage coverage rather than a table this file does not have.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01PLjgrxvHjm1sJgUu9zBPH9
This commit is contained in:
@@ -272,17 +272,30 @@ tr, err := tm.Transcribe(ctx, audio.TranscriptionRequest{
|
|||||||
voices, err := ls.ListVoices(ctx, "kokoro") // []string of voice ids
|
voices, err := ls.ListVoices(ctx, "kokoro") // []string of voice ids
|
||||||
```
|
```
|
||||||
|
|
||||||
## Video: text-to-video + image-to-video
|
## Video: text-to-video, image-to-video, first-last-frame
|
||||||
|
|
||||||
Video generation lives in the `videogen` package (ADR-0019), mirroring
|
Video generation lives in the `videogen` package (ADR-0019), mirroring
|
||||||
imagegen/audio: one small `Model` contract, zero values mean backend
|
imagegen/audio: one small `Model` contract, zero values mean backend
|
||||||
defaults, bytes in/out. Text-to-video and image-to-video are one surface —
|
defaults, bytes in/out. All modes are one surface, selected by which
|
||||||
a nil `InitImage` is a pure text prompt; setting it conditions generation
|
keyframes are set rather than by a mode flag:
|
||||||
on that frame (hybrid checkpoints like Wan 2.2 TI2V serve both). First
|
|
||||||
backend: llama-swap (blocking `/v1/videos/sync`, vLLM-Omni style — the
|
| `InitImage` | `LastImage` | mode |
|
||||||
|
|---|---|---|
|
||||||
|
| nil | nil | text-to-video |
|
||||||
|
| set | nil | image-to-video (hybrid checkpoints like Wan 2.2 TI2V serve both) |
|
||||||
|
| set | set | first-last-frame — both ends pinned |
|
||||||
|
| nil | set | pin the destination, model invents the approach |
|
||||||
|
|
||||||
|
First backend: llama-swap (blocking `/v1/videos/sync`, vLLM-Omni style — the
|
||||||
response body is the encoded clip, so `Result` carries a single `Video`).
|
response body is the encoded clip, so `Result` carries a single `Video`).
|
||||||
Generation runs for minutes; bound the call with a context deadline.
|
Generation runs for minutes; bound the call with a context deadline.
|
||||||
|
|
||||||
|
**`LastImage` support is per-model and cannot be detected.** A backend that
|
||||||
|
does not understand a trailing keyframe ignores the part and returns an
|
||||||
|
ordinary clip — indistinguishable from success. There is no capability bit,
|
||||||
|
because the contract has no way to learn one, so a caller depending on the
|
||||||
|
pin must establish support out of band.
|
||||||
|
|
||||||
```go
|
```go
|
||||||
vm, _ := ls.VideoModel("videogen-wan22-5b")
|
vm, _ := ls.VideoModel("videogen-wan22-5b")
|
||||||
res, err := vm.Generate(ctx, videogen.Request{Prompt: "a cat surfing"},
|
res, err := vm.Generate(ctx, videogen.Request{Prompt: "a cat surfing"},
|
||||||
|
|||||||
@@ -38,10 +38,13 @@ type videoModel struct {
|
|||||||
// bound the call with a context deadline.
|
// bound the call with a context deadline.
|
||||||
//
|
//
|
||||||
// Parameter names follow vLLM-Omni's videos API (num_frames, fps,
|
// Parameter names follow vLLM-Omni's videos API (num_frames, fps,
|
||||||
// num_inference_steps, guidance_scale); the conditioning frame is sent as an
|
// num_inference_steps, guidance_scale); the leading conditioning frame is sent
|
||||||
// `input_reference` file part, following OpenAI's videos API. Upstreams
|
// as an `input_reference` file part, following OpenAI's videos API, and a
|
||||||
|
// trailing keyframe (Request.LastImage) as `input_reference_last`. Upstreams
|
||||||
// ignore fields they don't understand, and optional fields stay off the wire
|
// ignore fields they don't understand, and optional fields stay off the wire
|
||||||
// entirely so the model's own defaults apply.
|
// entirely so the model's own defaults apply — which is also why a backend
|
||||||
|
// without first-last-frame support returns an ordinary clip here rather than
|
||||||
|
// an error.
|
||||||
func (m *videoModel) Generate(ctx context.Context, req videogen.Request, opts ...videogen.Option) (*videogen.Result, error) {
|
func (m *videoModel) Generate(ctx context.Context, req videogen.Request, opts ...videogen.Option) (*videogen.Result, error) {
|
||||||
req = req.Apply(opts...)
|
req = req.Apply(opts...)
|
||||||
if strings.TrimSpace(req.Prompt) == "" {
|
if strings.TrimSpace(req.Prompt) == "" {
|
||||||
|
|||||||
@@ -59,10 +59,11 @@ type Request struct {
|
|||||||
//
|
//
|
||||||
// Support is per-model and NOT advertised anywhere in this contract: a
|
// Support is per-model and NOT advertised anywhere in this contract: a
|
||||||
// backend that does not understand a trailing keyframe ignores it and
|
// backend that does not understand a trailing keyframe ignores it and
|
||||||
// returns an ordinary clip, which is indistinguishable from success. A
|
// returns an ordinary clip, which is indistinguishable from success.
|
||||||
// caller that needs to know whether the pin took effect must establish
|
// There is no capability bit to consult, because the contract has no way
|
||||||
// that out of band — see the note on LastImage support in
|
// to learn one. A caller that needs to know whether the pin actually took
|
||||||
// provider/llamaswap.
|
// effect must establish that out of band — by configuration it controls,
|
||||||
|
// not by inspecting the result.
|
||||||
LastImage *Image
|
LastImage *Image
|
||||||
|
|
||||||
// Size is the requested resolution, e.g. "1280x704"; "" = backend default.
|
// Size is the requested resolution, e.g. "1280x704"; "" = backend default.
|
||||||
|
|||||||
Reference in New Issue
Block a user