feat(imagegen): reference-image editing for instruction-edit models
Gadfly review (reusable) / review (pull_request) Successful in 4m24s
Adversarial Review (Gadfly) / review (pull_request) Successful in 4m24s
CI / Tidy (pull_request) Successful in 9m40s
CI / Build & Test (pull_request) Successful in 11m17s

FLUX.1 Kontext and Qwen-Image-Edit are a different kind of edit from img2img
and reach sd-server by a different path, and nothing in imagegen could
express it: EditRequest only had Init, which is noised and denoised back
under the prompt.

Measured against FLUX.1-Kontext on the netherstorm host 2026-07-30, on a
synthetic scene with a red rectangle, a blue rectangle and a flat background,
prompted "change the blue rectangle on the right to bright green, keep
everything else exactly the same":

  via init_images (the only path that existed)
      right rect (60,60,200) -> (47,82,228)   still blue, instruction ignored
      left rect  (200,60,60) -> (229,43,50)   drifted
      background (150,200,240) -> (154,211,229) drifted

  via extra_images (this change)
      right rect (60,60,200) -> (70,254,4)    green, as asked
      left rect  (200,60,60) -> (204,57,57)   intact
      background (150,200,240) -> (151,202,247) intact

No mask, no strength, no compositing — the model is handed the picture as
conditioning and the prompt as an instruction about it.

EditRequest.RefImages selects the path; when set, Init/Mask/Strength are
ignored rather than rejected, so a caller handing the same request to
whichever model is configured gets the better result on a Kontext-class model
instead of an error. The provider posts /sdapi/v1/txt2img with extra_images
(sd-server reads that field on both routes into gen_params.ref_images, where
the CLI's -r/--ref-image also lands); there is no init latent to denoise, so
sending one would only add noise to a pipeline that does not want any.

An all-empty reference set is refused: it would otherwise degrade into a
plain txt2img and render the prompt from scratch, which is not the request.
This commit is contained in:
2026-07-30 21:21:35 -04:00
parent a941f5ff4a
commit 2c70d32fd4
3 changed files with 154 additions and 2 deletions
+31 -1
View File
@@ -9,9 +9,32 @@ type EditRequest struct {
// Prompt is the text description of the desired edit.
Prompt string
// Init is the initial image the edit starts from. Required.
// Init is the initial image the edit starts from. Required, EXCEPT when
// RefImages is set — see there.
Init Image
// RefImages carries reference images for INSTRUCTION-EDIT models
// (FLUX.1 Kontext, Qwen-Image-Edit), which are a different kind of edit
// from img2img and reach the model by a different path.
//
// img2img noises Init and denoises it back under the prompt: the prompt
// describes the DESIRED IMAGE, and how much of the original survives is a
// function of Strength. An instruction-edit model instead takes the
// picture as conditioning and the prompt as an INSTRUCTION about it
// ("change the sign to read OPEN"), leaving everything it was not asked
// to touch bit-for-bit intact — no mask, no strength, no compositing.
//
// Sending one of these models an Init instead of a RefImage does not
// degrade gracefully, it silently does the wrong thing: measured against
// FLUX.1-Kontext on 2026-07-30, "change the blue rectangle to green" via
// init_images left the rectangle blue and drifted every other region,
// while the same prompt via a reference image turned it green and left
// the rest of the frame numerically unchanged.
//
// When RefImages is non-empty, Init/Mask/Strength are IGNORED: they
// describe a pipeline this model does not run.
RefImages []Image
// Mask restricts the edit to a region (inpainting): a single-channel or
// RGB image the same size as Init where WHITE pixels are repainted and
// BLACK pixels are kept. Empty = whole-image edit. Backends without mask
@@ -54,6 +77,13 @@ type EditOption func(*EditRequest)
// WithEditMask restricts the edit to a region (white = repaint, black = keep).
func WithEditMask(m Image) EditOption { return func(r *EditRequest) { r.Mask = m } }
// WithEditRefImages supplies reference images for an instruction-edit model
// (Kontext / Qwen-Image-Edit). See EditRequest.RefImages — this selects a
// different edit path, not a variation on img2img.
func WithEditRefImages(imgs ...Image) EditOption {
return func(r *EditRequest) { r.RefImages = imgs }
}
// WithEditStrength sets the denoising strength in [0,1].
func WithEditStrength(s float64) EditOption { return func(r *EditRequest) { r.Strength = &s } }