feat(imagegen): reference-image editing for instruction-edit models
FLUX.1 Kontext and Qwen-Image-Edit are a different kind of edit from img2img
and reach sd-server by a different path, and nothing in imagegen could
express it: EditRequest only had Init, which is noised and denoised back
under the prompt.
Measured against FLUX.1-Kontext on the netherstorm host 2026-07-30, on a
synthetic scene with a red rectangle, a blue rectangle and a flat background,
prompted "change the blue rectangle on the right to bright green, keep
everything else exactly the same":
via init_images (the only path that existed)
right rect (60,60,200) -> (47,82,228) still blue, instruction ignored
left rect (200,60,60) -> (229,43,50) drifted
background (150,200,240) -> (154,211,229) drifted
via extra_images (this change)
right rect (60,60,200) -> (70,254,4) green, as asked
left rect (200,60,60) -> (204,57,57) intact
background (150,200,240) -> (151,202,247) intact
No mask, no strength, no compositing — the model is handed the picture as
conditioning and the prompt as an instruction about it.
EditRequest.RefImages selects the path; when set, Init/Mask/Strength are
ignored rather than rejected, so a caller handing the same request to
whichever model is configured gets the better result on a Kontext-class model
instead of an error. The provider posts /sdapi/v1/txt2img with extra_images
(sd-server reads that field on both routes into gen_params.ref_images, where
the CLI's -r/--ref-image also lands); there is no init latent to denoise, so
sending one would only add noise to a pipeline that does not want any.
An all-empty reference set is refused: it would otherwise degrade into a
plain txt2img and render the prompt from scratch, which is not the request.
This commit is contained in:
+31
-1
@@ -9,9 +9,32 @@ type EditRequest struct {
|
||||
// Prompt is the text description of the desired edit.
|
||||
Prompt string
|
||||
|
||||
// Init is the initial image the edit starts from. Required.
|
||||
// Init is the initial image the edit starts from. Required, EXCEPT when
|
||||
// RefImages is set — see there.
|
||||
Init Image
|
||||
|
||||
// RefImages carries reference images for INSTRUCTION-EDIT models
|
||||
// (FLUX.1 Kontext, Qwen-Image-Edit), which are a different kind of edit
|
||||
// from img2img and reach the model by a different path.
|
||||
//
|
||||
// img2img noises Init and denoises it back under the prompt: the prompt
|
||||
// describes the DESIRED IMAGE, and how much of the original survives is a
|
||||
// function of Strength. An instruction-edit model instead takes the
|
||||
// picture as conditioning and the prompt as an INSTRUCTION about it
|
||||
// ("change the sign to read OPEN"), leaving everything it was not asked
|
||||
// to touch bit-for-bit intact — no mask, no strength, no compositing.
|
||||
//
|
||||
// Sending one of these models an Init instead of a RefImage does not
|
||||
// degrade gracefully, it silently does the wrong thing: measured against
|
||||
// FLUX.1-Kontext on 2026-07-30, "change the blue rectangle to green" via
|
||||
// init_images left the rectangle blue and drifted every other region,
|
||||
// while the same prompt via a reference image turned it green and left
|
||||
// the rest of the frame numerically unchanged.
|
||||
//
|
||||
// When RefImages is non-empty, Init/Mask/Strength are IGNORED: they
|
||||
// describe a pipeline this model does not run.
|
||||
RefImages []Image
|
||||
|
||||
// Mask restricts the edit to a region (inpainting): a single-channel or
|
||||
// RGB image the same size as Init where WHITE pixels are repainted and
|
||||
// BLACK pixels are kept. Empty = whole-image edit. Backends without mask
|
||||
@@ -54,6 +77,13 @@ type EditOption func(*EditRequest)
|
||||
// WithEditMask restricts the edit to a region (white = repaint, black = keep).
|
||||
func WithEditMask(m Image) EditOption { return func(r *EditRequest) { r.Mask = m } }
|
||||
|
||||
// WithEditRefImages supplies reference images for an instruction-edit model
|
||||
// (Kontext / Qwen-Image-Edit). See EditRequest.RefImages — this selects a
|
||||
// different edit path, not a variation on img2img.
|
||||
func WithEditRefImages(imgs ...Image) EditOption {
|
||||
return func(r *EditRequest) { r.RefImages = imgs }
|
||||
}
|
||||
|
||||
// WithEditStrength sets the denoising strength in [0,1].
|
||||
func WithEditStrength(s float64) EditOption { return func(r *EditRequest) { r.Strength = &s } }
|
||||
|
||||
|
||||
Reference in New Issue
Block a user