Measured against the instruction-edit models on 2026-07-31: asking a diffusion
model to put a SPECIFIC person's face into a photo does not work by any route.
qwen-image-edit returns the picture essentially unchanged whether asked by
name, by attribute, or by supplying the portrait as a second reference image;
flux-kontext replaces the face with a different generic person. Identity
transfer is a detect/align/blend pipeline, not a better prompt, so it gets its
own interface rather than more Edit options.
imagegen.FaceSwapper is optional and type-asserted, like Editor — a provider
that cannot do this must not have Edit quietly stand in for it.
ListFaces is part of the interface, not a convenience: a caller asked to
change "the man on the right" needs a stable way to NAME one face, and pixel
boxes let it check its own choice. The llamaswap shim orders faces left to
right for exactly that reason (insightface's own order is score-ranked and
unstable between near-identical images), and a malformed box is a protocol
error rather than a zero-filled struct, because a wrong box aims the swap at
the wrong person.
The provider is the first here to POST more than one file, so buildMultipart
gained buildMultipartFiles and now delegates to it — one writer loop, so the
two cannot drift in how they escape names or terminate the body.
index and all are mutually exclusive ON THE WIRE: the shim ignores index under
all=true, and sending both would imply a precedence the caller cannot see.
A JSON body is refused rather than returned as image bytes — the shim answers
JSON on a semantic miss (no face in the source), and handing that back as a
picture would report success while delivering a file that is not one.
FLUX.1 Kontext and Qwen-Image-Edit are a different kind of edit from img2img
and reach sd-server by a different path, and nothing in imagegen could
express it: EditRequest only had Init, which is noised and denoised back
under the prompt.
Measured against FLUX.1-Kontext on the netherstorm host 2026-07-30, on a
synthetic scene with a red rectangle, a blue rectangle and a flat background,
prompted "change the blue rectangle on the right to bright green, keep
everything else exactly the same":
via init_images (the only path that existed)
right rect (60,60,200) -> (47,82,228) still blue, instruction ignored
left rect (200,60,60) -> (229,43,50) drifted
background (150,200,240) -> (154,211,229) drifted
via extra_images (this change)
right rect (60,60,200) -> (70,254,4) green, as asked
left rect (200,60,60) -> (204,57,57) intact
background (150,200,240) -> (151,202,247) intact
No mask, no strength, no compositing — the model is handed the picture as
conditioning and the prompt as an instruction about it.
EditRequest.RefImages selects the path; when set, Init/Mask/Strength are
ignored rather than rejected, so a caller handing the same request to
whichever model is configured gets the better result on a Kontext-class model
instead of an error. The provider posts /sdapi/v1/txt2img with extra_images
(sd-server reads that field on both routes into gen_params.ref_images, where
the CLI's -r/--ref-image also lands); there is no init latent to denoise, so
sending one would only add noise to a pipeline that does not want any.
An all-empty reference set is refused: it would otherwise degrade into a
plain txt2img and render the prompt from scratch, which is not the request.
- SubmitChain rejects NaN/±Inf segment seconds with ErrUnsupported
(previously an obscure json.Marshal error; NaN fails every comparison
and +Inf passed the >= 0 check).
- ChainStatus skips segment entries with no usable id — JSON null
(which no-op-unmarshals into a string, previously appending ""),
empty strings, and id-less objects; the unfiltered list survives in
Raw. ChainJob.SegmentIDs doc now also says ChainSegmentResult takes
the segment index, not an id string.
- jobPath rejects '%' in job ids — %2F/%2E%2E percent-escapes decode
back into path structure server-side, bypassing the literal check on
this upstream-echoed value.
- singleVideoResult moves to video.go next to videoMIME, and the two
remaining hand-rolled copies of the video-result validation
(videoModel.Generate, Interpolate) now use it — one validation, one
message shape.
- videogen.LipSyncer renamed to videogen.Lipsyncer for consistency with
the rest of the surface's Lipsync* naming (LipsyncProvider,
LipsyncModel, LipsyncRequest); not yet consumed downstream, so the
rename is free now and never again.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WWCQcYStWXBYUy5sZWnbLT
- speakWithReference now validates the response IS audio via the same
audioResultMIME sniff the sfx/enhance surfaces use — a 2xx JSON soft
error or HTML proxy page was previously wrapped up as audio/wav bytes.
- audioResultMIME moves to audio.go (next to speechMIME; it was defined
in sfx.go but shared by enhance/clone) and learns the Ogg container
normalization (application/ogg -> audio/ogg).
- Stems zip unpack gains entry-count (16) and total-decompressed (1GB)
caps on top of the existing per-entry cap — the per-entry bound alone
still let a many-entry bomb multiply up.
- sanitizeFilename drops NUL and both path separators too, so upload
metadata can never smuggle directory structure to a file-writing shim.
- New maxAudioResponseBytes (256MB) for bodies that ARE one audio clip
(clone, enhance, sfx): a long WAV legitimately passes the 64MB JSON
cap.
- speakWithReference local renamed path -> upPath (naming parity with
stems/enhance).
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WWCQcYStWXBYUy5sZWnbLT
- Segment: reject NaN thresholds (NaN fails every comparison, so it
passed the [0,1] range check and reached the shim as the literal
string "NaN"); ±Inf were already caught by the range comparisons,
now covered by tests too.
- upstreamPath: reject '%' in model ids — %2F/%2E%2E percent-escapes
decode back into path structure server-side, bypassing the literal
/?#/.. rejection. Ids never legitimately contain '%'.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WWCQcYStWXBYUy5sZWnbLT
- audio.StemSeparator/StemSeparationProvider: Demucs zip transport via
POST /upstream/<id>/v1/stems (Mode two -> two_stems=vocals; model +
format fields); bounded zip unpack, entry name -> stem, ext -> MIME.
- SFXModel reuses musicgen against the sync /upstream/<id>/v1/sfx route
(JSON prompt/seconds/steps/cfg_scale/seed -> WAV); musicgen.Request
gains CFGScale.
- audio.SpeechEnhancer/SpeechEnhancementProvider:
POST /upstream/<id>/v1/enhance -> WAV (result reuses SpeechResult).
- SpeechRequest.ReferenceAudio/ReferenceMIME (+WithReferenceAudio):
llamaswap switches to the chatterbox clone route
POST /upstream/<id>/v1/audio/speech/upload (input + voice_file),
wav MIME fallback.
- TranscriptionRequest.Translate (+WithTranslate): translate=true form
field, language=auto forced when no explicit hint (whisper.cpp default
en would skip translation).
- httptest contract tests (zip unpack, clone-route switch, translate +
auto-language injection); ADR-0024 (index row deferred — MJ-A backfills
the ADR index table and parallel edits would conflict).
Co-Authored-By: Claude Fable 5 <[email protected]>
Round 2 from live smokes on netherstorm (2026-07-14):
- ACE-Step result blob is an ARRAY of objects and carries RAW control
characters inside string values (literal newlines) — strict JSON
rejected it. parseMusicResult sanitizes control chars (only legal
inside string values in the double-encoded blob) and accepts array or
object shapes. Regression test uses the live payload shape.
- Hunyuan3D GenerationRequest has NO output-format field (the documented
type param is fiction) — it always returns GLB. Results are now
labelled by sniffed magic bytes, never by the requested format.
- NEW meshgen.Converter/ConverterProvider optional surface + llamaswap
impl over the mediautils shim POST /v1/convert_mesh — the STL hop for
the printer pipeline.
Co-Authored-By: Claude Fable 5 <[email protected]>
- pollResult tolerates up to 5 CONSECUTIVE bad polls (transport blip,
unparseable payload, task momentarily absent) instead of killing a
multi-minute exclusive-GPU job on the first hiccup; only status=2, a
failure run, or ctx deadline aborts
- server-supplied result.File must be server-relative; combined with the
upstreamPath dot-dot/scheme rejection this stops a hostile upstream
from steering the follow-up GET at other proxy endpoints (test:
../../api/models/unload refused)
- WithSteps(<=0) rejected; embed responses repeating an index rejected;
musicFormatMIME now wraps speechMIME (one format table, wav32
normalized); poll interval is a test-shrinkable var (CI no longer
burns 2s+ per music test); parens + comments per review
Co-Authored-By: Claude Fable 5 <[email protected]>
- NEW musicgen leaf package: blocking Generate over ACE-Step's async job
queue (release_task -> poll query_result -> fetch file, all via
/upstream); tolerant envelope parsing, double-encoded result handled
- NEW embeddings leaf package: EmbedModel + RerankModel as separate mints
(two server instances on the host, llama.cpp #20085); InstructedQuery
helper for Qwen3-style query/document asymmetry
- provider/llamaswap: /v1/embeddings + /v1/rerank clients with strict
validation (index-ordered vectors, count mismatch and out-of-range
index are hard errors; rerank sorted descending, minimal parser)
Co-Authored-By: Claude Fable 5 <[email protected]>
- upstreamPath rejects '..' in model ids AND in the rest path — the rest
can embed SERVER-SUPPLIED components (ACE-Step result file URLs), so
dot-dot/scheme smuggling toward other proxy endpoints is refused
- singleImageResult requires positive image evidence (sniffed magic OR
declared image/*): an empty-Content-Type error page can no longer pass
as 'the image' via sniffImageMIME's PNG-default labelling
- upscale/background responses get a dedicated 256MB cap (the 64MB cap
is JSON-sized; a 4x PNG legitimately exceeds it)
- mesh JSON-detection widened (512-byte whitespace-tolerant peek + reject
declared application/json)
- Transcribe now reuses buildMultipart; transcriptionFilename takes
(filename, mime) so diarize shares it without a fake request struct;
truncateForError stops shadowing builtin cap; OnlyMask doc de-ambiguated
Co-Authored-By: Claude Fable 5 <[email protected]>
maxVideoResponseBytes (512MB) replaces the shared 64MB JSON cap on the
/v1/videos/sync read path (doRaw now takes the cap per call) — 3/6
models flagged that a legitimate long/high-bitrate clip would be
discarded after minutes of GPU work. Plus a stale stable-diffusion
comment in initImageFilename and a test-handler early return.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01AXQxVhXBw8PwFAtsVrXSmj
- videoMIME no longer hard-falls-back to video/mp4: a 2xx body that is
neither declared nor sniffable as video (JSON job envelope, HTML error
page) is now an APIError instead of a 'successful' corrupt clip.
- Resolution rides the wire as width/height AND the OpenAI-style size
string, so either upstream convention honors an explicit request.
- writeFormFields + mimeFromContentType shared helpers replace the
copied multipart loop (audio.go/video.go) and Content-Type branch.
- ADR-0019 indexed in docs/adr/README.md; README gains the videogen
section + support-matrix mention (docs-parity rule).
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01AXQxVhXBw8PwFAtsVrXSmj
New videogen/ contract package (ADR-0019): Request/Result/Model/Provider
with the imagegen conventions. Text-to-video and image-to-video are one
surface (Request.InitImage, nil = t2v) since hybrid checkpoints like
Wan 2.2 TI2V serve both from one model; Result carries a single clip.
provider/llamaswap gains VideoModel(id) targeting the blocking
POST {base}/v1/videos/sync (multipart, model-routed by the fork's new
video routes): vLLM-Omni parameter names, OpenAI-style input_reference
file part, optional fields stay off the wire so per-model launch-flag
defaults apply. CLAUDE.md package map picks up audio/ (missed in #12)
and videogen/.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01AXQxVhXBw8PwFAtsVrXSmj
- Transcribe: sanitize the caller-supplied multipart filename (CR/LF
would inject Content-Disposition headers; upload metadata is
untrusted), always send the required model/response_format fields,
parse MIME parameters before extension matching, and give audio/opus
its own .opus extension.
- doRaw: a response larger than maxResponseBytes is now an error, not a
silent truncation.
- Shared plumbing: requireBaseURL() + newRequest() helpers replace the
7x-duplicated guard/error string and the triplicated request
building across doJSON/doRaw/Health.
- Health: non-2xx now returns *llm.APIError (package convention,
programmatically distinguishable from transport failure) instead of
a one-off unexported error type.
- Speak: reject negative Speed; speechMIME no longer accepts video/*
Content-Types.
- image.go: Generate/Edit share one sdWire validate+map helper.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01AXQxVhXBw8PwFAtsVrXSmj
- New leaf package `audio` (ADR-0017): SpeechModel/SpeechProvider and
TranscriptionModel/TranscriptionProvider with imagegen conventions
(zero value = backend default, functional options + Apply, bytes
in/out, never URLs). Root re-exports added.
- imagegen.Editor (ADR-0018): optional image-to-image interface —
EditRequest carries the generation knobs plus Init image and
denoising Strength; separate interface so existing Models keep
compiling.
- provider/llamaswap implements all of it: POST /v1/audio/speech (JSON,
raw-audio response, MIME from Content-Type with format fallback),
POST /v1/audio/transcriptions (multipart, response_format=json),
ListVoices (GET /v1/audio/voices?model=, tolerant of string-list and
object-list shapes), POST /sdapi/v1/img2img (txt2img wire +
init_images/denoising_strength, shared image decode), and Health(ctx)
(GET /health) — a cheap liveness probe for often-offline hosts.
- Hermetic httptest coverage for every new wire shape and validation
path; README sections + support-matrix footnote updated in the same
commit (also corrects the stale /v1/images/generations claim — the
image path has been SDAPI since the seed fix).
First consumer: mort's llamaswap media tool cluster (status / image /
TTS / STT agent tools against the netherstorm host).
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01AXQxVhXBw8PwFAtsVrXSmj
The OpenAI /v1/images/generations endpoint ignores `seed` on our
stable-diffusion.cpp build — every render of a given prompt comes back
byte-identical, so a drawbot batch of N collapsed to one image. Switch the
image provider to sd-server's A1111 /sdapi/v1/txt2img endpoint, which honors
`seed` (verified live: distinct seeds -> distinct images on SDXL and
Qwen-Image). Size is split into width/height; llama-swap still routes by the
`model` field. Tests + ADR-0016 updated.
Add Steps, CFGScale, NegativePrompt, Sampler, Seed to imagegen.Request
(pointer/empty = leave the backend's per-model default), with mirror
options, and forward them in the llamaswap wire payload as the
stable-diffusion.cpp fields (steps/cfg_scale/negative_prompt/
sample_method/seed). Unset fields are omitted so sd-server keeps its
baked defaults.
Lets callers (e.g. mort drawbots) override only what they explicitly set.
- Unload: reject model ids containing path separators (/?#) so a model name
can't redirect the request to another endpoint; ":" (common in ids) stays
verbatim.
- doJSON: take a model arg so image/management HTTP errors carry the target id
(was always ""); add a base-URL guard so management methods fail clearly
instead of building a bare-path request; cap the success-path JSON decode with
io.LimitReader (64 MiB) and drain the body when out is nil for conn reuse.
- image: reject negative Request.N before sending.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Add provider/llamaswap, a tailored provider for llama-swap (the model-swapping
proxy over llama.cpp / stable-diffusion.cpp). Its chat path delegates to
provider/openai at {base}/v1 — no duplicated wire client (ADR-0007) — with
legacy max_tokens, a Bearer no-key placeholder for keyless local instances, and
a timeout-free client so cold model swaps rely on context deadlines. The
"tailored" surface is concrete management methods (ListModels / Running /
Unload) that don't belong on the canonical llm.Provider interface. The
llama-swap:// DSN scheme builds an http base URL (local-first); a no-URL
built-in errors clearly on use, mirroring foreman.
Add imagegen, a new canonical text-to-image interface separate from llm
(Request/Result/Model/Provider; Image = llm.ImagePart so generated images feed
straight back into chat). First backend is llama-swap via OpenAI
/v1/images/generations (b64_json, bytes-only). Re-exported from the root. v1 is
txt2img only.
Hermetic httptest coverage for chat delegation, management endpoints, image
decode, and scheme wiring. ADR-0015 + ADR-0016, README support matrix +
image-gen section, CLAUDE.md package map, and progress.md updated in the same
commit.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
steve
