Author SHA1 Message Date
steve f837115a55 Merge pull request 'feat(qwen): Alibaba Qwen built-in over Model Studio's OpenAI-compatible mode' (#27)
CI / Tidy (push) Successful in 9m24s
CI / Build & Test (push) Successful in 11m10s
2026-08-12 21:03:34 +00:00
steveandClaude Opus 5 f8ced9c629 docs(progress): describe the shape this PR actually landed in
CI / Tidy (pull_request) Successful in 9m22s
CI / Build & Test (pull_request) Successful in 10m28s
The progress entry was written before four review rounds reshaped the change:
it credited openaiCompatScheme alone, listed the tests as six per-provider
cases, and mentioned a captureRT detail that has since moved. Rewritten to
match what merges — registerOpenAICompatBuiltin owning both halves,
envKeyForProvider as the single LLM_<NAME> definition, and the shared table
that every OpenAI-compat built-in is now checked against.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-12 16:57:33 -04:00
steveandClaude Opus 5 0760cf96d4 docs: gadfly round 4 — two doc-comment fixes
CI / Tidy (pull_request) Successful in 9m35s
CI / Build & Test (pull_request) Successful in 10m17s
DSN.Scheme's list named kimi but not qwen — the same sibling drift this PR
keeps finding, in a doc comment this time (I updated the README's scheme list
and not this one). Added qwen, and llama-swap/llama-swaps while there, since
they were missing too.

envKeyForProvider's example ran backwards: "LLM_M1 → m1" describes registry
naming, not what the function does. Now reads name → variable in one
direction.

Not taking the third: the const block's alignment is gofmt's own output
(gofmt -l is empty), and the uneven padding is forced by the doc comments that
split the block into alignment groups. glm-5.2 reached that same conclusion in
round 2 before flagging it here.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-12 16:53:05 -04:00
steveandClaude Opus 5 8670ed22be refactor: gadfly round 3 — one table owns the OpenAI-compat contract
CI / Tidy (pull_request) Successful in 9m21s
CI / Build & Test (pull_request) Successful in 10m23s
Three findings, and the first two are the same recurring shape.

envKeyForProvider (env.go) is now the single definition of the LLM_<NAME>
form. It lived in two places — lazy resolution in registry.go and the
missing-key hint in openaiCompatScheme — with a comment on the second asserting
it matched the first. A comment is not enforcement: if either had drifted, a
keyless DSN target would have named a variable that does nothing, and nothing
would have failed.

The kimi and qwen test files had become near-identical, which is round 1's
finding at the level above it: I deduped the fixtures, then left two parallel
suites asserting the same four things. They are now ONE table in
builtin_openaicompat_test.go — endpoint + credential, missing key fails closed
naming its own variable and never reaching the network, the name:// DSN
reaching another host, and a keyless DSN naming LLM_<NAME> instead of the
built-in's key. Adding an OpenAI-compat built-in is a table row that
immediately owes all four; builtin_kimi_test.go is deleted because the table
covers it. Only genuinely qwen-specific tests remain in the qwen file: the
reverse credential leak and the reasoning_effort wire claim ADR-0027 rests on.

Also trimmed ProviderQwen's doc comment, which restated the ADR-0027 rationale
already given at the registration site.

The break-check suite caught its own rot again — two mutations went stale when
these tests were renamed, and the landed-check reported them loudly instead of
passing them off as green. Now 9 cases, including one that drifts
envKeyForProvider to prove the shared helper is load-bearing. 9/9 apply and are
caught.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-12 16:40:53 -04:00
steveandClaude Opus 5 f1f2b653c3 refactor: gadfly round 2 — both halves of an OpenAI-compat built-in register together
CI / Tidy (pull_request) Successful in 9m25s
CI / Build & Test (pull_request) Successful in 9m50s
Same class of finding as round 1, one level in: I factored the DSN-scheme half
of the kimi/qwen duplication into openaiCompatScheme and left the eager
provider half copy-pasted, so a third built-in still had six lines to clone —
including both credential rules, which is exactly the pair you do not want
re-typed.

registerOpenAICompatBuiltin now installs both halves from one call. The rules
that matter hold by construction for every future caller: WithAPIKey passed
unconditionally (an unset key must not fall through to OPENAI_API_KEY), and
WithAPIKeyName naming that same variable in the 401 hint. Registering kimi and
qwen is now one line each.

Also fixed a cross-reference the ADR got wrong: Qwen's image-input caveat is
README matrix footnote ⁴, not ³ — ³ is kimi's. I wrote "³, shared with kimi"
in the ADR and then gave Qwen its own footnote in the README.

The break-check harness needed fixing before any of this could be trusted:
three of its mutations targeted lines this refactor moved, so they matched
nothing, the code was never broken, and the suite reported "test still passed"
— identical output to a test that genuinely misses the bug. Mutations are now
verified to have landed (sha before/after) and the suite fails loudly if one
doesn't. Two new cases cover the helper: dropping the unconditional WithAPIKey,
and dropping the scheme-half registration. 8/8 apply and are caught.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-12 16:25:38 -04:00
steveandClaude Opus 5 31d6b59356 refactor(test): gadfly round 1 — share the OpenAI-compat test fixtures
CI / Tidy (pull_request) Successful in 9m25s
CI / Build & Test (pull_request) Successful in 9m50s
Both findings were the same one, and both were fair: the PR that retires two
byte-identical DSN factories into openaiCompatScheme then copy-pasted the test
fixtures. qwenResponse was byte-identical to kimiResponse, and the single-key
env-lookup closure appeared three times in the new file (plus a fourth in the
kimi file, which neither reviewer was looking at).

Fixed for the class rather than for qwen: captureRT, the canned Chat
Completions body (now chatCompletionOK), and a new singleKeyEnv helper move to
builtin_openaicompat_test.go, owned by no single provider. The kimi tests adopt
them too, so the next OpenAI-compat built-in has nothing left to copy — the
same argument the production helper makes.

Also aligned the test model ids to the current Model Studio names
(qwen3.8-max / qwen3.7-plus), which the docs already cited. One reviewer called
those ids fictional and named the 2025 ones instead; they shipped 2026-08-03
and 2026-05-21 respectively, so that finding is stale model knowledge, not a
defect — but having tests and prose name the same models removes the smell that
prompted it. A dotted id also now proves it passes through verbatim.

Break-checked again after the refactor: all six mutations still fail their test.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-12 16:16:54 -04:00
steveandClaude Opus 5 02cd561eaf feat(qwen): Alibaba Qwen built-in over Model Studio's OpenAI-compatible mode
Gadfly review (reusable) / review (pull_request) Successful in 5m14s
Adversarial Review (Gadfly) / review (pull_request) Successful in 5m14s
CI / Tidy (pull_request) Successful in 9m24s
CI / Build & Test (pull_request) Successful in 9m53s
Adds the `qwen` built-in provider and the `qwen://` DSN scheme, keyed by
QWEN_API_KEY and defaulting to Model Studio's international host. Like kimi
(ADR-0026) it is `provider/openai` pointed elsewhere — no new client.

Model Studio serves the same models over two protocols, so the real decision
was which wire format to speak. ADR-0027 records why it is the OpenAI one:
down the anthropic client `ReasoningEffort` is ignored by design, structured
output rides the first-party `output_config.format` mechanism the shim does
not implement, and cached-token accounting reads Anthropic-only usage fields.
Each of those fails silently rather than loudly, which is what makes the
choice worth writing down. The shim stays reachable ad hoc via an
`anthropic://` DSN.

The kimi and qwen DSN factories were byte-identical, so they now share one
`openaiCompatScheme` helper: the "credential comes from the DSN token, and
the missing-key hint names LLM_<NAME>" rules hold by construction instead of
by copy.

Tests are hermetic and break-checked (all six fail on a deliberate mutation),
including the reverse credential leak — a visible QWEN_API_KEY must not
authenticate the openai built-in — and reasoning_effort asserted on the wire
body, which is the ADR's load-bearing claim.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-12 16:07:41 -04:00
steve e779169416 Merge pull request 'feat(videogen): LastImage — pin the trailing keyframe (first-last-frame-to-video)' (#26) from feat/videogen-last-frame into main
CI / Tidy (push) Successful in 9m23s
CI / Build & Test (push) Successful in 9m47s
2026-08-08 07:10:26 +00:00
steveandClaude Opus 5 5994d96921 refactor(llamaswap): drop initImageFilename — one caller left, and it was a rename of imageFilename
CI / Tidy (pull_request) Successful in 9m40s
CI / Build & Test (pull_request) Successful in 11m38s
The 2/4 finding is right: after writeImagePart started passing an explicit
filename stem, initImageFilename had no caller in video.go, and its
"conditioning frame" doc no longer described its one remaining user
(lipsync.go's avatar image). A one-line wrapper that survives only to be
misdescribed is not indirection worth keeping.

lipsync now calls imageFilename(mime, "frame") directly, and imageFilename's
doc lists the real bases — including WHY the video keyframes need distinct
ones: a backend that stages uploads by filename would otherwise have the
second overwrite the first.

Not taken: consolidating the first/last-frame rationale to a single canonical
site. The copies address different readers — the wire encoding (provider), the
contract's undetectable-support caveat (videogen), and the mode table (README)
— and last round's finding was a doc pointing at a note that did not exist.
Trading duplication for cross-references is what produced that.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01PLjgrxvHjm1sJgUu9zBPH9
2026-08-08 03:06:21 -04:00
steveandClaude Opus 5 588e092465 docs(videogen): gadfly — README FL2V section, and stop pointing at a note that does not exist
CI / Tidy (pull_request) Successful in 9m25s
CI / Build & Test (pull_request) Successful in 9m50s
- README documented only t2v/i2v. Now a table of the four keyframe
  combinations, plus the undetectable-support caveat, which is the one thing a
  caller cannot work out for itself.
- The LastImage doc comment said "see the note on LastImage support in
  provider/llamaswap" — there was no such note. A pointer to something that
  does not exist is worse than no pointer; the comment is now self-contained.
- Generate's doc described only input_reference; it now names
  input_reference_last and explains why an unsupporting backend returns a clip
  rather than an error.

The 2/4 finding (writeImagePart reusing the "frame" base for both parts) was
already fixed in dbc9689 — from the receiving end, where the consequence is
concrete rather than stylistic: ComfyUI stages uploads by FILENAME with
overwrite=true, so a shared name means the second clobbers the first and both
keyframes resolve to one image.

Not taken: initImageFilename's name is no longer misleading (writeImagePart
stopped calling it), and it is still used by lipsync.go so it is not dead.
The empty-LastImage test stays standalone — it mirrors the existing standalone
empty-InitImage coverage rather than a table this file does not have.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01PLjgrxvHjm1sJgUu9zBPH9
2026-08-08 02:59:12 -04:00
steveandClaude Opus 5 dbc96898ab fix(videogen): distinct FILENAMES for the two keyframes, not just distinct field names
CI / Tidy (pull_request) Successful in 9m25s
CI / Build & Test (pull_request) Successful in 10m28s
Caught while writing the receiving end. Distinct multipart field names are not
sufficient: backends stage an uploaded frame under a name derived from the
FILENAME, and our own ComfyUI shim posts to /upload/image with overwrite=true.
Both parts were sending initImageFilename(mime) — literally "frame.png" for
each — so the second upload would have clobbered the first and BOTH keyframe
inputs would have resolved to the same stored image.

The failure mode is the worst kind: a clip pinned at both ends to the same
frame renders cleanly, returns 200, and looks like the feature not working
rather than like a bug. Nothing upstream or downstream would report a fault.

writeImagePart now takes the filename stem (frame / frame_last), and the test
asserts the two arrive under different filenames.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01PLjgrxvHjm1sJgUu9zBPH9
2026-08-08 02:50:43 -04:00
steveandClaude Opus 5 44fcfbb273 feat(videogen): LastImage — pin the trailing keyframe (first-last-frame-to-video)
Gadfly review (reusable) / review (pull_request) Successful in 3m41s
Adversarial Review (Gadfly) / review (pull_request) Successful in 3m41s
CI / Tidy (pull_request) Successful in 9m26s
CI / Build & Test (pull_request) Successful in 9m48s
videogen.Request gains LastImage alongside InitImage, so one Request covers
t2v, i2v and FL2V without a mode flag. With InitImage it pins both ends of the
clip; alone it pins the destination and lets the backend invent the approach.

The llamaswap provider sends it as a SEPARATE `input_reference_last` part
rather than a second `input_reference`. Multipart permits repeated names, but
then which frame is first and which is last depends on part ORDER — an
ordering contract invisible in the payload, that nothing notices breaking. A
backend that does not know the new name ignores the part, the same degradation
as any other unknown field.

Both parts go through one writeImagePart helper so their encoding cannot
drift, and an empty LastImage is rejected up front exactly as InitImage
already is.

Support is per-model and deliberately NOT advertised in this contract: a
backend that ignores a trailing keyframe returns an ordinary clip, which is
indistinguishable from success. The doc comment says so, because a caller that
needs to know whether the pin took effect has to establish that out of band —
and the mort side gates on a convar for exactly this reason.

Motivated by mort's #1567 (long-form video): with both ends pinned, drift
becomes structurally bounded inside each shot instead of compounding across an
autoregressive chain.

Tests break-checked: sending the last frame under the shared name fails both
the distinct-name assertion and the last-alone case.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01PLjgrxvHjm1sJgUu9zBPH9
2026-08-08 02:47:39 -04:00
steve 203895696c Merge pull request 'fix(agent): recover the front-loaded answer over a summary closer' (#25) from fix/finalize-summary-closer into main
CI / Tidy (push) Successful in 9m22s
CI / Build & Test (push) Successful in 9m44s
2026-08-06 00:39:39 +00:00
steveandClaude Fable 5 1bbbdaa1e5 refactor(agent): gadfly round 2 — shared leadingMarkers, explicit mode, comment altitude
CI / Tidy (pull_request) Successful in 9m24s
CI / Build & Test (pull_request) Successful in 9m52s
All tidiness, no behavior change: the leading-marker class is one shared
constant for citationLabelRe and summaryCloserRe (hand-copying it is how
'+' went missing the first time); the deliberate 'all' duplication across
summaryCopulas/summaryArticle is now stated at both sites; the weak-final
switch case assigns modeBackRef explicitly; test comments state the
constraint they guard instead of which reviewer asked for them.

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-08-05 20:29:19 -04:00
steveandClaude Fable 5 21b4775d16 fix(agent): gadfly round 1 — user-boundary scan, back-ref precedence, regex legibility
CI / Tidy (pull_request) Successful in 9m39s
CI / Build & Test (pull_request) Successful in 10m5s
Two behavioral fixes from the review:

- modeSummary's backward scan now stops at the most recent user message.
  With the dwarf ratio rejecting the current turn's 1x-3x answer, the old
  unbounded scan could walk into WithHistory content and resurrect a stale
  answer to a DIFFERENT question — strictly worse than keeping the closer
  (opus, correctness). Other modes keep their historical unbounded scan.
- A terminal matching BOTH the ack shape and a back-reference is now
  classified back-ref: it carries no answer content, so the looser bar is
  the right one (opus, error-handling).

Plus the nits: summaryCloserRe assembled from named fragments, the leading
marker class gains '+' (parity with citationLabelRe), verb-first form takes
'all the', dwarf ratio hoisted into one named local, and the 151-vs-153
char/byte comment inaccuracy corrected.

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-08-05 20:17:56 -04:00
steveandClaude Fable 5 127966bb3a fix(agent): recover the front-loaded answer over a summary closer
CI / Build & Test (pull_request) Successful in 9m49s
CI / Tidy (pull_request) Successful in 10m28s
Gadfly review (reusable) / review (pull_request) Successful in 11m49s
Adversarial Review (Gadfly) / review (pull_request) Successful in 11m49s
A third degenerate terminal shape from the glm-5.2 cite pattern: the model
front-loads its full answer into the cite-call turn, then closes with a
bookkeeping ack plus a one-line compression ("Citations are logged. Short
version: ..."). mort run b3cb9ee9 delivered 151 chars of a 2,089-char
answer this way — the closer was neither a back-reference (over the 120
cap, no back-ref phrase) nor a citations addendum (no label-colon, no
links), so finalOutput let it stand.

isSummaryCloser keys on the ack sentence alone (the verb must end the
sentence, so prose about citations never matches; a compression marker
without the ack is deliberately out of scope), and the new modeSummary
recovery bar makes the 3x dwarf ratio mandatory at every length: unlike a
back-reference this closer carries real answer content, so it is only
displaced by the clearly-fuller original it compressed.

The citations/back-ref bool becomes a three-way recoveryMode; existing
behavior for both old modes is unchanged.

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-08-05 08:47:40 -04:00
steve 0bd14e01b3 Merge pull request 'feat(faceswap): report whether the likeness actually transferred' (#24) from feat/imagegen-faceswap into main
CI / Tidy (push) Successful in 9m31s
CI / Build & Test (push) Successful in 10m38s
2026-07-31 21:51:12 +00:00
steve 6995a8dee1 feat(faceswap): expose yaw on enumeration too
Gadfly review (reusable) / review (pull_request) Successful in 4m54s
Adversarial Review (Gadfly) / review (pull_request) Successful in 4m55s
CI / Tidy (pull_request) Successful in 9m23s
CI / Build & Test (pull_request) Successful in 9m48s
ListFaces now carries head yaw, so a caller choosing WHICH face to swap can
see the thing that decides whether the swap will read — not only learn it
afterwards from the swap report. In the run that prompted this the target's
three faces sat at -82, -8 and -11 degrees; only the first was hopeless, and
nothing in a bounding box said so.
2026-07-31 17:50:27 -04:00
steve ae2615ca68 feat(faceswap): carry the measured outcome, not just the image
A face swap always returns an image and always looks like success. Whether the
likeness actually transferred is a different question, and until now nothing in
the response answered it — so a caller wanting to know went and asked a vision
model instead. That is wrong in precisely the cases that matter: shown a jogger
in a Georgetown cap holding McDonald's cups, a VLM answers "Bill Clinton"
whoever's face is on him. In the run that prompted this it reported failure on
six consecutive CORRECT swaps (measured afterwards at 0.79-0.84 cosine), and
the caller burned 21 minutes chasing a problem that did not exist.

Result.SwappedFaces now carries, per replaced face: pixel size, the target
image's dimensions, head yaw, and cosine similarity between the source face and
the face actually present in the output.

Yaw and FractionOfImage are the two that explain the complaint. The swap in
question replaced a 138px face in a 1010px-wide photo — 14% of the width,
correct and invisible at a glance — and elsewhere a face turned -82 degrees,
where the features carrying identity are edge-on and any swap reads as a
generic person. Same code on a 168px face in a 385px picture (44%, yaw 2) is
unmistakable. None of that was inferable from a bounding box.

Typed on Result rather than stuffed into Raw: a caller has to act on this, and
a value reachable only by type-asserting an `any` is one nobody finds in time.

doRawHeaders is doRaw with the whole header instead of only Content-Type; doRaw
delegates to it, so the other 25 call sites are untouched and there is still
one place where the status check and the size cap live.

A missing or malformed header yields nil, not an error — an older shim sends no
header, and a swap that produced a good image must not fail because the
diagnostics beside it were unreadable. Covered for absent/garbage/wrong-type,
and the parse is break-checked.
2026-07-31 17:49:27 -04:00
steve ff832cb6b5 Merge pull request 'feat(imagegen): face swap (identity transfer), a separate operation from Edit' (#23) from feat/imagegen-faceswap into main
CI / Tidy (push) Successful in 9m26s
CI / Build & Test (push) Successful in 9m48s
2026-07-31 16:55:09 +00:00
steve 372bf826aa fix(llamaswap): a headerless non-image response was returned as a PNG
CI / Tidy (pull_request) Successful in 9m26s
CI / Build & Test (pull_request) Successful in 10m4s
Gadfly on #23, blocking, 2/2 agreement — and it is the exact defect this
whole line of work has been about: a call that succeeds while handing back
the wrong bytes.

sniffImageMIME falls back to image/png when detection is inconclusive, and
the guard only consulted Content-Type. A response with NO Content-Type
therefore skipped the check entirely and was labelled a PNG. The shim answers
JSON on a semantic miss (no face found in the source or target), which is
precisely the body that would have sailed through as a successful image.

The check now validates the BYTES — http.DetectContentType must say image/ —
and the reported MIME prefers the server's own label only when that label is
itself an image type. Break-checked by restoring the header-only condition,
which fails the new test.

Also from that review:
  - index is documented as ignored under all=true, so a negative one is no
    longer rejected there; it is still rejected when it would actually be
    sent, and both halves are tested.
  - initImageFilename (video.go) was imageFilename with the base fixed to
    "frame" and now delegates to it — two copies of one extension table is
    how they drift.
  - DetectedFace carried Width/Height alongside Box, two sources of truth for
    one fact that can disagree after any transform. Now a Size() method
    derived from Box.
  - a dead `apiErr` in the test (declared, then `_ = apiErr`) was an
    abandoned errors.As check; it is wired up and now asserts callers can
    classify the error.
  - swapImg duplicated editInit verbatim; removed.

Not taken: adding a FaceSwapProvider/ModelOption surface to match the other
optional imagegen capabilities (single-model finding). There are no options
to carry yet, and inventing an empty option type to look symmetrical would be
API surface with nothing behind it. Worth revisiting when a real knob exists.
2026-07-31 12:34:58 -04:00
steve 0ff90d80f6 feat(imagegen): face swap (identity transfer), a separate operation from Edit
Gadfly review (reusable) / review (pull_request) Successful in 5m7s
Adversarial Review (Gadfly) / review (pull_request) Successful in 5m7s
CI / Tidy (pull_request) Successful in 9m24s
CI / Build & Test (pull_request) Successful in 9m52s
Measured against the instruction-edit models on 2026-07-31: asking a diffusion
model to put a SPECIFIC person's face into a photo does not work by any route.
qwen-image-edit returns the picture essentially unchanged whether asked by
name, by attribute, or by supplying the portrait as a second reference image;
flux-kontext replaces the face with a different generic person. Identity
transfer is a detect/align/blend pipeline, not a better prompt, so it gets its
own interface rather than more Edit options.

imagegen.FaceSwapper is optional and type-asserted, like Editor — a provider
that cannot do this must not have Edit quietly stand in for it.

ListFaces is part of the interface, not a convenience: a caller asked to
change "the man on the right" needs a stable way to NAME one face, and pixel
boxes let it check its own choice. The llamaswap shim orders faces left to
right for exactly that reason (insightface's own order is score-ranked and
unstable between near-identical images), and a malformed box is a protocol
error rather than a zero-filled struct, because a wrong box aims the swap at
the wrong person.

The provider is the first here to POST more than one file, so buildMultipart
gained buildMultipartFiles and now delegates to it — one writer loop, so the
two cannot drift in how they escape names or terminate the body.

index and all are mutually exclusive ON THE WIRE: the shim ignores index under
all=true, and sending both would imply a precedence the caller cannot see.
A JSON body is refused rather than returned as image bytes — the shim answers
JSON on a semantic miss (no face in the source), and handing that back as a
picture would report success while delivering a file that is not one.
2026-07-31 12:25:12 -04:00
steve 316a430116 Merge pull request 'feat(imagegen): reference-image editing for instruction-edit models' (#22) from feat/imagegen-reference-edit into main
CI / Tidy (push) Successful in 9m26s
CI / Build & Test (push) Successful in 9m48s
2026-07-31 01:35:36 +00:00
steve 2c70d32fd4 feat(imagegen): reference-image editing for instruction-edit models
Gadfly review (reusable) / review (pull_request) Successful in 4m24s
Adversarial Review (Gadfly) / review (pull_request) Successful in 4m24s
CI / Tidy (pull_request) Successful in 9m40s
CI / Build & Test (pull_request) Successful in 11m17s
FLUX.1 Kontext and Qwen-Image-Edit are a different kind of edit from img2img
and reach sd-server by a different path, and nothing in imagegen could
express it: EditRequest only had Init, which is noised and denoised back
under the prompt.

Measured against FLUX.1-Kontext on the netherstorm host 2026-07-30, on a
synthetic scene with a red rectangle, a blue rectangle and a flat background,
prompted "change the blue rectangle on the right to bright green, keep
everything else exactly the same":

  via init_images (the only path that existed)
      right rect (60,60,200) -> (47,82,228)   still blue, instruction ignored
      left rect  (200,60,60) -> (229,43,50)   drifted
      background (150,200,240) -> (154,211,229) drifted

  via extra_images (this change)
      right rect (60,60,200) -> (70,254,4)    green, as asked
      left rect  (200,60,60) -> (204,57,57)   intact
      background (150,200,240) -> (151,202,247) intact

No mask, no strength, no compositing — the model is handed the picture as
conditioning and the prompt as an instruction about it.

EditRequest.RefImages selects the path; when set, Init/Mask/Strength are
ignored rather than rejected, so a caller handing the same request to
whichever model is configured gets the better result on a Kontext-class model
instead of an error. The provider posts /sdapi/v1/txt2img with extra_images
(sd-server reads that field on both routes into gen_params.ref_images, where
the CLI's -r/--ref-image also lands); there is no init latent to denoise, so
sending one would only add noise to a pipeline that does not want any.

An all-empty reference set is refused: it would otherwise degrade into a
plain txt2img and render the prompt from scratch, which is not the request.
2026-07-30 21:21:35 -04:00
steve a941f5ff4a Merge pull request 'fix(agent): make same-call repeat guard progress-aware' (#21) from fix/progress-aware-same-call-guard into main
CI / Tidy (push) Successful in 9m23s
CI / Build & Test (push) Successful in 9m44s
2026-07-18 23:22:10 +00:00
steveandClaude Opus 4.8 9922166d7a review(agent): address gadfly on progress-aware guard
CI / Tidy (pull_request) Successful in 9m24s
CI / Build & Test (pull_request) Successful in 9m43s
Merge the parallel callCounts/lastResults maps into one repeatState struct
map (removes the "two maps in sync" smell + double lookup), drop the dead
i<len(results) bounds branch that contradicted the documented index invariant,
and note in-code that exact-string result equality is a deliberate err-toward-
not-tripping choice (a hung job whose poll reports a ticking field is left to
MaxRuntime / the job ceiling rather than risking a false kill of real progress).
No behavior change; guard tests still green.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01HgEuVfZJN9mhRhzEsMEVog
2026-07-18 19:11:30 -04:00
steveandClaude Opus 4.8 68bf7157d3 fix(agent): make same-call repeat guard progress-aware
CI / Tidy (pull_request) Successful in 9m25s
Gadfly review (reusable) / review (pull_request) Successful in 9m48s
Adversarial Review (Gadfly) / review (pull_request) Successful in 9m48s
CI / Build & Test (pull_request) Successful in 10m33s
The maxSameCallRepeats guard counted identical (name+arguments) tool calls
across a run and tripped ErrToolLoop past the ceiling — regardless of whether
each call made progress. This killed legitimate polling of long-running
background jobs: code_exec_poll must be called with identical args (same
job_id), so a render/encode that needs more than N polls was guillotined
mid-flight even as each poll returned an advancing result (elapsed/status
moving forward).

Only count an identical call toward the trip when its RESULT is unchanged
from the previous identical call. A call whose result keeps changing is
progress and resets its count; a genuinely stuck call returning the same
output still trips. This can never trip more than before, only less, and
covers every idempotent poller with no per-tool configuration — matching the
progress-over-usage thesis behind the stall-detection work.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01HgEuVfZJN9mhRhzEsMEVog
2026-07-18 18:56:56 -04:00
steve 54b295efc3 chore(gadfly): bump reusable pin to c9dab69 — provider-wide lens budget [skip ci]
Adopt gadfly's single per-provider lens budget (PR #27, image sha-b37cd09).
Pin-only version bump; central swarm config is unchanged.
2026-07-18 16:54:01 +00:00
steve c5f84b95d8 Merge pull request 'feat: kimi (Moonshot AI) built-in provider (ADR-0026)' (#20) from feat/kimi-provider into main
CI / Tidy (push) Successful in 9m25s
CI / Build & Test (push) Successful in 10m17s
2026-07-18 07:26:10 +00:00
steveandClaude Opus 4.8 fcbb01b729 fix: address Gadfly findings on kimi provider
CI / Tidy (pull_request) Successful in 9m29s
CI / Build & Test (pull_request) Successful in 9m43s
- kimi:// DSN scheme: missing-credential hint now names the LLM_<NAME> env
  var that defines the provider (its token comes from the DSN, not
  KIMI_API_KEY), matching providerFor's lazy-resolution key form. Fixes the
  correctness/error-handling findings that the old hint misdirected users to
  set KIMI_API_KEY when the fix is adding a token to the DSN.
- parse_test.go: add kimi to TestBuiltinsResolve. (llama-swap stays excluded
  and is now documented — its no-URL built-in errors at Model() construction,
  not just on use, so it can't resolve there; the finding's llama-swap half
  was a false lead the test surfaced.)
- Add TestKimiSchemeMissingToken covering the corrected hint.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-07-18 03:24:48 -04:00
steveandClaude Opus 4.8 2bfffff47a feat: kimi (Moonshot AI) built-in provider (ADR-0026)
CI / Tidy (pull_request) Successful in 9m29s
CI / Build & Test (pull_request) Successful in 10m23s
Gadfly review (reusable) / review (pull_request) Successful in 18m38s
Adversarial Review (Gadfly) / review (pull_request) Successful in 18m38s
Add a first-class `kimi` provider and `kimi://` DSN scheme for Moonshot AI's
OpenAI-compatible Chat Completions API. Both reuse provider/openai (no new
client, mirroring llama-swap's chat path). Default endpoint is the
international host; the China endpoint is reachable via a kimi:// LLM_* DSN.

- Credential is KIMI_API_KEY, read through the registry's injected envLookup
  so it stays hermetically testable. WithAPIKey is passed unconditionally so
  an unset KIMI_API_KEY can never fall through to the openai client's
  OPENAI_API_KEY default.
- New openai.WithAPIKeyName option customizes the missing-key error hint
  (default OPENAI_API_KEY); kimi names KIMI_API_KEY.
- Hermetic tests: built-in base URL + bearer, missing-key hint names
  KIMI_API_KEY with no OPENAI fallthrough and no network hit, kimi:// scheme
  round-trips against the China host.
- Docs in sync: README built-in table + DSN scheme list + support matrix,
  .env.example, env.go DSN doc, ADR-0026 (+ index, backfilling 0024/0025),
  progress.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-07-18 02:59:30 -04:00
steve a9c864d3f7 chore(gadfly): bump reusable pin to 0d51879 (opencode-capable image)
CI / Tidy (push) Successful in 9m27s
CI / Build & Test (push) Successful in 9m52s
Picks up gadfly's opencode/<model> engine — the reusable at this ref pins the
sha-bb98fae reviewer image that bundles the OpenCode CLI. Also correct the pin
comment: this is an immutable sha pin, not the v1 tag it claimed to track.
2026-07-18 06:00:19 +00:00
steve 95147f7582 Merge pull request 'feat: wave-3 video surfaces — lipsync, video matte, video upscale, chain jobs (ADR-0025)' (#19) from feat/wave3-video-surfaces into main
CI / Tidy (push) Successful in 9m24s
CI / Build & Test (push) Successful in 9m46s
2026-07-16 23:25:18 +00:00
steve 036406b221 Merge pull request 'feat: wave-3 audio surfaces — stems, SFX, speech enhance, voice clone, translate (ADR-0024)' (#18) from feat/wave3-audio-surfaces into main
CI / Tidy (push) Successful in 9m22s
CI / Build & Test (push) Successful in 9m44s
2026-07-16 23:24:42 +00:00
steve 2660693132 Merge pull request 'feat: wave-3 image + document surfaces — segmentation, colorize, face restore, OCR (ADR-0023)' (#17) from feat/wave3-image-doc-surfaces into main
CI / Tidy (push) Successful in 9m30s
CI / Build & Test (push) Successful in 10m23s
2026-07-16 23:24:33 +00:00
steveandClaude Fable 5 56b5b000a6 fix: review findings — chain NaN/Inf + id hygiene, percent-escape jobPath, shared singleVideoResult
CI / Tidy (pull_request) Successful in 9m25s
CI / Build & Test (pull_request) Successful in 9m46s
- SubmitChain rejects NaN/±Inf segment seconds with ErrUnsupported
  (previously an obscure json.Marshal error; NaN fails every comparison
  and +Inf passed the >= 0 check).
- ChainStatus skips segment entries with no usable id — JSON null
  (which no-op-unmarshals into a string, previously appending ""),
  empty strings, and id-less objects; the unfiltered list survives in
  Raw. ChainJob.SegmentIDs doc now also says ChainSegmentResult takes
  the segment index, not an id string.
- jobPath rejects '%' in job ids — %2F/%2E%2E percent-escapes decode
  back into path structure server-side, bypassing the literal check on
  this upstream-echoed value.
- singleVideoResult moves to video.go next to videoMIME, and the two
  remaining hand-rolled copies of the video-result validation
  (videoModel.Generate, Interpolate) now use it — one validation, one
  message shape.
- videogen.LipSyncer renamed to videogen.Lipsyncer for consistency with
  the rest of the surface's Lipsync* naming (LipsyncProvider,
  LipsyncModel, LipsyncRequest); not yet consumed downstream, so the
  rename is free now and never again.

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WWCQcYStWXBYUy5sZWnbLT
2026-07-16 19:12:42 -04:00
steveandClaude Fable 5 31bccf6e19 fix: review findings — clone-route audio sniffing, zip caps, filename sanitization, WAV response caps
CI / Tidy (pull_request) Successful in 9m27s
CI / Build & Test (pull_request) Successful in 9m46s
- speakWithReference now validates the response IS audio via the same
  audioResultMIME sniff the sfx/enhance surfaces use — a 2xx JSON soft
  error or HTML proxy page was previously wrapped up as audio/wav bytes.
- audioResultMIME moves to audio.go (next to speechMIME; it was defined
  in sfx.go but shared by enhance/clone) and learns the Ogg container
  normalization (application/ogg -> audio/ogg).
- Stems zip unpack gains entry-count (16) and total-decompressed (1GB)
  caps on top of the existing per-entry cap — the per-entry bound alone
  still let a many-entry bomb multiply up.
- sanitizeFilename drops NUL and both path separators too, so upload
  metadata can never smuggle directory structure to a file-writing shim.
- New maxAudioResponseBytes (256MB) for bodies that ARE one audio clip
  (clone, enhance, sfx): a long WAV legitimately passes the 64MB JSON
  cap.
- speakWithReference local renamed path -> upPath (naming parity with
  stems/enhance).

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WWCQcYStWXBYUy5sZWnbLT
2026-07-16 19:07:37 -04:00
steveandClaude Fable 5 966ea16166 fix: review findings — NaN threshold guard, percent-escape rejection in upstream model ids
CI / Tidy (pull_request) Successful in 9m31s
CI / Build & Test (pull_request) Successful in 9m48s
- Segment: reject NaN thresholds (NaN fails every comparison, so it
  passed the [0,1] range check and reached the shim as the literal
  string "NaN"); ±Inf were already caught by the range comparisons,
  now covered by tests too.
- upstreamPath: reject '%' in model ids — %2F/%2E%2E percent-escapes
  decode back into path structure server-side, bypassing the literal
  /?#/.. rejection. Ids never legitimately contain '%'.

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WWCQcYStWXBYUy5sZWnbLT
2026-07-16 19:00:11 -04:00
steveandClaude Fable 5 5f175ecf82 feat: wave-3 video surfaces — lipsync, video matte, video upscale, chain jobs (ADR-0025)
CI / Tidy (pull_request) Successful in 9m27s
CI / Build & Test (pull_request) Successful in 10m43s
Gadfly review (reusable) / review (pull_request) Successful in 10m9s
Adversarial Review (Gadfly) / review (pull_request) Successful in 10m9s
- videogen.LipSyncer/LipsyncProvider: SadTalker talking heads via
  POST /upstream/<id>/v1/talking_head (multipart image+audio parts,
  still/enhance/preprocess flags) -> mp4.
- videogen.VideoBackgroundRemover/VideoBackgroundRemovalProvider:
  POST /upstream/<id>/v1/video/matte (output greenscreen_mp4|alpha_webm).
- videogen.VideoUpscaler/VideoUpscaleProvider:
  POST /upstream/<id>/v1/video/upscale (scale 2|4).
- videogen.Chainer/ChainerProvider: async long-video chain-job client —
  SubmitChain (JSON POST /v1/video/chain, init_image_b64), ChainStatus
  (GET /v1/jobs/{id}, tolerant segment-id decode), ChainResult,
  ChainSegmentResult (partial delivery after mid-chain failure); hostile
  job-id path rejection.
- Shared singleVideoResult validation (positive video evidence) + a
  videoInputFilename hint helper; httptest contract tests per surface;
  ADR-0025 (index row deferred — MJ-A backfills the ADR index table and
  parallel edits would conflict).

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-07-16 17:07:27 -04:00
steveandClaude Fable 5 b3a172a053 feat: wave-3 audio surfaces — stems, sfx, speech enhance, voice clone, translate (ADR-0024)
CI / Tidy (pull_request) Successful in 9m27s
CI / Build & Test (pull_request) Successful in 10m37s
Gadfly review (reusable) / review (pull_request) Successful in 41m14s
Adversarial Review (Gadfly) / review (pull_request) Successful in 41m14s
- audio.StemSeparator/StemSeparationProvider: Demucs zip transport via
  POST /upstream/<id>/v1/stems (Mode two -> two_stems=vocals; model +
  format fields); bounded zip unpack, entry name -> stem, ext -> MIME.
- SFXModel reuses musicgen against the sync /upstream/<id>/v1/sfx route
  (JSON prompt/seconds/steps/cfg_scale/seed -> WAV); musicgen.Request
  gains CFGScale.
- audio.SpeechEnhancer/SpeechEnhancementProvider:
  POST /upstream/<id>/v1/enhance -> WAV (result reuses SpeechResult).
- SpeechRequest.ReferenceAudio/ReferenceMIME (+WithReferenceAudio):
  llamaswap switches to the chatterbox clone route
  POST /upstream/<id>/v1/audio/speech/upload (input + voice_file),
  wav MIME fallback.
- TranscriptionRequest.Translate (+WithTranslate): translate=true form
  field, language=auto forced when no explicit hint (whisper.cpp default
  en would skip translation).
- httptest contract tests (zip unpack, clone-route switch, translate +
  auto-language injection); ADR-0024 (index row deferred — MJ-A backfills
  the ADR index table and parallel edits would conflict).

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-07-16 17:01:11 -04:00
steveandClaude Fable 5 e6987f54b2 feat: wave-3 image + document surfaces — segmentation, colorize, face restore, ocr (ADR-0023)
CI / Tidy (pull_request) Successful in 9m25s
CI / Build & Test (pull_request) Successful in 9m48s
Gadfly review (reusable) / review (pull_request) Successful in 47m4s
Adversarial Review (Gadfly) / review (pull_request) Successful in 47m4s
- imagegen.Segmenter/SegmentationProvider: prompted mask via
  POST /upstream/<id>/v1/segment (file, prompt[, threshold], output=mask);
  white = prompted region, EditRequest.Mask polarity.
- imagegen.Colorizer/ColorizeProvider: POST /upstream/<id>/v1/colorize.
- imagegen.FaceRestorer/FaceRestoreProvider:
  POST /upstream/<id>/v1/restore_faces (upscale 1|2).
- New ocr leaf package (Request/Page/Result, Recognize) + llamaswap
  OCRModel: POST /upstream/<id>/v1/ocr (file[, langs, max_pages]),
  tolerant per-page decode (join lines when page text absent), Raw
  escape hatch.
- httptest contract tests per surface; ADR-0023; ADR index backfilled
  (0020-0022 rows were missing).

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-07-16 16:54:30 -04:00
steveandClaude Fable 5 cd43009672 ci: pin gadfly reusable @8eb0265 + thread dispatch pr_number [skip ci]
Gitea >= 1.27 does not propagate workflow_dispatch inputs into a called
workflow's github.event; the stub must pass pr_number as an explicit
workflow_call input or manual dispatches die at 'PR required'. Mirrors
gadfly#24.

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-07-15 23:34:54 -04:00
steveandClaude Fable 5 499ee16222 ci: pin gadfly reusable @3664ce8 (ragnaros endpoint replaced by netherstorm) [skip ci]
Co-Authored-By: Claude Fable 5 <[email protected]>
2026-07-15 23:32:43 -04:00
steveandClaude Fable 5 a07cba25dc ci: pin gadfly reusable @f542d4e (forward netherstorm endpoint) [skip ci]
The netherstorm reviewer failed with 'unknown provider': the correctly
formatted GADFLY_ENDPOINT_NETHERSTORM user var was never forwarded by
the reusable workflow (hardcoded env list). Mirrors gadfly#22.

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-07-15 23:31:04 -04:00
steveandClaude Fable 5 a4cf9202cd ci: pin gadfly reusable @b6a33dc (Gitea 1.27 workflow_call hotfix) [skip ci]
Gitea 1.27 hands called workflows event_name=workflow_call, so every
review since 2026-07-14 self-skipped in 1s while reporting success. The
hotfix lineage (gadfly 5007597 + entrypoint reclassification, image
sha-ed9e946) restores the exact pre-upgrade reviewer; gadfly main
carries the same fix for the executus re-platform rollout.

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-07-15 23:02:06 -04:00
steve 19972b1864 Merge pull request 'fix: live-API corrections — music result shapes, mesh format honesty, mesh conversion' (#16) from fix/live-smoke-round2 into main
CI / Tidy (push) Successful in 9m23s
CI / Build & Test (push) Successful in 9m47s
Reviewed-on: #16
2026-07-14 16:45:57 +00:00
steveandClaude Fable 5 fd8d56c3c1 fix: live-API corrections — music result shapes, mesh format honesty, mesh conversion
Gadfly review (reusable) / review (pull_request) Successful in 15s
Adversarial Review (Gadfly) / review (pull_request) Successful in 15s
CI / Tidy (pull_request) Successful in 9m26s
CI / Build & Test (pull_request) Successful in 10m0s
Round 2 from live smokes on netherstorm (2026-07-14):

- ACE-Step result blob is an ARRAY of objects and carries RAW control
  characters inside string values (literal newlines) — strict JSON
  rejected it. parseMusicResult sanitizes control chars (only legal
  inside string values in the double-encoded blob) and accepts array or
  object shapes. Regression test uses the live payload shape.
- Hunyuan3D GenerationRequest has NO output-format field (the documented
  type param is fiction) — it always returns GLB. Results are now
  labelled by sniffed magic bytes, never by the requested format.
- NEW meshgen.Converter/ConverterProvider optional surface + llamaswap
  impl over the mediautils shim POST /v1/convert_mesh — the STL hop for
  the printer pipeline.

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-07-14 12:22:06 -04:00
steve 89a0b2bdd3 Merge pull request 'feat: musicgen + embeddings/rerank surfaces (ADR-0021, ADR-0022)' (#15) from feat/musicgen-embeddings into main
CI / Tidy (push) Successful in 9m42s
CI / Build & Test (push) Successful in 10m26s
2026-07-13 23:14:00 +00:00
steve 148069f6ef Merge pull request 'feat: media expansion surfaces — edit mask, upscale, background removal, interpolation, diarization, meshgen (ADR-0020)' (#14) from feat/media-expansion-surfaces into main
CI / Tidy (push) Successful in 9m29s
CI / Build & Test (push) Successful in 9m47s
2026-07-13 23:13:36 +00:00
steveandClaude Fable 5 cf2d83f157 fix: music poll resilience + hostile-URL guard + embed dup-index (gadfly round 1)
- pollResult tolerates up to 5 CONSECUTIVE bad polls (transport blip,
  unparseable payload, task momentarily absent) instead of killing a
  multi-minute exclusive-GPU job on the first hiccup; only status=2, a
  failure run, or ctx deadline aborts
- server-supplied result.File must be server-relative; combined with the
  upstreamPath dot-dot/scheme rejection this stops a hostile upstream
  from steering the follow-up GET at other proxy endpoints (test:
  ../../api/models/unload refused)
- WithSteps(<=0) rejected; embed responses repeating an index rejected;
  musicFormatMIME now wraps speechMIME (one format table, wav32
  normalized); poll interval is a test-shrinkable var (CI no longer
  burns 2s+ per music test); parens + comments per review

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-07-13 00:52:06 -04:00
steveandClaude Fable 5 cacce61ddc feat: musicgen + embeddings/rerank surfaces (ADR-0021, ADR-0022)
- NEW musicgen leaf package: blocking Generate over ACE-Step's async job
  queue (release_task -> poll query_result -> fetch file, all via
  /upstream); tolerant envelope parsing, double-encoded result handled
- NEW embeddings leaf package: EmbedModel + RerankModel as separate mints
  (two server instances on the host, llama.cpp #20085); InstructedQuery
  helper for Qwen3-style query/document asymmetry
- provider/llamaswap: /v1/embeddings + /v1/rerank clients with strict
  validation (index-ordered vectors, count mismatch and out-of-range
  index are hard errors; rerank sorted descending, minimal parser)

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-07-13 00:50:36 -04:00
steveandClaude Fable 5 4a752ffa2a fix: harden upstream path + binary-body validation (gadfly round 1)
CI / Tidy (pull_request) Successful in 9m24s
CI / Build & Test (pull_request) Successful in 10m17s
- upstreamPath rejects '..' in model ids AND in the rest path — the rest
  can embed SERVER-SUPPLIED components (ACE-Step result file URLs), so
  dot-dot/scheme smuggling toward other proxy endpoints is refused
- singleImageResult requires positive image evidence (sniffed magic OR
  declared image/*): an empty-Content-Type error page can no longer pass
  as 'the image' via sniffImageMIME's PNG-default labelling
- upscale/background responses get a dedicated 256MB cap (the 64MB cap
  is JSON-sized; a 4x PNG legitimately exceeds it)
- mesh JSON-detection widened (512-byte whitespace-tolerant peek + reject
  declared application/json)
- Transcribe now reuses buildMultipart; transcriptionFilename takes
  (filename, mime) so diarize shares it without a fake request struct;
  truncateForError stops shadowing builtin cap; OnlyMask doc de-ambiguated

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-07-13 00:50:21 -04:00
steveandClaude Fable 5 e98493bcfb feat: media expansion surfaces — edit mask, upscale, background removal, interpolation, diarization, meshgen (ADR-0020)
CI / Tidy (pull_request) Successful in 9m23s
CI / Build & Test (pull_request) Successful in 10m24s
Adversarial Review (Gadfly) / review (pull_request) Successful in 15m37s
- imagegen.EditRequest.Mask -> sd-server img2img inpainting (white=repaint)
- imagegen.Upscaler + BackgroundRemover, videogen.Interpolator,
  audio.DiarizationModel: new optional provider-minted surfaces
- NEW meshgen leaf package (image->3D, glb/stl/obj)
- provider/llamaswap: all five via the /upstream/<model>/<path> passthrough
  (upstreamPath helper, shared one-file multipart builder); binary success
  bodies validated (non-image, non-video, JSON-mesh rejection); diarization
  pins output=json (vtt/srt drop speaker labels)

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-07-12 23:52:34 -04:00
steve 900317af4e Merge pull request 'feat(videogen): canonical video-generation surface + llama-swap client' (#13) from feat/videogen into main
CI / Tidy (push) Successful in 9m27s
CI / Build & Test (push) Successful in 10m7s
Reviewed-on: #13
2026-07-12 14:21:08 +00:00
steveandClaude Fable 5 776ef6fda9 fix: gadfly review — video-sized response cap, stale comment
CI / Tidy (pull_request) Successful in 9m31s
CI / Build & Test (pull_request) Successful in 10m23s
maxVideoResponseBytes (512MB) replaces the shared 64MB JSON cap on the
/v1/videos/sync read path (doRaw now takes the cap per call) — 3/6
models flagged that a legitimate long/high-bitrate clip would be
discarded after minutes of GPU work. Plus a stale stable-diffusion
comment in initImageFilename and a test-handler early return.

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01AXQxVhXBw8PwFAtsVrXSmj
2026-07-12 09:57:53 -04:00
steveandClaude Fable 5 89210346b8 fix: address review — fail loud on non-video bodies, dedupe form plumbing, doc parity
CI / Tidy (pull_request) Successful in 9m28s
CI / Build & Test (pull_request) Successful in 10m0s
- videoMIME no longer hard-falls-back to video/mp4: a 2xx body that is
  neither declared nor sniffable as video (JSON job envelope, HTML error
  page) is now an APIError instead of a 'successful' corrupt clip.
- Resolution rides the wire as width/height AND the OpenAI-style size
  string, so either upstream convention honors an explicit request.
- writeFormFields + mimeFromContentType shared helpers replace the
  copied multipart loop (audio.go/video.go) and Content-Type branch.
- ADR-0019 indexed in docs/adr/README.md; README gains the videogen
  section + support-matrix mention (docs-parity rule).

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01AXQxVhXBw8PwFAtsVrXSmj
2026-07-12 09:53:34 -04:00
steveandClaude Fable 5 4629c6af0f feat(videogen): canonical video-generation surface + llama-swap client
Adversarial Review (Gadfly) / review (pull_request) Successful in 8m43s
CI / Tidy (pull_request) Successful in 9m28s
CI / Build & Test (pull_request) Successful in 10m11s
New videogen/ contract package (ADR-0019): Request/Result/Model/Provider
with the imagegen conventions. Text-to-video and image-to-video are one
surface (Request.InitImage, nil = t2v) since hybrid checkpoints like
Wan 2.2 TI2V serve both from one model; Result carries a single clip.

provider/llamaswap gains VideoModel(id) targeting the blocking
POST {base}/v1/videos/sync (multipart, model-routed by the fork's new
video routes): vLLM-Omni parameter names, OpenAI-style input_reference
file part, optional fields stay off the wire so per-model launch-flag
defaults apply. CLAUDE.md package map picks up audio/ (missed in #12)
and videogen/.

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01AXQxVhXBw8PwFAtsVrXSmj
2026-07-12 09:31:08 -04:00
steve 5ed18abb94 Merge pull request 'feat: audio surfaces (TTS + transcription), imagegen.Editor, llamaswap health probe' (#12) from feat/audio-and-image-edit into main
CI / Tidy (push) Successful in 9m28s
CI / Build & Test (push) Successful in 10m27s
Reviewed-on: #12
2026-07-12 04:09:30 +00:00
steveandClaude Fable 5 9c0ac1d60b fix: address gadfly review — filename sanitization, truncation guard, shared plumbing
CI / Tidy (pull_request) Successful in 10m2s
CI / Build & Test (pull_request) Successful in 10m24s
- Transcribe: sanitize the caller-supplied multipart filename (CR/LF
  would inject Content-Disposition headers; upload metadata is
  untrusted), always send the required model/response_format fields,
  parse MIME parameters before extension matching, and give audio/opus
  its own .opus extension.
- doRaw: a response larger than maxResponseBytes is now an error, not a
  silent truncation.
- Shared plumbing: requireBaseURL() + newRequest() helpers replace the
  7x-duplicated guard/error string and the triplicated request
  building across doJSON/doRaw/Health.
- Health: non-2xx now returns *llm.APIError (package convention,
  programmatically distinguishable from transport failure) instead of
  a one-off unexported error type.
- Speak: reject negative Speed; speechMIME no longer accepts video/*
  Content-Types.
- image.go: Generate/Edit share one sdWire validate+map helper.

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01AXQxVhXBw8PwFAtsVrXSmj
2026-07-11 23:46:23 -04:00
steveandClaude Fable 5 434d721b99 feat: audio surfaces (TTS + transcription), imagegen.Editor, llamaswap health probe
CI / Tidy (pull_request) Successful in 10m9s
CI / Build & Test (pull_request) Successful in 11m16s
Adversarial Review (Gadfly) / review (pull_request) Successful in 14m51s
- New leaf package `audio` (ADR-0017): SpeechModel/SpeechProvider and
  TranscriptionModel/TranscriptionProvider with imagegen conventions
  (zero value = backend default, functional options + Apply, bytes
  in/out, never URLs). Root re-exports added.
- imagegen.Editor (ADR-0018): optional image-to-image interface —
  EditRequest carries the generation knobs plus Init image and
  denoising Strength; separate interface so existing Models keep
  compiling.
- provider/llamaswap implements all of it: POST /v1/audio/speech (JSON,
  raw-audio response, MIME from Content-Type with format fallback),
  POST /v1/audio/transcriptions (multipart, response_format=json),
  ListVoices (GET /v1/audio/voices?model=, tolerant of string-list and
  object-list shapes), POST /sdapi/v1/img2img (txt2img wire +
  init_images/denoising_strength, shared image decode), and Health(ctx)
  (GET /health) — a cheap liveness probe for often-offline hosts.
- Hermetic httptest coverage for every new wire shape and validation
  path; README sections + support-matrix footnote updated in the same
  commit (also corrects the stale /v1/images/generations claim — the
  image path has been SDAPI since the seed fix).

First consumer: mort's llamaswap media tool cluster (status / image /
TTS / STT agent tools against the netherstorm host).

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01AXQxVhXBw8PwFAtsVrXSmj
2026-07-11 23:24:14 -04:00
steveandClaude Opus 4.8 6abb399f5b fix(agent): recover front-loaded answer past a citations-only terminal turn (#11)
CI / Tidy (push) Successful in 9m35s
CI / Build & Test (push) Successful in 10m31s
finalOutput now recovers the front-loaded answer when the terminal turn is a
sources/citations-only addendum ("Sources: [x](url), ..."), not just when it is
empty or a back-reference. It recovers the prior substantive answer and appends
the (real) citations below it. Guards: citation-DOMINANCE (a prose answer that
merely opens with "Source: ... http://..." is left as the answer), ^-anchored
heading, citations recovery decoupled from the terminal-length ratio (concise
answers recover too), preamble filter applied only in the borderline band
(long answers opening with "Sure,"/"Let me" are not vetoed), and a dedup that
ignores <url> angle-bracket wrappers. Healthy terminal answers unchanged; zero
extra model calls.

Fixes mort #1418. Gadfly-reviewed (6 reviewers) + adversarially pre-verified;
all findings graded, real ones addressed.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-07-10 16:47:32 +00:00
steve fe44a6da26 Merge pull request 'fix(llamaswap): use A1111 /sdapi/v1/txt2img so seed is honored' (#10) from fix/llamaswap-a1111-seed into main
CI / Tidy (push) Successful in 9m25s
CI / Build & Test (push) Successful in 9m42s
2026-06-29 03:09:29 +00:00
steve a213c18263 fix(llamaswap): use A1111 /sdapi/v1/txt2img so seed is honored
CI / Tidy (pull_request) Successful in 9m24s
CI / Build & Test (pull_request) Successful in 9m45s
Adversarial Review (Gadfly) / review (pull_request) Successful in 11m30s
The OpenAI /v1/images/generations endpoint ignores `seed` on our
stable-diffusion.cpp build — every render of a given prompt comes back
byte-identical, so a drawbot batch of N collapsed to one image. Switch the
image provider to sd-server's A1111 /sdapi/v1/txt2img endpoint, which honors
`seed` (verified live: distinct seeds -> distinct images on SDXL and
Qwen-Image). Size is split into width/height; llama-swap still routes by the
`model` field. Tests + ADR-0016 updated.
2026-06-28 22:56:25 -04:00
Steve Dudenhoeffer 88d3fc3279 chore: repin gadfly reusable to @5007597 (structured findings + consensus + inline review)
Adopts gadfly's review-representation overhaul: one ranked consensus comment
across the swarm + an advisory COMMENT-state inline PR review, on image
sha-3095ebf. Swarm config still rides the owner variables.

[skip ci]
2026-06-28 22:13:24 -04:00
steve 531fe1922e Merge pull request 'feat(imagegen): optional per-request generation settings' (#9) from feat/imagegen-settings into main
CI / Tidy (push) Successful in 9m25s
CI / Build & Test (push) Successful in 9m47s
2026-06-29 02:00:40 +00:00
steve a744cdc335 feat(imagegen): optional per-request generation settings
CI / Tidy (pull_request) Successful in 9m27s
CI / Build & Test (pull_request) Successful in 9m47s
Add Steps, CFGScale, NegativePrompt, Sampler, Seed to imagegen.Request
(pointer/empty = leave the backend's per-model default), with mirror
options, and forward them in the llamaswap wire payload as the
stable-diffusion.cpp fields (steps/cfg_scale/negative_prompt/
sample_method/seed). Unset fields are omitted so sd-server keeps its
baked defaults.

Lets callers (e.g. mort drawbots) override only what they explicitly set.
2026-06-28 19:05:49 -04:00
steve 8b924700fb Merge pull request 'fix(media): drop oldest images on over-count instead of refusing' (#8) from fix/image-overflow-drop-oldest into main
CI / Tidy (push) Successful in 9m23s
CI / Build & Test (push) Successful in 9m45s
2026-06-28 22:43:20 +00:00
steveandClaude Opus 4.8 70b7aebd86 test(media): match the overflow placeholder by const, not substring (gadfly #8)
CI / Tidy (pull_request) Successful in 9m25s
CI / Build & Test (pull_request) Successful in 9m49s
ragnaros/qwen3.6-27b noted TestNormalizeOverCount matched 'omitted' by substring;
the test is in-package, so assert == imageOverflowPlaceholder instead — robust to
wording changes. No behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-28 18:33:01 -04:00
steveandClaude Opus 4.8 52bb910f4d media: address gadfly review — single-pass elide, drop helpers, stronger test
CI / Tidy (pull_request) Successful in 9m27s
CI / Build & Test (pull_request) Successful in 9m41s
Review fixes (no behavior change):
- Fold the over-cap elide INTO the existing copy-on-write normalize pass: one
  loop now replaces the first toElide (oldest) images with the placeholder and
  size-normalizes the rest, so the Messages slice is copied at most once (the
  prior dropOldestImages + the normalize loop double-copied when overflow and a
  transform both applied — the dominant review finding, 5 models).
- Remove dropOldestImages (the name implied removal; it substituted) and the
  one-shot hasImagePart helper — both subsumed by the single pass.
- Trim the 9-line inline comment that restated the package doc.
- Test: rename TestNormalizeTooManyImages_DropsOldest → TestNormalizeOverCount
  (file convention) and assert the EXACT survivors ([b, c], in order) + a
  content-based non-mutation check (first input part is still image a, which a
  len check wouldn't catch).

Build + media + majordomo suites green (-race).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-28 18:06:17 -04:00
steveandClaude Opus 4.8 d71aca4c3a fix(media): drop oldest images on over-count instead of refusing the request
Adversarial Review (Gadfly) / review (pull_request) Has been cancelled
CI / Tidy (pull_request) Successful in 9m27s
CI / Build & Test (pull_request) Successful in 9m44s
media.Normalize refused (ErrUnsupported) when a request carried more images than
the target's MaxImagesPerReq, on the theory that a failover chain would try a
roomier target. In practice the chain's targets share the same cap — an agent loop
that accumulates a preview image per iteration (e.g. scaddy's write_scad) blows
past the cap, EVERY target rejects ("9 images, target allows at most 8"), and the
run dies. Observed live on ollama-cloud (cap 8).

Now: over-count keeps the most-recent MaxImagesPerReq images and replaces each
older one with a short text placeholder ("[earlier image omitted to fit this
model's per-request image limit]"), preserving each message's turn structure and
telling the model an image was elided. The most-recent images are the relevant
ones in an iterative run. Copy-on-write; the input request is never mutated. The
per-model threshold stays configurable via Capabilities.MaxImagesPerReq (0 still
means no image support); SupportsImages / MIME / byte-budget / dimension behavior
is unchanged, and the provider-side count backstop remains.

Test: TestNormalizeTooManyImages_DropsOldest — 3 images, cap 2 → 2 kept (the most
recent), 1 placeholder, no error, oldest dropped, input unmutated.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-28 17:38:21 -04:00
Steve DudenhoefferandClaude Opus 4.8 51f5ea0d2b ci: pin gadfly reusable to immutable @7bc3c98 (vars-config reusable) [skip ci]
The reusable now reads swarm config from user-scope vars (GADFLY_DEFAULT_* +
GADFLY_ENDPOINT_*); this immutable @sha bumps past the long-lived-runner ref
cache so the vars-config reusable is adopted. Direct to main + [skip ci] to
avoid triggering the review swarm.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-28 02:05:29 -04:00
steve a457e76ac7 ci: track gadfly's v1 release tag instead of a pinned sha (#7)
CI / Tidy (push) Successful in 9m25s
CI / Build & Test (push) Successful in 9m45s
2026-06-28 04:08:34 +00:00
steve 78a1d1c3bb ci: switch gadfly review to the reusable workflow (curated swarm, 5 lenses) (#6)
CI / Tidy (push) Successful in 9m25s
CI / Build & Test (push) Successful in 10m13s
2026-06-28 02:48:28 +00:00
steve aa25b2c334 Merge pull request 'feat(llamaswap): add llama-swaps (TLS) DSN scheme' (#4) from feat/llama-swaps-tls into main
CI / Tidy (push) Successful in 9m23s
CI / Build & Test (push) Successful in 10m13s
2026-06-27 22:56:59 +00:00
steve 2b35f1741c Merge pull request 'ci(gadfly): trim the weakest reviewers from the swarm' (#5) from ci/trim-gadfly-reviewers into main
CI / Tidy (push) Successful in 9m25s
CI / Build & Test (push) Successful in 10m1s
2026-06-27 22:56:57 +00:00
steveandClaude Opus 4.8 de2b2f0f28 feat(llamaswap): add llama-swaps (TLS) DSN scheme
CI / Tidy (pull_request) Successful in 9m43s
CI / Build & Test (pull_request) Successful in 10m26s
Adversarial Review (Gadfly) / review (pull_request) Successful in 11m47s
llama-swap was http-only by DSN, pushing TLS-fronted instances onto the openai://
scheme (which loses the management/image methods). Add a "llama-swaps" scheme
that builds an https base URL, alongside "llama-swap" (http, local-first) —
mirroring redis/rediss. Both share one factory; llama-swaps is scheme-only (no
default built-in). The choice stays explicit because a DSN has no reliable
http-vs-https signal.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-27 17:58:59 -04:00
97 changed files with 11715 additions and 226 deletions
+2
View File
@@ -6,6 +6,8 @@ OLLAMA_API_KEY=your-ollama-cloud-key-here
# Built-in provider keys (each optional; only needed for the providers you use).
#OPENAI_API_KEY=sk-...
#KIMI_API_KEY=sk-... # Moonshot AI (Kimi); provider name "kimi"
#QWEN_API_KEY=sk-... # Alibaba Model Studio (Qwen); provider name "qwen"
#ANTHROPIC_API_KEY=sk-ant-...
#GOOGLE_API_KEY=...
+26 -50
View File
@@ -1,12 +1,8 @@
# Gadfly — agentic adversarial PR reviewer (https://gitea.stevedudenhoeffer.com/steve/gadfly).
#
# Runs the published Gadfly image (pinned to an immutable :sha- tag — act_runner
# caches :latest, and this build is what carries foreman provider-type support)
# as a specialist swarm and posts
# ONE consolidated review comment as gitea-actions. Advisory only — never blocks a
# merge. This reviews majordomo PRs with 6 ollama-cloud models (3-lens suite).
# Gadfly is a simple system — findings are advisory; always double-check before
# acting.
# Gadfly adversarial review — subscribes to steve/gadfly's reusable workflow and
# INHERITS its default swarm. This stub holds only the triggers, the actor gate,
# secret forwarding, and the allow-list; the swarm config (models, lenses,
# concurrency, timeouts) lives centrally in gadfly's review-reusable.yml so it is
# tuned in ONE place. Advisory only — never blocks a merge.
name: Adversarial Review (Gadfly)
@@ -33,50 +29,30 @@ concurrency:
jobs:
review:
# Security: only trusted users may trigger a secret-bearing run via a PR
# comment (pull_request + workflow_dispatch are already trusted). Mirrors
# GADFLY_ALLOWED_USERS, the in-container belt-and-suspenders check.
# comment (pull_request + workflow_dispatch are already trusted). Mirrors the
# allowed_users input below (the in-container belt-and-suspenders check) — both
# lists must stay in sync; a workflow if: can't read a workflow_call input.
if: >-
github.event_name != 'issue_comment'
|| (github.event.issue.pull_request
&& (github.actor == 'steve'
|| github.actor == 'fizi'
|| github.actor == 'dazed'))
runs-on: ubuntu-latest
# Fleet: 6 ollama-cloud models (lens fan-out), no local Macs. (Trimmed the
# weakest reviewers by grade — m5/qwen3.6, gemma4, gpt-oss, kimi-k2.7 — plus
# the earlier M1 drop.) Plenty of headroom for the cloud lanes.
timeout-minutes: 45
steps:
- uses: docker://gitea.stevedudenhoeffer.com/steve/gadfly:sha-d7f364d
env:
GITEA_API: ${{ github.server_url }}/api/v1/repos/${{ github.repository }}
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
OLLAMA_CLOUD_API_KEY: ${{ secrets.OLLAMA_CLOUD_API_KEY }}
# Cloud-only fleet (no local Macs). Cloud concurrency lives in the
# LENSES: models run a few at a time (ollama-cloud=3) with their 3 lenses
# concurrent (LENS ollama-cloud=3) so comments land sooner.
GADFLY_MODELS: "minimax-m3:cloud,glm-5.2:cloud,glm-5.1:cloud,deepseek-v4-pro:cloud,nemotron-3-super:cloud,qwen3-coder:480b-cloud"
GADFLY_PROVIDER_CONCURRENCY: "ollama-cloud=3"
GADFLY_PROVIDER_LENS_CONCURRENCY: "ollama-cloud=3"
# Default => the 3-lens suite (security, correctness, error-handling).
# Set the repo var GADFLY_SPECIALISTS to override (csv / "all" / "auto").
GADFLY_SPECIALISTS: ${{ vars.GADFLY_SPECIALISTS || 'security,correctness,error-handling' }}
# Per-lens deadline + bounded steps to keep each reviewer's run sane.
GADFLY_TIMEOUT_SECS: "600"
GADFLY_MAX_STEPS: "14"
# Allow-list for the comment trigger (mirrors the job-level if: guard).
GADFLY_ALLOWED_USERS: "steve,fizi,dazed"
# --- findings telemetry: POST runs + findings to the gadfly-reports store ---
# Advisory & off unless GADFLY_FINDINGS_URL is set; failures only log to
# stderr and never affect the review. GADFLY_REPO / GADFLY_PR are derived
# in-container; the URL + token are user-scope secrets.
GADFLY_FINDINGS_URL: ${{ secrets.GADFLY_FINDINGS_URL }}
GADFLY_FINDINGS_TOKEN: ${{ secrets.GADFLY_FINDINGS_TOKEN }}
# --- event context (leave as-is) ---
EVENT_NAME: ${{ github.event_name }}
PR: ${{ github.event.pull_request.number || github.event.issue.number || github.event.inputs.pr_number }}
PR_BRANCH: ${{ github.head_ref }}
IS_DRAFT: ${{ github.event.pull_request.draft }}
COMMENT_BODY: ${{ github.event.comment.body }}
COMMENT_ID: ${{ github.event.comment.id }}
ACTOR: ${{ github.actor }}
# Pinned to an immutable gadfly commit (not @v1): our act_runners are long-lived
# and cache the reusable-workflow ref, so a moved v1 tag keeps resolving to the
# stale cached copy. A unique sha forces a cache miss → fresh fetch. Bump this
# sha to adopt central swarm changes.
uses: steve/gadfly/.gitea/workflows/review-reusable.yml@c9dab69d143cb614c1840a5b06d6ffc358f4752d
# Least privilege: forward only the review secrets (not `secrets: inherit`,
# which would expose every repo secret). GITEA_TOKEN is the automatic token.
secrets:
OLLAMA_CLOUD_API_KEY: ${{ secrets.OLLAMA_CLOUD_API_KEY }}
CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
GADFLY_FINDINGS_URL: ${{ secrets.GADFLY_FINDINGS_URL }}
GADFLY_FINDINGS_TOKEN: ${{ secrets.GADFLY_FINDINGS_TOKEN }}
with:
# Consumer-specific allow-list; everything else is inherited.
allowed_users: "steve,fizi,dazed"
# Gitea >= 1.27 does not propagate dispatch inputs into a called workflow's
# github.event — thread the PR number explicitly (empty on non-dispatch events).
pr_number: ${{ github.event.inputs.pr_number }}
+19 -10
View File
@@ -34,6 +34,11 @@ majordomo Registry, Parse, env-DSN loading, chain executor, re-exports
Tool/Toolbox, Capabilities, Stream, Model, Provider, errors
imagegen/ canonical text-to-image contract: Request/Result/Model/
Provider (separate from llm; Image = llm.ImagePart) (ADR-0016)
audio/ canonical speech contracts: SpeechModel (TTS) +
TranscriptionModel (STT), same conventions (ADR-0017)
videogen/ canonical video-generation contract: one Model for
text-to-video + image-to-video (InitImage), single-clip
Result (Image = llm.ImagePart) (ADR-0019)
health/ clock-injected health tracker (bench/backoff)
media/ image normalization to target capabilities (sniff real
format, downscale, transcode, byte ladder; ErrUnsupported
@@ -44,7 +49,8 @@ majordomo Registry, Parse, env-DSN loading, chain executor, re-exports
provider/ollama/ one native /api/chat client serving the ollama,
ollama-cloud, and foreman built-ins via presets
provider/llamaswap/ llama-swap proxy: chat delegates to provider/openai,
plus management methods + imagegen image client (ADR-0015)
plus management methods + imagegen/audio/videogen
clients (ADR-0015..0019)
provider/google/ Gemini on google.golang.org/genai (the one approved
dependency; lazy client, raw-JSON-schema tools,
ThinkingLevel reasoning, iter.Pull2 streaming)
@@ -79,12 +85,13 @@ alias := bare token (no slash), expands INLINE, recursively, cycle-checked
`LLM_<NAME>=scheme://[token@]host[/path]` — e.g.
`LLM_M5=foreman://[email protected]` defines provider `m5`; then
`m5/qwen3:30b` works in Parse, chains, and aliases. Scheme ∈ {foreman,
ollama, ollama-cloud, openai, anthropic, google, gemini, llama-swap}
RegisterScheme. Token = credential; base URL = `https://host` always —
**except `llama-swap`, which builds `http://host` (local-first; ADR-0015).**
`New()` scans the process env eagerly; unknown names also resolve lazily at
Parse time (`my-prov``LLM_MY_PROV`). Malformed entries fail on use, not at
startup.
ollama, ollama-cloud, openai, anthropic, google, gemini, llama-swap,
llama-swaps} RegisterScheme. Token = credential; base URL = `https://host`
always — **except `llama-swap`, which builds `http://host` (local-first);
`llama-swaps` is its TLS twin (`https://host`), mirroring redis/rediss
(ADR-0015).** `New()` scans the process env eagerly; unknown names also resolve
lazily at Parse time (`my-prov``LLM_MY_PROV`). Malformed entries fail on use,
not at startup.
## Health & failover (ADR-0006, ADR-0008)
@@ -141,9 +148,11 @@ Ship work through PRs and let Gadfly review it before merge:
- **Push to a PR, never straight to `main`.** Branch, push, open a PR.
`.gitea/workflows/adversarial-review.yml` runs Gadfly (the standalone
agentic adversarial reviewer) — a fleet of 6 ollama-cloud models, each
running the 3-lens suite (security, correctness, error-handling). Advisory
only; it never blocks the merge.
agentic adversarial reviewer) by subscribing to gadfly's reusable workflow
and inheriting its default swarm — 3 cloud models + the Claude Code engine
(sonnet/opus/opus:max), each running the 5-lens suite (security, correctness,
maintainability, performance, error-handling). The swarm is tuned centrally
in gadfly, not here. Advisory only; it never blocks the merge.
- **Wait for Gadfly to finish, then read its output.** Don't merge while the
review is still running. Each model posts one consolidated comment; weigh
every finding on its merits and fix the real ones (Gadfly is a simple
+119 -12
View File
@@ -121,6 +121,8 @@ Chains are health-tracked per target:
| Provider | Spec name | Key env var | Default endpoint |
|----------|-----------|-------------|------------------|
| OpenAI (+compatible) | `openai` | `OPENAI_API_KEY` | https://api.openai.com/v1 |
| Kimi (Moonshot AI) | `kimi` | `KIMI_API_KEY` | https://api.moonshot.ai/v1 |
| Qwen (Alibaba) | `qwen` | `QWEN_API_KEY` | https://dashscope-intl.aliyuncs.com/compatible-mode/v1 |
| Anthropic (+compatible) | `anthropic` | `ANTHROPIC_API_KEY` | https://api.anthropic.com |
| Google (Gemini) | `google` | `GOOGLE_API_KEY` / `GEMINI_API_KEY` | Gemini API (official SDK) |
| Ollama Cloud | `ollama-cloud` | `OLLAMA_API_KEY` | https://ollama.com |
@@ -128,6 +130,24 @@ Chains are health-tracked per target:
| foreman | `foreman` | — (token via DSN) | requires an LLM_* DSN or `ollama.Foreman(url, token)` |
| llama-swap | `llama-swap` | — (token via DSN) | requires an LLM_* DSN or `llamaswap.New(...)` |
Kimi is Moonshot AI's OpenAI-compatible Chat Completions endpoint, so it reuses
the openai client (like llama-swap). The `kimi` built-in defaults to the
international endpoint; reach the China endpoint (or any other host) with a
`kimi://` DSN, e.g. `LLM_KCN=kimi://[email protected]/v1`.
Qwen is the same shape: Alibaba Model Studio's OpenAI-compatible mode, reusing
the openai client. The `qwen` built-in defaults to the international
(Singapore) host; reach the China host or a workspace-scoped regional one with
a `qwen://` DSN, e.g.
`LLM_QCN=qwen://[email protected]/compatible-mode/v1`. Model Studio
also fronts the same models with an Anthropic-compatible `/v1/messages` shim —
majordomo does **not** use it, because on that surface `reasoning_effort` is
dropped, `Request.Schema` stops being enforced, and cached-token accounting
disappears; see [ADR-0027](docs/adr/0027-qwen-builtin.md). Two Alibaba-side
quirks are worth knowing: thinking is on by default for some models (e.g.
`qwen3.7-plus`), and the Qwen3 open-source models require streaming while
thinking, so buffered `Generate` calls want a Max/Plus model.
OpenAI-compatible / Anthropic-compatible endpoints: construct the provider
with a name and base URL and register it —
@@ -159,21 +179,25 @@ m, _ := reg.Parse("m5/qwen3:30b,m1/qwen3:30b,thinking")
```
DSN format: `scheme://[token@]host[/path]`, scheme ∈ `foreman`, `ollama`,
`ollama-cloud`, `openai`, `anthropic`, `google`/`gemini`, `llama-swap`, or any
scheme you add with `RegisterScheme`. The token is the credential (bearer token
/ API key); the base URL is always `https://host[/path]` — except `llama-swap`,
which builds `http://host[:port]` since it's local-first. `New()` loads `LLM_*`
vars eagerly; unknown provider names also resolve lazily at Parse time
(`my-prov/x``LLM_MY_PROV`).
`ollama-cloud`, `openai`, `kimi`, `qwen`, `anthropic`, `google`/`gemini`, `llama-swap`,
`llama-swaps`, or any scheme you add with `RegisterScheme`. The token is the
credential (bearer token / API key); the base URL is always `https://host[/path]`
— except `llama-swap`, which builds `http://host[:port]` since it's local-first
(`llama-swaps` is the TLS twin → `https://host`, mirroring redis/rediss). `New()`
loads `LLM_*` vars eagerly; unknown provider names also resolve lazily at Parse
time (`my-prov/x``LLM_MY_PROV`).
```
LLM_LS=llama-swap://[email protected]:8080 # then "ls/qwen3:14b" parses
LLM_LS=llama-swap://[email protected]:8080 # http → "ls/qwen3:14b" parses
LLM_LS=llama-swaps://[email protected] # https → TLS-fronted instance
```
[llama-swap](https://github.com/mostlygeek/llama-swap) is a model-swapping proxy
over llama.cpp. Its chat API is OpenAI-compatible (majordomo reuses the openai
client), and the `*llamaswap.Provider` adds management methods
(`ListModels`/`Running`/`Unload`) plus image generation (see below). A cold
(`ListModels`/`Running`/`Unload`/`ListVoices`), a cheap liveness probe
(`Health`, GET /health — the proxy answers without touching a model), image
generation and editing, and speech synthesis/transcription (see below). A cold
model swap can take many seconds — bound calls with a context deadline, not a
client timeout.
@@ -209,8 +233,9 @@ resp, err := m.Generate(ctx, majordomo.Request{
Text-to-image is a separate contract (`imagegen`) from chat, because it shares
none of the message/tool/stream machinery. Generated images come back as
`llm.ImagePart`, so they drop straight back into a chat turn. The first backend
is llama-swap (OpenAI `/v1/images/generations` a stable-diffusion.cpp
upstream).
is llama-swap (the A1111-style `/sdapi/v1/txt2img` on a stable-diffusion.cpp
upstream — chosen over OpenAI `/v1/images/generations` because that route
ignores `seed` there).
```go
ls := llamaswap.New(llamaswap.WithBaseURL("http://box.local:8080"))
@@ -222,9 +247,76 @@ res, err := im.Generate(ctx, imagegen.Request{Prompt: "a red bicycle"},
// majordomo.UserParts(majordomo.Text("describe this"), res.Images[0])
```
Image-to-image editing is the optional `imagegen.Editor` interface (ADR-0018)
— llama-swap's image models implement it via `/sdapi/v1/img2img`:
```go
ed := im.(imagegen.Editor)
res, err := ed.Edit(ctx, imagegen.EditRequest{
Prompt: "make it night",
Init: res.Images[0], // any llm.ImagePart
}, imagegen.WithEditStrength(0.6)) // 0..1: how far to depart from Init
```
`*llamaswap.Provider` also exposes management methods: `ListModels` (what
llama-swap can serve), `Running` (what's loaded), and `Unload` (free a model).
## Speech: synthesis + transcription
Text-to-speech and speech-to-text live in the `audio` package (ADR-0017),
mirroring imagegen: small `SpeechModel`/`TranscriptionModel` contracts,
zero values mean backend defaults, bytes in/out (never URLs). First backend:
llama-swap (OpenAI `/v1/audio/speech` + `/v1/audio/transcriptions` routed to
kokoro/whisper.cpp-style upstreams).
```go
ls := llamaswap.New(llamaswap.WithBaseURL("http://box.local:8080"))
sm, _ := ls.SpeechModel("kokoro")
speech, err := sm.Speak(ctx, audio.SpeechRequest{Input: "hello world"},
audio.WithVoice("af_heart"), audio.WithFormat("mp3"))
// speech.Audio ([]byte) + speech.MIME ("audio/mpeg")
tm, _ := ls.TranscriptionModel("whisper-large-v3-turbo")
tr, err := tm.Transcribe(ctx, audio.TranscriptionRequest{
Audio: speech.Audio, MIME: speech.MIME,
})
// tr.Text
voices, err := ls.ListVoices(ctx, "kokoro") // []string of voice ids
```
## Video: text-to-video, image-to-video, first-last-frame
Video generation lives in the `videogen` package (ADR-0019), mirroring
imagegen/audio: one small `Model` contract, zero values mean backend
defaults, bytes in/out. All modes are one surface, selected by which
keyframes are set rather than by a mode flag:
| `InitImage` | `LastImage` | mode |
|---|---|---|
| nil | nil | text-to-video |
| set | nil | image-to-video (hybrid checkpoints like Wan 2.2 TI2V serve both) |
| set | set | first-last-frame — both ends pinned |
| nil | set | pin the destination, model invents the approach |
First backend: llama-swap (blocking `/v1/videos/sync`, vLLM-Omni style — the
response body is the encoded clip, so `Result` carries a single `Video`).
Generation runs for minutes; bound the call with a context deadline.
**`LastImage` support is per-model and cannot be detected.** A backend that
does not understand a trailing keyframe ignores the part and returns an
ordinary clip — indistinguishable from success. There is no capability bit,
because the contract has no way to learn one, so a caller depending on the
pin must establish support out of band.
```go
vm, _ := ls.VideoModel("videogen-wan22-5b")
res, err := vm.Generate(ctx, videogen.Request{Prompt: "a cat surfing"},
videogen.WithSize("1280x704"), videogen.WithNumFrames(81))
// res.Video.Data ([]byte) + res.Video.MIME ("video/mp4")
```
## Tool calls
```go
@@ -341,6 +433,8 @@ to build one.
| Provider | Resolve/Parse | Chat | Streaming | Tools | Structured | Images | Env DSN |
|----------------------|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| OpenAI (+compatible) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Kimi (Moonshot AI) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅³ | ✅ |
| Qwen (Alibaba) | ✅ | ✅ | ✅ | ✅ | ✅⁴ | ✅⁴ | ✅ |
| Anthropic (+compat) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Google (Gemini) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Ollama Cloud | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
@@ -356,8 +450,21 @@ response as a single delta plus final event.
² llama-swap's chat is OpenAI-compatible and reuses the openai client, so these
capabilities are present at the client level; whether a given call succeeds
depends on the llama.cpp model llama-swap loads. llama-swap also provides
**image generation** (a separate `imagegen` axis, not shown above) and
management methods on `*llamaswap.Provider`.
**image generation + editing** (`imagegen`), **speech synthesis +
transcription** (`audio`), **video generation** (`videogen`) — separate
axes, not shown above — plus a `Health`
probe and management methods on `*llamaswap.Provider`.
³ Kimi reuses the openai client, so image *inputs* are supported at the client
level; whether a call succeeds depends on the Moonshot model — only the vision
variants (e.g. `moonshot-v1-8k-vision-preview`) accept images.
⁴ Qwen also reuses the openai client (ADR-0027), so both columns are present at
the client level and gated by the Model Studio model you name: `json_schema`
structured output is on the Max/Plus families, image inputs on the `qwen-vl-*`
/ `qwen3-vl-*` models. `reasoning_effort` rides through as a top-level field —
one reason the built-in speaks OpenAI-compat rather than Model Studio's
Anthropic-compat shim.
Notes: Ollama has no native tool_choice — `"none"` drops the tools;
`"required"`/named choices are best-effort ignored there. Ollama Cloud
+56 -13
View File
@@ -125,10 +125,12 @@ func WithCompactor(fn func(ctx context.Context, msgs []llm.Message) ([]llm.Messa
}
// WithToolErrorLimits installs loop guards: maxConsecutiveErrors bounds
// successive steps whose tool results were ALL errors, and
// maxSameCallRepeats bounds identical (name + arguments) tool calls within
// one run. Either guard tripping ends the run with ErrToolLoop and the
// partial result. Zero disables a guard.
// successive steps whose tool results were ALL errors, and maxSameCallRepeats
// bounds identical (name + arguments) tool calls that ALSO return an unchanged
// result within one run — a call whose result keeps advancing (e.g. polling a
// long-running background job) is progress and never trips this guard. Either
// guard tripping ends the run with ErrToolLoop and the partial result. Zero
// disables a guard.
func WithToolErrorLimits(maxConsecutiveErrors, maxSameCallRepeats int) Option {
return func(a *Agent) {
a.maxConsecutiveToolErrors = maxConsecutiveErrors
@@ -251,6 +253,15 @@ func (a *Agent) mergedTools() (map[string]llm.Tool, []llm.Tool, error) {
// Run executes the loop: send the conversation; while the model requests
// tools, execute them and feed results back; stop on a final answer,
// MaxSteps, or an unrecoverable model error.
// repeatState is the per-signature bookkeeping for the progress-aware same-call
// guard: count is the run-length of consecutive identical calls that returned
// lastResult (the previous call's encoded result). A changed result resets count
// to 1. See the guard block in Run.
type repeatState struct {
count int
lastResult string
}
func (a *Agent) Run(ctx context.Context, input string, opts ...RunOption) (*Result, error) {
var rc runConfig
for _, opt := range opts {
@@ -274,9 +285,12 @@ func (a *Agent) Run(ctx context.Context, input string, opts ...RunOption) (*Resu
reqOpts := append(append([]llm.Option(nil), a.reqOpts...), rc.reqOpts...)
system := a.systemPrompt()
// Loop-guard state (WithToolErrorLimits).
// Loop-guard state (WithToolErrorLimits). repeatStates tracks, per identical
// (name+arguments) signature, the run-length of consecutive calls that
// returned the same result and that last result — see the same-call guard
// below.
consecutiveErrorSteps := 0
callCounts := make(map[string]int)
repeatStates := make(map[string]*repeatState)
maxSteps := func() int {
if a.maxStepsFunc != nil {
@@ -330,13 +344,6 @@ func (a *Agent) Run(ctx context.Context, input string, opts ...RunOption) (*Resu
result.Messages = msgs
return result, err
}
if a.maxSameCallRepeats > 0 {
sig := call.Name + "\x00" + string(call.Arguments)
callCounts[sig]++
if callCounts[sig] > a.maxSameCallRepeats {
repeatTripped = call.Name
}
}
tool, ok := byName[call.Name]
if !ok {
results = append(results, llm.ToolResult{
@@ -356,6 +363,42 @@ func (a *Agent) Run(ctx context.Context, input string, opts ...RunOption) (*Resu
a.notify(rc, step)
msgs = append(msgs, llm.ToolResultsMessage(results...))
// Same-call repeat guard (progress-aware). An identical (name+arguments)
// call only counts toward the loop trip when its result is unchanged from
// the previous identical call: a call whose result keeps advancing —
// canonically polling a long-running background job — is progress and
// resets its count, while a genuinely stuck call returning the same output
// trips once it exceeds the ceiling. Result equality is exact-string on
// the full encoded content, chosen to err toward NOT tripping: a hung job
// whose poll still reports a ticking field is left to MaxRuntime / the
// job's own ceiling rather than risking a false kill of real progress.
// results[i] pairs with resp.ToolCalls[i] — every call appends exactly one
// result (unknown tools append an error result before continue) and the
// only early exit above is a full return on ctx cancellation.
if a.maxSameCallRepeats > 0 {
for i, call := range resp.ToolCalls {
sig := call.Name + "\x00" + string(call.Arguments)
resKey := results[i].Content
if results[i].IsError {
resKey = "e\x00" + resKey
}
st := repeatStates[sig]
if st == nil {
st = &repeatState{}
repeatStates[sig] = st
}
if st.count > 0 && st.lastResult == resKey {
st.count++
} else {
st.count = 1
}
st.lastResult = resKey
if st.count > a.maxSameCallRepeats {
repeatTripped = call.Name
}
}
}
if repeatTripped != "" {
result.Messages = msgs
return result, fmt.Errorf("%w: %q called identically more than %d times",
+236 -18
View File
@@ -14,45 +14,192 @@ import (
// their answer to the final, tool-free turn. But some models — notably several
// open-weight ones — "front-load" their full answer into an earlier turn that
// ALSO calls a tool (e.g. answer text alongside a citation call), then close
// with a trivial pointer such as "(Already answered above.)". Returning only
// the terminal text would discard the real answer, which is still present
// earlier in the transcript. When the terminal text is weak (empty, or a short
// back-reference) fall back to the last substantive assistant content in msgs.
// with a degenerate terminal turn that is not itself the answer. Three shapes
// are recovered from the transcript (zero extra model calls):
//
// - a trivial back-reference ("(Already answered above.)", "see above", …):
// the real answer sits earlier, so recover it and DISCARD the worthless
// closer.
// - a sources/citations-only addendum ("Sources: [x](…), [y](…)"): the model
// front-loaded the prose answer and closed with just its citations (the
// glm-5.2 "cite" pattern behind mort issue #1418). The citations are real,
// useful content — unlike a back-reference — so recover the prior answer and
// KEEP the citations, appended below it.
// - a bookkeeping closer ("Citations are logged. Short version: …"): the
// model acknowledged the citation round and compressed the answer it had
// already written into a one-liner (mort run b3cb9ee9 — a 2,089-char answer
// shrank to a 153-byte closer at delivery). The compression is strictly
// poorer than the front-loaded answer, so recover the prior turn and
// DISCARD the closer — but only when the prior turn clearly dwarfs it,
// because unlike a back-reference this closer DOES carry answer content
// (see modeSummary).
//
// A citations addendum is tested first and wins over the other two (a short
// terminal can match more than one shape), so its links are never discarded.
// The back-reference test wins over the summary-closer test: a terminal
// matching both ("Citations are logged. As I said above…") carries no answer
// content of its own, so the looser back-ref recovery bar — not the summary
// closer's dwarf ratio — is the right one. When the terminal text stands
// on its own it is returned unchanged; when it is degenerate but nothing
// better can be recovered, it is returned as-is (a compressed answer still
// beats nothing).
//
// msgs must already include the terminal assistant message as its last element
// (the loop appends it before calling this); terminal is that message's text.
func finalOutput(msgs []llm.Message, terminal string) string {
if !isWeakFinal(terminal) {
mode := modeBackRef
switch {
case isCitationsOnly(terminal):
mode = modeCitations
case isWeakFinal(terminal):
mode = modeBackRef
case isSummaryCloser(terminal):
mode = modeSummary
default:
return terminal
}
if rec, ok := lastSubstantiveAssistantText(msgs, terminal); ok {
return rec
rec, ok := lastSubstantiveAssistantText(msgs, terminal, mode)
if !ok {
return terminal
}
return terminal
if mode == modeCitations {
// Preserve the citations addendum below the recovered answer, unless the
// recovered turn already carries it (guards against a duplicate sources
// block when the front-loaded turn included its own citations). The
// containment test ignores <url> angle-bracket wrappers so a turn that
// listed the same sources unwrapped still suppresses the duplicate.
if tail := strings.TrimSpace(terminal); !strings.Contains(stripURLAngles(rec), stripURLAngles(tail)) {
return rec + "\n\n" + tail
}
}
return rec
}
// stripURLAngles removes the <…> wrappers Discord uses to suppress link embeds,
// so the citations dedup compares URLs regardless of that formatting delta.
func stripURLAngles(s string) string {
if !strings.ContainsAny(s, "<>") {
return s
}
return strings.NewReplacer("<", "", ">", "").Replace(s)
}
// recoveryMode selects the bar a prior assistant turn must clear to replace
// the terminal turn (see isSubstantiveAnswer) and what finalOutput does with
// the terminal once recovery succeeds.
type recoveryMode int
const (
// modeBackRef: the terminal is empty or a pure back-reference — worthless
// on its own, so any real prior answer replaces it and it is discarded.
modeBackRef recoveryMode = iota
// modeCitations: the terminal is a sources-only addendum — not a rival
// answer, so the dwarf ratio is skipped and the addendum is kept, appended
// below the recovered answer.
modeCitations
// modeSummary: the terminal acknowledges the citation round and may carry
// a short compression of the front-loaded answer. Unlike a back-reference
// it DOES contain answer content, so it is only replaced when a prior turn
// clearly dwarfs it — the ratio is mandatory at every length, the recovery
// scan stops at the most recent user message (a compression can only be of
// THIS turn's answer; never resurrect one from an earlier question), and
// the closer is discarded (its content is a strict subset of what it
// replaced).
modeSummary
)
// backRefRe matches a terminal turn that merely points back to an earlier
// message instead of stating the answer ("(Already answered above.)",
// "see above", "as I said", ...).
var backRefRe = regexp.MustCompile(`(?i)(already answered|see above|as (i )?(said|mentioned|stated|noted)|answered (that )?above|per my (previous|earlier))`)
// summaryCloserRe matches a terminal turn that OPENS with a bookkeeping
// acknowledgment of the citation round — "Citations are logged.", "Sources
// cited.", "Logged the citations." — the shape a model produces when it
// front-loaded its answer into an earlier cite-call turn and closes by
// acknowledging the tool results, often followed by a "Short version: …"
// compression of the answer it already wrote. The ack clause must end at a
// sentence terminator ([.!]) DIRECTLY after the verb: "The citations are
// recorded in the court transcript…" is a real answer about citations, not
// bookkeeping, and must never match. A compression marker without the ack
// ("Short version: no.") is deliberately out of scope — a user who asked for
// brevity would be answered with exactly that shape, and misclassifying it
// would hijack a legitimate answer; an unmatched closer merely keeps today's
// behavior (fail closed). Assembled from named fragments so the alternations
// stay legible and extendable.
const (
summaryPreface = `((done|all set|ok(ay)?)[\s,.!:—-]+)?` // optional "Done —" style opener
summaryNouns = `(citations?|sources?|references?|claims?)`
// "all" appears here AND in summaryArticle on purpose: as a quantifier
// between noun and verb ("Citations all logged.") and as a determiner
// before the noun ("All claims cited.", "Logged all the citations.").
summaryCopulas = `((are|were|have\s+been|all)\s+)*`
summaryVerbs = `(logged|recorded|cited|saved|noted|captured|filed)`
summaryArticle = `((all|the)\s+)*` // star, not ?: "Logged all the citations."
)
var summaryCloserRe = regexp.MustCompile(`(?i)^` + leadingMarkers + summaryPreface +
`(` + summaryArticle + summaryNouns + `\s+` + summaryCopulas + summaryVerbs +
`|logged\s+` + summaryArticle + summaryNouns + `)[.!]`)
// preambleRe matches intent-announcing prefixes ("Let me search...", "I'll
// check...") so a preamble is never mistaken for the answer during recovery.
var preambleRe = regexp.MustCompile(`(?i)^(let me|let'?s|i'?ll|i will|first[, ]|sure[,. ]|okay[,. ]|on it|checking)`)
// citationLabelRe matches a terminal turn that OPENS with a sources/citations
// heading — the shape a model produces when it front-loads its answer into an
// earlier tool-call turn and closes with only its sources. Leading markdown
// emphasis (*, _), list (-, +, *), block-quote (>), and ATX-heading (#) markers
// — with their whitespace, since \s is in the class — are tolerated before the
// label, as is closing emphasis (** / __) plus whitespace between the label and
// the colon/dash separator. Anchored at ^ so a normal answer that merely
// mentions "sources" mid-sentence, or ends with a "Sources:" section AFTER its
// prose, is never matched.
var citationLabelRe = regexp.MustCompile(`(?i)^` + leadingMarkers +
`(sources?|references?|citations?|works cited|further reading)\b[\s*_]*[:\-—]`)
// leadingMarkers tolerates markdown noise before a label: emphasis (*, _),
// list (-, +, *), block-quote (>), and ATX-heading (#) markers, with their
// whitespace. Shared by citationLabelRe and summaryCloserRe so the two
// classifiers cannot drift apart (the first draft of the summary class
// dropped '+' by hand-copying this set).
const leadingMarkers = `[\s>#*_+-]*`
// linkRe matches a whole markdown link "[label](url)" or a bare URL. Used both
// to require that a citations terminal carries at least one link and to strip
// links out when measuring how citation-dominated the terminal is.
var linkRe = regexp.MustCompile(`\[[^\]]*\]\([^)]*\)|https?://\S+`)
// citationResidueCutset is trimmed from the ends of a citations terminal's
// non-link remainder before measuring it — list bullets, separators, and the
// short per-source annotations models add in parentheses.
const citationResidueCutset = " \t\r\n,;.:|·•*_()[]—-"
const (
// weakFinalMaxChars bounds how long a back-reference closer can be. A
// genuine final answer that merely contains "as I said" mid-sentence is
// longer than this, so it is never treated as weak.
weakFinalMaxChars = 120
// recoverMinChars: a prior assistant turn this long is treated as a real
// answer regardless of how it opens.
// answer regardless of how it opens (the preamble filter is not applied at
// this length — see isSubstantiveAnswer).
recoverMinChars = 200
// recoverFloorChars / recoverRatio gate the borderline band: a shorter
// prior turn must still clearly dwarf the (very short) terminal and not
// look like a preamble.
// prior turn must clear the floor and — unless the terminal is a citations
// addendum, which is not a rival answer — also clearly dwarf the (very
// short) terminal. See isSubstantiveAnswer.
recoverFloorChars = 80
recoverRatio = 3
// citationDominatedDivisor: a citations terminal's non-link remainder must
// be at most len/N of the whole, so a prose answer that merely opens with
// "Source:" and cites a URL mid-sentence is not mistaken for a bare list.
citationDominatedDivisor = 3
// summaryCloserMaxChars bounds a summary closer: room for the ack sentence
// plus a couple of compression sentences (the b3cb9ee9 closer was 153
// bytes — Go len(), which is what every threshold here compares). Beyond
// this the "short version" is substantial enough that replacing it risks
// losing content the front-loaded turn never had.
summaryCloserMaxChars = 300
)
// isWeakFinal reports whether a terminal turn's text fails to stand on its own
@@ -65,13 +212,65 @@ func isWeakFinal(s string) bool {
return len(t) <= weakFinalMaxChars && backRefRe.MatchString(t)
}
// isCitationsOnly reports whether a terminal turn is essentially just a
// sources/citations addendum: it OPENS with a citations heading, carries at
// least one link, and — once the heading and links are removed — is dominated
// by that citation structure (only list punctuation and short per-source
// annotations remain). The dominance check is what separates a bare sources
// list (recover the front-loaded answer, keep the links) from a real prose
// answer that merely opens with "Source:" and references a URL mid-sentence
// (leave it as the answer). Unlike a back-reference closer the links are worth
// keeping, so finalOutput appends them to the recovered answer.
//
// A citations terminal whose sources are bare domains (no scheme, no markdown
// link) is intentionally out of scope — there is no reliable link signal, so it
// is left as-is rather than risk misclassifying prose.
func isCitationsOnly(s string) bool {
t := strings.TrimSpace(s)
if !citationLabelRe.MatchString(t) {
return false
}
body := citationLabelRe.ReplaceAllString(t, "")
if !linkRe.MatchString(body) {
return false
}
residue := strings.Trim(linkRe.ReplaceAllString(body, ""), citationResidueCutset)
return len(residue) <= len(t)/citationDominatedDivisor
}
// isSummaryCloser reports whether a terminal turn is a bookkeeping closer: it
// opens with a complete "citations are logged"-style ack sentence (see
// summaryCloserRe) and is short enough that whatever follows the ack can only
// be a compression of an earlier, fuller answer. Whether that fuller answer
// actually exists is modeSummary's job — the dwarf ratio in
// isSubstantiveAnswer keeps a matching closer in place when nothing earlier
// clearly outweighs it.
func isSummaryCloser(s string) bool {
t := strings.TrimSpace(s)
if t == "" || len(t) > summaryCloserMaxChars {
return false
}
return summaryCloserRe.MatchString(t)
}
// lastSubstantiveAssistantText scans msgs newest→oldest (skipping the terminal
// turn and empty tool-only turns) for the most recent assistant turn whose text
// reads like a real answer. Returns ("", false) when nothing qualifies.
func lastSubstantiveAssistantText(msgs []llm.Message, terminal string) (string, bool) {
// reads like a real answer. mode selects the recovery bar (see
// isSubstantiveAnswer). Returns ("", false) when nothing qualifies.
func lastSubstantiveAssistantText(msgs []llm.Message, terminal string, mode recoveryMode) (string, bool) {
tt := strings.TrimSpace(terminal)
for i := len(msgs) - 1; i >= 0; i-- {
m := msgs[i]
if mode == modeSummary && m.Role == llm.RoleUser {
// A summary closer compresses THIS turn's front-loaded answer, so
// the scan must not cross into an earlier question: once the dwarf
// ratio has rejected the current turn's text, walking further back
// would resurrect a stale answer to a DIFFERENT question — strictly
// worse than keeping the closer. (A mid-run steer message is also a
// user-role boundary; recovery then fails closed, which is fine.)
// The other modes keep their historical unbounded scan.
break
}
if m.Role != llm.RoleAssistant {
continue
}
@@ -79,7 +278,7 @@ func lastSubstantiveAssistantText(msgs []llm.Message, terminal string) (string,
if txt == "" || txt == tt {
continue // the terminal turn itself, or an empty tool-only turn
}
if isSubstantiveAnswer(txt, tt) {
if isSubstantiveAnswer(txt, tt, mode) {
return txt, true
}
}
@@ -88,11 +287,30 @@ func lastSubstantiveAssistantText(msgs []llm.Message, terminal string) (string,
// isSubstantiveAnswer reports whether txt (a prior assistant turn) reads like a
// real answer rather than a preamble, relative to the terminal text.
func isSubstantiveAnswer(txt, terminal string) bool {
//
// modeSummary demands the dwarf ratio FIRST, at every length: a summary closer
// carries a real (compressed) answer, so replacing it is only justified when
// the prior turn is clearly the fuller original it was compressed from.
//
// A sufficiently long turn (>= recoverMinChars) is otherwise accepted
// unconditionally: a multi-hundred-char turn is an answer even when it opens
// conversationally ("Sure, here's…", "Let me explain: …"), so the preamble
// filter is NOT applied to it — applying it there would drop a legitimate long
// front-loaded answer. Only in the borderline band does a turn have to clear a
// floor, not read like a short planning preamble ("Let me look that up…"),
// and — for modeBackRef only — also clearly dwarf the terminal (a citations
// addendum is not a rival answer, so its length is irrelevant; a summary
// closer already proved the ratio above).
func isSubstantiveAnswer(txt, terminal string, mode recoveryMode) bool {
dwarfs := len(txt) >= recoverRatio*len(terminal)
if mode == modeSummary && !dwarfs {
return false
}
if len(txt) >= recoverMinChars {
return true
}
return len(txt) >= recoverFloorChars &&
len(txt) >= recoverRatio*len(terminal) &&
!preambleRe.MatchString(txt)
if len(txt) < recoverFloorChars || preambleRe.MatchString(txt) {
return false
}
return mode != modeBackRef || dwarfs
}
+358 -1
View File
@@ -36,6 +36,82 @@ func TestIsWeakFinal(t *testing.T) {
}
}
func TestIsCitationsOnly(t *testing.T) {
cases := []struct {
name string
in string
want bool
}{
{"sources-md-links", "Sources: [pcprice.watch](https://pcprice.watch/x), [ebay](https://ebay.com/1).", true},
{"lowercase-bare-url", "sources: see https://example.com/a", true},
{"bold-label-colon-inside", "**Sources:** [a](https://a), [b](https://b)", true},
{"bold-label-colon-outside", "**Sources**: [a](https://a), [b](https://b)", true}, // colon after the closing **
{"references-dash", "References — [a](https://a)", true},
{"citations-label", "Citations: https://x/y", true},
{"leading-list-marker", "- Sources: [a](https://a)", true},
{"atx-heading", "## Sources: [a](https://a), [b](https://b)", true}, // ATX heading marker + its trailing space
{"further-reading", "Further reading: https://example.com/deep-dive", true},
{"annotated-multi-source", "Sources: [pcprice.watch](https://a) (tracker), [eBay](https://b) (sold), [bestvaluegpu](https://c) (retail), [resaleprices](https://d) (asking).", true}, // the reported issue-1418 shape
{"backref-plus-links-is-citations", "References: as noted above, [pcprice.watch](https://pcprice.watch/x).", true}, // a back-ref phrase inside a real sources list is still citations
{"empty", "", false},
{"label-but-no-link", "Source: internal analysis, no URL here", false},
{"prose-then-sources", "It sells for ~$2,700. Sources: [a](https://a)", false}, // answer first → not a pure addendum
{"source-led-prose-answer", "Source: According to https://cdc.gov the flu vaccine is 40-60% effective, and the CDC recommends annual vaccination for everyone over six months old.", false}, // a prose answer that merely opens with a "Source:" label
{"mentions-sources-midsentence", "The sources of the leak were never confirmed.", false},
{"link-without-label", "Here is the link you asked for: [a](https://a)", false},
{"bare-domains-out-of-scope", "Sources: pcprice.watch (used ~$200), ebay.com (sold listings)", false}, // bare domains: no scheme or markdown link to key on
{"crisp-number", "42", false},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
if got := isCitationsOnly(c.in); got != c.want {
t.Errorf("isCitationsOnly(%q) = %v, want %v", c.in, got, c.want)
}
})
}
}
// b3cb9ee9Closer is the verbatim terminal turn from mort run b3cb9ee9: a
// 2,089-char answer was front-loaded into the cite-call turn and this 153-byte
// compression (151 runes — the em dash is 3 bytes, and byte length is what the
// thresholds compare) was all that got delivered.
const b3cb9ee9Closer = "Citations are logged. Short version: the bulk of that ~$64M was AIPAC and dark-money super PACs, not the party committees — and it still wasn't enough."
func TestIsSummaryCloser(t *testing.T) {
cases := []struct {
name string
in string
want bool
}{
{"b3cb9ee9-verbatim", b3cb9ee9Closer, true},
{"ack-only", "Citations are logged.", true},
{"ack-no-copula", "Citations logged.", true},
{"claims-cited", "All claims cited.", true},
{"verb-first", "Logged the citations.", true},
{"done-prefix", "Done — citations logged.", true},
{"ack-then-tldr", "Sources have been recorded! TL;DR: the GPU was the bottleneck.", true},
{"references-noted", "References noted. In short: yes, it ships Tuesday.", true},
{"plus-list-marker", "+ Citations are logged.", true},
{"logged-all-the", "Logged all the citations.", true},
{"empty", "", false},
{"ack-continues-midsentence", "The citations are recorded in the court transcript, which shows the filing dates.", false},
{"ack-verb-then-clause", "Citations are logged in Zotero whenever you click the save button.", false},
{"compression-without-ack", "Short version: yes.", false}, // deliberately out of scope
{"mentions-citations-midsentence", "The paper's citations are what got it retracted.", false},
{"crisp-number", "42", false},
{"over-cap", "Citations are logged. " + strings.Repeat("The long version has many more details worth keeping. ", 6), false}, // >300: too substantial to replace
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
if got := isSummaryCloser(c.in); got != c.want {
t.Errorf("isSummaryCloser(%q) = %v, want %v", c.in, got, c.want)
}
})
}
}
func asst(text string, tools ...llm.ToolCall) llm.Message {
m := llm.Message{Role: llm.RoleAssistant}
if text != "" {
@@ -47,7 +123,30 @@ func asst(text string, tools ...llm.ToolCall) llm.Message {
func TestFinalOutput(t *testing.T) {
cite := []llm.ToolCall{{ID: "c1", Name: "cite", Arguments: json.RawMessage(`{}`)}}
longAnswer := strings.TrimSpace(strings.Repeat("Free group calls are capped at sixty minutes. ", 6)) // >200
longAnswer := strings.TrimSpace(strings.Repeat("Free group calls are capped at sixty minutes. ", 6)) // >200
hugeAnswer := strings.TrimSpace(strings.Repeat("Free group calls are capped at sixty minutes. ", 12)) // >3x the b3cb9ee9 closer
// A sources/citations-only terminal — the glm-5.2 "cite" shape behind mort
// issue #1418: the prose answer was front-loaded into the tool-call turn and
// the terminal turn carried only the citations.
sources := "Sources: [pcprice.watch](https://pcprice.watch/x), [ebay](https://www.ebay.com/itm/1)."
answerWithSources := longAnswer + "\n\n" + sources
// A concise (>80, <200 byte) front-loaded answer + a long citations terminal:
// the ratio arm can't be met against the long terminal, so citations mode
// must fall back to the floor.
conciseAnswer := "It sells for about $2,700 used on eBay, typically $2,400 to $2,900 depending on condition and bundle."
longSources := "Sources: [pcprice.watch](https://pcprice.watch/gpu/rtx5090) (tracker), [ebay](https://www.ebay.com/sch/rtx5090) (sold), [newegg](https://newegg.com/rtx5090) (retail), [pcpartpicker](https://pcpartpicker.com/rtx5090) (history)."
// A substantive answer that merely OPENS with "Source:" (not a bare list).
sourceLedAnswer := "Source: https://nvd.nist.gov/vuln/detail/CVE-2024-1234 — this is the authoritative NVD entry for the vulnerability, rated CVSS 9.8 critical."
// A borderline-band (80200 byte) turn that opens like a planning preamble:
// it clears the floor, but the preamble filter still vetoes it (the filter
// applies only in the borderline band; a >=200-byte turn is accepted as-is).
preambleTurn := "Let me look that up across a few different sites and then compile the full comparison for you here."
// A >=200-byte real answer that merely OPENS with a conversational word
// ("Sure,"). The preamble filter must NOT veto it (gadfly regression guard).
longConversationalAnswer := "Sure, here's the rundown: it currently sells for about $2,700 used on eBay, typically $2,400 to $2,900 depending on condition and bundle, with the sealed Founders Edition commanding the top of that range while used AIB cards go a bit lower."
// Matches BOTH the summary ack and backRefRe, within the 120-byte weak cap,
// and long enough (>~92 bytes) that longAnswer would fail the summary bar.
bothMatchCloser := "Citations are logged. As I mentioned above, the full detail on the money sources is in my earlier message."
tests := []struct {
name string
@@ -110,6 +209,200 @@ func TestFinalOutput(t *testing.T) {
terminal: "(see above)",
want: "(see above)", // preamble excluded; falls back to terminal
},
{
name: "citations-only terminal recovers front-loaded answer and keeps sources",
msgs: []llm.Message{
llm.UserText("q?"),
asst(longAnswer, cite...),
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
asst(sources),
},
terminal: sources,
want: answerWithSources, // answer recovered, citations appended
},
{
name: "citations-only terminal but only a preamble prior: keeps the sources",
msgs: []llm.Message{
llm.UserText("q?"),
asst("Let me gather the sources.", cite...),
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
asst(sources),
},
terminal: sources,
want: sources, // nothing substantive to recover → keep the addendum
},
{
name: "citations already in the recovered answer are not duplicated",
msgs: []llm.Message{
llm.UserText("q?"),
asst(answerWithSources, cite...),
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
asst(sources),
},
terminal: sources,
want: answerWithSources, // recovered turn already carries the sources
},
{
// #1418 persisted for CONCISE answers: a <200-char front-loaded
// answer must still be recovered against a long citations terminal
// (the ratio arm is skipped in citations mode).
name: "concise front-loaded answer recovered against a long citations terminal",
msgs: []llm.Message{
llm.UserText("q?"),
asst(conciseAnswer, cite...),
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
asst(longSources),
},
terminal: longSources,
want: conciseAnswer + "\n\n" + longSources,
},
{
// A substantive answer that merely OPENS with "Source:" and cites a
// URL mid-sentence is NOT a citations addendum — return it verbatim,
// never prepend the prior planning turn.
name: "source-led substantive answer is not hijacked by a prior turn",
msgs: []llm.Message{
llm.UserText("what's the authoritative URL?"),
asst("I'll look up the CVE in the NVD database, cross-reference the vendor advisory, and confirm the canonical URL before I answer.", cite...),
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
asst(sourceLedAnswer),
},
terminal: sourceLedAnswer,
want: sourceLedAnswer,
},
{
// A borderline-length turn that opens like a preamble is vetoed
// during recovery; the older real answer is recovered instead. (A
// >=200-byte turn would be accepted verbatim — see the next case.)
name: "borderline preamble is skipped; older real answer recovered",
msgs: []llm.Message{
llm.UserText("q?"),
asst(conciseAnswer, cite...),
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
asst(preambleTurn, cite...),
llm.ToolResultsMessage(llm.ToolResult{ID: "c2", Name: "cite", Content: "ok"}),
asst(sources),
},
terminal: sources,
want: conciseAnswer + "\n\n" + sources,
},
{
// Guards the gadfly regression: a LONG (>=200-byte) front-loaded
// answer that merely opens with a conversational word ("Sure, …")
// must still be recovered — the preamble filter must not veto it.
name: "long answer opening with a conversational word is still recovered",
msgs: []llm.Message{
llm.UserText("q?"),
asst(longConversationalAnswer, cite...),
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
asst(sources),
},
terminal: sources,
want: longConversationalAnswer + "\n\n" + sources,
},
{
// The b3cb9ee9 shape: full answer front-loaded into the cite turn,
// then a summary closer. The closer is discarded — its content is a
// strict compression of the recovered answer.
name: "summary closer discarded when the front-loaded answer dwarfs it",
msgs: []llm.Message{
llm.UserText("where did the $64M come from?"),
asst(hugeAnswer, cite...),
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
asst(b3cb9ee9Closer),
},
terminal: b3cb9ee9Closer,
want: hugeAnswer,
},
{
// The dwarf ratio is mandatory for a summary closer at EVERY length:
// a prior turn that is longer but not clearly the fuller original
// (here ~275 chars vs a 151-char closer, under the 3x bar) must not
// displace a closer that carries real answer content.
name: "summary closer kept when the prior turn does not dwarf it",
msgs: []llm.Message{
llm.UserText("q?"),
asst(longAnswer, cite...),
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
asst(b3cb9ee9Closer),
},
terminal: b3cb9ee9Closer,
want: b3cb9ee9Closer,
},
{
// An ack-only closer ("Citations are logged.") is tiny, so even a
// modest front-loaded answer clears the ratio and replaces it.
name: "ack-only summary closer recovered over a modest answer",
msgs: []llm.Message{
llm.UserText("q?"),
asst(longAnswer, cite...),
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
asst("Citations are logged."),
},
terminal: "Citations are logged.",
want: longAnswer,
},
{
name: "summary closer with only a preamble prior keeps the closer",
msgs: []llm.Message{
llm.UserText("q?"),
asst("Let me gather the numbers.", cite...),
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
asst(b3cb9ee9Closer),
},
terminal: b3cb9ee9Closer,
want: b3cb9ee9Closer,
},
{
// The modeSummary scan must stop at the most recent user message.
// Here the current turn's answer sits in the 1x-3x band (rejected
// by the ratio) while a dwarfing answer to a DIFFERENT question
// sits in history — resurrecting it would be strictly worse than
// keeping the closer.
name: "summary closer never resurrects a stale answer across the user boundary",
msgs: []llm.Message{
llm.UserText("earlier, unrelated question?"),
asst(hugeAnswer),
llm.UserText("q?"),
asst(conciseAnswer, cite...),
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
asst(b3cb9ee9Closer),
},
terminal: b3cb9ee9Closer,
want: b3cb9ee9Closer,
},
{
// The boundary must not break the legitimate multi-turn case: the
// dwarfing front-loaded answer in THIS turn's window is recovered
// even with history behind it.
name: "summary closer recovery still works with history present",
msgs: []llm.Message{
llm.UserText("earlier, unrelated question?"),
asst(longAnswer),
llm.UserText("q?"),
asst(hugeAnswer, cite...),
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
asst(b3cb9ee9Closer),
},
terminal: b3cb9ee9Closer,
want: hugeAnswer,
},
{
// A closer matching BOTH the ack shape and a back-reference
// carries no answer content, so the back-ref test must win and the
// ordinary recovery bar apply — under the summary bar this
// ~106-byte terminal would demand a ~318-byte prior and wrongly
// keep the closer over longAnswer.
name: "back-reference wins over the summary ack when both match",
msgs: []llm.Message{
llm.UserText("q?"),
asst(longAnswer, cite...),
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
asst(bothMatchCloser),
},
terminal: bothMatchCloser,
want: longAnswer,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
@@ -179,3 +472,67 @@ func TestRun_HealthyTerminalUnchanged(t *testing.T) {
t.Errorf("Output = %q, want terminal answer unchanged", res.Output)
}
}
// TestRun_RecoversFrontLoadedAnswerOverSummaryCloser reproduces mort run
// b3cb9ee9 end-to-end: the model front-loads its full answer into the
// cite-call turn, the cite results come back, and the terminal turn is only a
// bookkeeping ack plus a one-line compression. The delivered output must be
// the front-loaded answer, with no extra model call.
func TestRun_RecoversFrontLoadedAnswerOverSummaryCloser(t *testing.T) {
hugeAnswer := strings.TrimSpace(strings.Repeat("Free group calls are capped at sixty minutes. ", 12))
fp := fake.New("fp")
fp.Enqueue("test-model",
fake.ReplyWith(llm.Response{
Parts: []llm.Part{llm.Text(hugeAnswer)},
ToolCalls: []llm.ToolCall{{ID: "c1", Name: "cite", Arguments: json.RawMessage(`{}`)}},
FinishReason: llm.FinishToolCalls,
Usage: llm.Usage{InputTokens: 10, OutputTokens: 5},
}),
fake.Reply(b3cb9ee9Closer),
)
a := New(newModel(t, fp), "sys", WithToolbox(citeToolbox(t)))
res, err := a.Run(context.Background(), "where did the $64M come from?")
if err != nil {
t.Fatalf("Run: %v", err)
}
if res.Output != hugeAnswer {
t.Errorf("Output = %q, want recovered front-loaded answer", res.Output)
}
if n := len(fp.Calls()); n != 2 {
t.Errorf("model calls = %d, want 2 (no extra nudge turn)", n)
}
}
// TestRun_RecoversFrontLoadedAnswerWithCitations reproduces mort issue #1418
// end-to-end: the model front-loads the prose answer into the tool-call turn
// and closes with a sources-only terminal turn. The delivered output must be
// the recovered answer with the citations appended (not the bare sources list),
// with no extra model call.
func TestRun_RecoversFrontLoadedAnswerWithCitations(t *testing.T) {
longAnswer := strings.TrimSpace(strings.Repeat("Free group calls are capped at sixty minutes. ", 6))
sources := "Sources: [docs](https://example.com/docs), [pricing](https://example.com/pricing)."
fp := fake.New("fp")
fp.Enqueue("test-model",
fake.ReplyWith(llm.Response{
Parts: []llm.Part{llm.Text(longAnswer)},
ToolCalls: []llm.ToolCall{{ID: "c1", Name: "cite", Arguments: json.RawMessage(`{}`)}},
FinishReason: llm.FinishToolCalls,
Usage: llm.Usage{InputTokens: 10, OutputTokens: 5},
}),
fake.Reply(sources),
)
a := New(newModel(t, fp), "sys", WithToolbox(citeToolbox(t)))
res, err := a.Run(context.Background(), "is there a meet time limit?")
if err != nil {
t.Fatalf("Run: %v", err)
}
want := longAnswer + "\n\n" + sources
if res.Output != want {
t.Errorf("Output = %q, want recovered answer + citations %q", res.Output, want)
}
if n := len(fp.Calls()); n != 2 {
t.Errorf("model calls = %d, want 2 (no extra nudge turn)", n)
}
}
+48
View File
@@ -173,3 +173,51 @@ func TestSameCallRepeatGuard(t *testing.T) {
t.Errorf("varied calls must not trip the guard: %v", err)
}
}
// TestSameCallRepeatGuardProgressAware: identical (name+args) calls whose
// RESULT keeps changing — canonically polling a long-running background job
// whose progress advances — do not trip the repeat guard even well past the
// limit; but identical calls returning an unchanged result still trip it.
func TestSameCallRepeatGuardProgressAware(t *testing.T) {
// A poll tool that advances every call, so identical args yield a
// different result each time.
calls := 0
polling := llm.NewToolbox("jobs", llm.Tool{
Name: "poll",
Handler: func(context.Context, json.RawMessage) (any, error) {
calls++
return map[string]any{"status": "running", "elapsed": calls}, nil
},
})
n := 0
fp := fake.New("fp", fake.WithDefault(func(string, llm.Request) fake.Step {
n++
if n > 6 { // six identical polls, well past the limit of 3
return fake.Reply("done")
}
return toolCallReply("c", "poll", `{"job":"x"}`)
}))
a := New(newModel(t, fp), "", WithToolbox(polling), WithToolErrorLimits(0, 3), WithMaxSteps(20))
res, err := a.Run(context.Background(), "go")
if err != nil {
t.Fatalf("advancing-result polls must not trip the guard: %v", err)
}
if res.Output != "done" {
t.Errorf("output = %q, want run to complete after polling", res.Output)
}
// A call that returns an UNCHANGED result each time still trips the guard.
frozen := llm.NewToolbox("jobs", llm.Tool{
Name: "poll",
Handler: func(context.Context, json.RawMessage) (any, error) {
return map[string]any{"status": "running"}, nil // never advances
},
})
fp2 := fake.New("fp", fake.WithDefault(func(string, llm.Request) fake.Step {
return toolCallReply("c", "poll", `{"job":"x"}`)
}))
a2 := New(newModel(t, fp2), "", WithToolbox(frozen), WithToolErrorLimits(0, 3), WithMaxSteps(20))
if _, err := a2.Run(context.Background(), "go"); !errors.Is(err, ErrToolLoop) {
t.Fatalf("frozen identical result must still trip the guard: %v", err)
}
}
+214
View File
@@ -0,0 +1,214 @@
// Package audio is majordomo's canonical speech surface: text-to-speech
// (SpeechModel) and audio transcription (TranscriptionModel). Like imagegen,
// it is a deliberately separate contract from the llm package — synthesis and
// transcription share none of the chat message/tool/stream machinery, so they
// get their own small Provider/Model interfaces rather than overloading
// llm.Model (ADR-0017).
//
// Zero values mean "backend default" throughout, mirroring imagegen: an empty
// Voice uses the model's default voice, an empty Format the backend's default
// container, a zero Speed the natural rate.
//
// The first implementation is provider/llamaswap, which targets the OpenAI
// /v1/audio/speech and /v1/audio/transcriptions endpoints routed to
// kokoro/whisper.cpp-style upstreams.
package audio
import "context"
// SpeechRequest is a text-to-speech request.
type SpeechRequest struct {
// Input is the text to speak.
Input string
// Voice selects the voice; "" = the model's default voice.
Voice string
// Format is the audio container ("mp3", "wav", "opus", ...);
// "" = backend default.
Format string
// Speed is the playback-rate multiplier; 0 = backend default (1.0).
Speed float64
// ReferenceAudio is a short voice sample for zero-shot voice cloning
// (chatterbox style): when set, the model speaks Input in the sampled
// voice instead of a named Voice. nil = normal synthesis. Backends
// without cloning support must reject a reference-carrying request
// rather than silently ignoring it.
ReferenceAudio []byte
// ReferenceMIME is the reference audio's MIME type (e.g. "audio/wav");
// "" = let the backend sniff it.
ReferenceMIME string
}
// SpeechResult is the canonical synthesis result: raw audio bytes plus the
// MIME type reported (or implied) by the backend.
type SpeechResult struct {
// Audio is the encoded audio.
Audio []byte
// MIME is the audio MIME type, e.g. "audio/mpeg".
MIME string
// Raw is the provider-native response object, an escape hatch for
// provider-specific fields. May be nil; never required for normal use.
Raw any
}
// SpeechOption mutates a SpeechRequest before it is sent. Options passed to
// Speak are applied to a copy, so a request value can be reused.
type SpeechOption func(*SpeechRequest)
// WithVoice selects the voice.
func WithVoice(v string) SpeechOption { return func(r *SpeechRequest) { r.Voice = v } }
// WithFormat sets the audio container format (e.g. "mp3", "wav").
func WithFormat(f string) SpeechOption { return func(r *SpeechRequest) { r.Format = f } }
// WithSpeed sets the playback-rate multiplier.
func WithSpeed(s float64) SpeechOption { return func(r *SpeechRequest) { r.Speed = s } }
// WithReferenceAudio provides a voice sample for zero-shot voice cloning.
func WithReferenceAudio(data []byte, mime string) SpeechOption {
return func(r *SpeechRequest) { r.ReferenceAudio, r.ReferenceMIME = data, mime }
}
// Apply returns a copy of the request with all options applied. Providers
// call this once at the top of Speak.
func (r SpeechRequest) Apply(opts ...SpeechOption) SpeechRequest {
for _, opt := range opts {
opt(&r)
}
return r
}
// SpeechModel synthesizes speech from text.
type SpeechModel interface {
// Speak renders the request's input text as audio.
Speak(ctx context.Context, req SpeechRequest, opts ...SpeechOption) (*SpeechResult, error)
}
// SpeechModelOption configures a SpeechModel at construction time. Reserved
// for future per-model settings; present so the interface is
// forward-compatible (mirrors imagegen.ModelOption).
type SpeechModelOption func(*SpeechModelConfig)
// SpeechModelConfig carries per-model construction settings.
type SpeechModelConfig struct{}
// ApplySpeechModelOptions folds options into a config.
func ApplySpeechModelOptions(opts []SpeechModelOption) SpeechModelConfig {
var cfg SpeechModelConfig
for _, opt := range opts {
opt(&cfg)
}
return cfg
}
// SpeechProvider mints speech models bound to one backend.
type SpeechProvider interface {
// Name is the registry identifier for the provider.
Name() string
// SpeechModel returns a SpeechModel bound to the given id (passed through
// to the backend verbatim; no catalog validation).
SpeechModel(id string, opts ...SpeechModelOption) (SpeechModel, error)
}
// TranscriptionRequest is a speech-to-text request. Audio is carried as bytes
// (never a URL), mirroring llm.ImagePart's bytes-only contract.
type TranscriptionRequest struct {
// Audio is the encoded audio to transcribe.
Audio []byte
// MIME is the audio MIME type (e.g. "audio/mpeg"); "" = let the backend
// sniff it.
MIME string
// Filename is the multipart filename hint some backends key their format
// detection on; "" derives one from MIME ("audio.mp3") or falls back to
// "audio".
Filename string
// Language is a BCP-47/ISO-639 hint (e.g. "en"); "" = auto-detect.
Language string
// Prompt is optional context or vocabulary to bias decoding; "" = none.
Prompt string
// Translate requests an English translation of the speech instead of a
// same-language transcript (whisper's translate task). When set and no
// Language is given, providers must force source-language auto-detection
// — a backend whose default language is "en" would otherwise skip
// translation entirely.
Translate bool
}
// TranscriptionResult is the canonical transcription result.
type TranscriptionResult struct {
// Text is the transcript.
Text string
// Raw is the provider-native response object. May be nil.
Raw any
}
// TranscriptionOption mutates a TranscriptionRequest before it is sent.
type TranscriptionOption func(*TranscriptionRequest)
// WithLanguage sets the language hint (e.g. "en").
func WithLanguage(l string) TranscriptionOption {
return func(r *TranscriptionRequest) { r.Language = l }
}
// WithPrompt sets the decoding context/vocabulary hint.
func WithPrompt(p string) TranscriptionOption {
return func(r *TranscriptionRequest) { r.Prompt = p }
}
// WithTranslate requests an English translation instead of a transcript.
func WithTranslate() TranscriptionOption {
return func(r *TranscriptionRequest) { r.Translate = true }
}
// Apply returns a copy of the request with all options applied.
func (r TranscriptionRequest) Apply(opts ...TranscriptionOption) TranscriptionRequest {
for _, opt := range opts {
opt(&r)
}
return r
}
// TranscriptionModel transcribes audio to text.
type TranscriptionModel interface {
// Transcribe converts the request's audio into text.
Transcribe(ctx context.Context, req TranscriptionRequest, opts ...TranscriptionOption) (*TranscriptionResult, error)
}
// TranscriptionModelOption configures a TranscriptionModel at construction
// time. Reserved for future per-model settings.
type TranscriptionModelOption func(*TranscriptionModelConfig)
// TranscriptionModelConfig carries per-model construction settings.
type TranscriptionModelConfig struct{}
// ApplyTranscriptionModelOptions folds options into a config.
func ApplyTranscriptionModelOptions(opts []TranscriptionModelOption) TranscriptionModelConfig {
var cfg TranscriptionModelConfig
for _, opt := range opts {
opt(&cfg)
}
return cfg
}
// TranscriptionProvider mints transcription models bound to one backend.
type TranscriptionProvider interface {
// Name is the registry identifier for the provider.
Name() string
// TranscriptionModel returns a TranscriptionModel bound to the given id
// (passed through to the backend verbatim; no catalog validation).
TranscriptionModel(id string, opts ...TranscriptionModelOption) (TranscriptionModel, error)
}
+38
View File
@@ -0,0 +1,38 @@
package audio
import "testing"
func TestSpeechRequestApply(t *testing.T) {
base := SpeechRequest{Input: "hello"}
got := base.Apply(WithVoice("af_heart"), WithFormat("wav"), WithSpeed(1.5))
if got.Input != "hello" {
t.Errorf("Input = %q, want %q", got.Input, "hello")
}
if got.Voice != "af_heart" || got.Format != "wav" || got.Speed != 1.5 {
t.Errorf("got = %+v", got)
}
// Apply must not mutate the receiver (options apply to a copy).
if base.Voice != "" || base.Format != "" || base.Speed != 0 {
t.Errorf("base mutated: %+v", base)
}
}
func TestTranscriptionRequestApply(t *testing.T) {
base := TranscriptionRequest{Audio: []byte{1, 2, 3}}
got := base.Apply(WithLanguage("en"), WithPrompt("names"))
if got.Language != "en" || got.Prompt != "names" {
t.Errorf("got = %+v", got)
}
if base.Language != "" || base.Prompt != "" {
t.Errorf("base mutated: %+v", base)
}
}
func TestApplyModelOptions(t *testing.T) {
// No options yet; just verify they return usable zero configs.
_ = ApplySpeechModelOptions(nil)
_ = ApplyTranscriptionModelOptions(nil)
}
+113
View File
@@ -0,0 +1,113 @@
package audio
import "context"
// DiarizationRequest is a speaker-labelled transcription request ("who said
// what"). Audio is carried as bytes (never a URL), mirroring
// TranscriptionRequest. Zero values mean "backend default" (ADR-0020).
type DiarizationRequest struct {
// Audio is the encoded audio (or video container) to transcribe.
Audio []byte
// MIME is the audio MIME type (e.g. "audio/mpeg"); "" = let the backend
// sniff it.
MIME string
// Filename is the multipart filename hint some backends key their format
// detection on; "" derives one from MIME or falls back to "audio".
Filename string
// Language is a BCP-47/ISO-639 hint (e.g. "en"); "" = auto-detect.
Language string
// MinSpeakers / MaxSpeakers bound the speaker count when the caller knows
// it; 0 = let the backend estimate.
MinSpeakers int
MaxSpeakers int
}
// DiarizationSegment is one speaker turn.
type DiarizationSegment struct {
// Start and End are offsets into the audio, in seconds.
Start float64
End float64
// Speaker is the backend's per-file speaker label (e.g. "SPEAKER_00").
// Labels are relative to this one file — they are NOT stable identities
// across files.
Speaker string
// Text is the transcript of this turn.
Text string
}
// DiarizationResult is the canonical diarization result.
type DiarizationResult struct {
// Text is the full transcript, unlabelled.
Text string
// Language is the detected (or requested) language code; may be "".
Language string
// Segments are the speaker turns in time order.
Segments []DiarizationSegment
// Raw is the provider-native response object. May be nil.
Raw any
}
// DiarizationOption mutates a DiarizationRequest before it is sent.
type DiarizationOption func(*DiarizationRequest)
// WithDiarizationLanguage sets the language hint (e.g. "en").
func WithDiarizationLanguage(l string) DiarizationOption {
return func(r *DiarizationRequest) { r.Language = l }
}
// WithSpeakerBounds bounds the expected speaker count (0 leaves a bound
// unset).
func WithSpeakerBounds(minSpeakers, maxSpeakers int) DiarizationOption {
return func(r *DiarizationRequest) {
r.MinSpeakers, r.MaxSpeakers = minSpeakers, maxSpeakers
}
}
// Apply returns a copy of the request with all options applied.
func (r DiarizationRequest) Apply(opts ...DiarizationOption) DiarizationRequest {
for _, opt := range opts {
opt(&r)
}
return r
}
// DiarizationModel transcribes audio with speaker labels.
type DiarizationModel interface {
// Diarize converts the request's audio into speaker-labelled segments.
Diarize(ctx context.Context, req DiarizationRequest, opts ...DiarizationOption) (*DiarizationResult, error)
}
// DiarizationModelOption configures a DiarizationModel at construction time.
// Reserved for future per-model settings.
type DiarizationModelOption func(*DiarizationModelConfig)
// DiarizationModelConfig carries per-model construction settings.
type DiarizationModelConfig struct{}
// ApplyDiarizationModelOptions folds options into a config.
func ApplyDiarizationModelOptions(opts []DiarizationModelOption) DiarizationModelConfig {
var cfg DiarizationModelConfig
for _, opt := range opts {
opt(&cfg)
}
return cfg
}
// DiarizationProvider mints diarization models bound to one backend.
type DiarizationProvider interface {
// Name is the registry identifier for the provider.
Name() string
// DiarizationModel returns a DiarizationModel bound to the given id
// (passed through to the backend verbatim; no catalog validation).
DiarizationModel(id string, opts ...DiarizationModelOption) (DiarizationModel, error)
}
+66
View File
@@ -0,0 +1,66 @@
package audio
import "context"
// EnhancementRequest asks a speech-enhancement backend (DeepFilterNet style)
// to denoise a recording. Audio is carried as bytes (never a URL), mirroring
// TranscriptionRequest (ADR-0024).
type EnhancementRequest struct {
// Audio is the encoded audio to enhance.
Audio []byte
// MIME is the audio MIME type (e.g. "audio/mpeg"); "" = let the backend
// sniff it.
MIME string
// Filename is the multipart filename hint some backends key their format
// detection on; "" derives one from MIME or falls back to "audio".
Filename string
}
// EnhancementOption mutates an EnhancementRequest before it is sent.
// Reserved for future request settings (the reference backend takes no
// parameters).
type EnhancementOption func(*EnhancementRequest)
// Apply returns a copy of the request with all options applied.
func (r EnhancementRequest) Apply(opts ...EnhancementOption) EnhancementRequest {
for _, opt := range opts {
opt(&r)
}
return r
}
// SpeechEnhancer denoises speech recordings. The result reuses SpeechResult
// (audio bytes + MIME + Raw) — enhancement is audio-in/audio-out exactly
// like synthesis.
type SpeechEnhancer interface {
// Enhance returns the denoised audio.
Enhance(ctx context.Context, req EnhancementRequest, opts ...EnhancementOption) (*SpeechResult, error)
}
// SpeechEnhancerModelOption configures a SpeechEnhancer at construction time.
// Reserved for future per-model settings.
type SpeechEnhancerModelOption func(*SpeechEnhancerModelConfig)
// SpeechEnhancerModelConfig carries per-model construction settings.
type SpeechEnhancerModelConfig struct{}
// ApplySpeechEnhancerModelOptions folds options into a config.
func ApplySpeechEnhancerModelOptions(opts []SpeechEnhancerModelOption) SpeechEnhancerModelConfig {
var cfg SpeechEnhancerModelConfig
for _, opt := range opts {
opt(&cfg)
}
return cfg
}
// SpeechEnhancementProvider mints SpeechEnhancers bound to one backend.
type SpeechEnhancementProvider interface {
// Name is the registry identifier for the provider.
Name() string
// SpeechEnhancerModel returns a SpeechEnhancer bound to the given id
// (passed through to the backend verbatim; no catalog validation).
SpeechEnhancerModel(id string, opts ...SpeechEnhancerModelOption) (SpeechEnhancer, error)
}
+113
View File
@@ -0,0 +1,113 @@
package audio
import "context"
// StemSeparationRequest asks a source-separation backend (Demucs style) to
// split a mix into stems. Audio is carried as bytes (never a URL), mirroring
// TranscriptionRequest. Zero values mean "backend default" (ADR-0024).
type StemSeparationRequest struct {
// Audio is the encoded mix to separate.
Audio []byte
// MIME is the audio MIME type (e.g. "audio/mpeg"); "" = let the backend
// sniff it.
MIME string
// Filename is the multipart filename hint some backends key their format
// detection on; "" derives one from MIME or falls back to "audio".
Filename string
// Mode selects the split: "two" (vocals + accompaniment) or "four"
// (vocals/drums/bass/other); "" = backend default (four).
Mode string
// Model selects the separator's internal network where the backend
// offers several (Demucs: "htdemucs", "htdemucs_ft"); "" = backend
// default. This is NOT the provider model id — that is fixed when the
// StemSeparator is minted (mirrors BackgroundRemovalRequest.Net).
Model string
// Format is the per-stem audio container ("mp3" or "wav");
// "" = backend default.
Format string
}
// Stem is one separated source.
type Stem struct {
// Name is the stem's name ("vocals", "drums", "bass", "other",
// "no_vocals", ...), taken from the backend's own labelling.
Name string
// Audio is the encoded stem.
Audio []byte
// MIME is the stem's audio MIME type, e.g. "audio/mpeg".
MIME string
}
// StemSeparationResult is the canonical separation result.
type StemSeparationResult struct {
// Stems are the separated sources, in the order the backend returned
// them.
Stems []Stem
}
// StemSeparationOption mutates a StemSeparationRequest before it is sent.
type StemSeparationOption func(*StemSeparationRequest)
// WithStemMode selects the split ("two" or "four").
func WithStemMode(m string) StemSeparationOption {
return func(r *StemSeparationRequest) { r.Mode = m }
}
// WithStemModel selects the separator's internal network.
func WithStemModel(m string) StemSeparationOption {
return func(r *StemSeparationRequest) { r.Model = m }
}
// WithStemFormat sets the per-stem audio container ("mp3", "wav").
func WithStemFormat(f string) StemSeparationOption {
return func(r *StemSeparationRequest) { r.Format = f }
}
// Apply returns a copy of the request with all options applied.
func (r StemSeparationRequest) Apply(opts ...StemSeparationOption) StemSeparationRequest {
for _, opt := range opts {
opt(&r)
}
return r
}
// StemSeparator splits a mix into stems.
type StemSeparator interface {
// SeparateStems returns the separated sources. Separation is CPU-bound
// and slow (minutes for a full song); bound the call with a context
// deadline.
SeparateStems(ctx context.Context, req StemSeparationRequest, opts ...StemSeparationOption) (*StemSeparationResult, error)
}
// StemSeparatorModelOption configures a StemSeparator at construction time.
// Reserved for future per-model settings.
type StemSeparatorModelOption func(*StemSeparatorModelConfig)
// StemSeparatorModelConfig carries per-model construction settings.
type StemSeparatorModelConfig struct{}
// ApplyStemSeparatorModelOptions folds options into a config.
func ApplyStemSeparatorModelOptions(opts []StemSeparatorModelOption) StemSeparatorModelConfig {
var cfg StemSeparatorModelConfig
for _, opt := range opts {
opt(&cfg)
}
return cfg
}
// StemSeparationProvider mints StemSeparators bound to one backend.
type StemSeparationProvider interface {
// Name is the registry identifier for the provider.
Name() string
// StemSeparatorModel returns a StemSeparator bound to the given id
// (passed through to the backend verbatim; no catalog validation).
StemSeparatorModel(id string, opts ...StemSeparatorModelOption) (StemSeparator, error)
}
+99 -12
View File
@@ -13,15 +13,84 @@ import (
// Built-in provider names.
const (
ProviderOpenAI = "openai"
ProviderOpenAI = "openai"
// ProviderKimi is Moonshot AI's Kimi models over their OpenAI-compatible
// Chat Completions endpoint. Reuses the openai client (like llama-swap);
// keyed by KIMI_API_KEY, default base URL kimiBaseURL.
ProviderKimi = "kimi"
// ProviderQwen is Alibaba's Qwen models over Model Studio's
// OpenAI-compatible Chat Completions endpoint. Reuses the openai client
// (like kimi and llama-swap); keyed by QWEN_API_KEY, default base URL
// qwenBaseURL. ADR-0027 records why the OpenAI surface and not the
// Anthropic-compatible one Model Studio also exposes.
ProviderQwen = "qwen"
ProviderAnthropic = "anthropic"
ProviderGoogle = "google"
ProviderOllama = "ollama"
ProviderOllamaCloud = "ollama-cloud"
ProviderForeman = "foreman"
ProviderLlamaSwap = "llama-swap"
// ProviderLlamaSwapTLS is the DSN scheme for a TLS-fronted llama-swap
// (https base URL). It is a scheme only, not a default built-in provider
// name. Why a separate scheme rather than auto-detecting: a DSN carries no
// reliable signal for http vs https, so the choice is explicit
// (llama-swap = http local-first, llama-swaps = https), mirroring rediss.
ProviderLlamaSwapTLS = "llama-swaps"
)
// kimiBaseURL is Moonshot AI's international OpenAI-compatible endpoint. The
// China endpoint (api.moonshot.cn/v1) is reachable via a kimi:// LLM_* DSN.
const kimiBaseURL = "https://api.moonshot.ai/v1"
// qwenBaseURL is Alibaba Model Studio's international (Singapore) endpoint in
// OpenAI-compatible mode. The China endpoint
// (dashscope.aliyuncs.com/compatible-mode/v1) and any regional host are
// reachable via a qwen:// LLM_* DSN.
const qwenBaseURL = "https://dashscope-intl.aliyuncs.com/compatible-mode/v1"
// openaiCompatScheme builds the DSN factory shared by every built-in that is
// "the openai client pointed somewhere else" (kimi, qwen, ...). The provider
// is named after the LLM_<NAME> var that defined it, takes its credential from
// the DSN token — not the built-in's own env var, which does nothing for a
// DSN-defined provider — and so names that same LLM_<NAME> var in the
// missing-key hint, matching the lazy-resolution key form in providerFor.
//
// wrap is the caller's option-decorator (it injects the registry's HTTP
// client), so a DSN provider is built exactly like the eager built-ins.
func openaiCompatScheme(wrap func(...openai.Option) []openai.Option) SchemeFactory {
return func(name string, dsn DSN) (llm.Provider, error) {
return openai.New(wrap(
openai.WithName(name),
openai.WithBaseURL(dsn.BaseURL()),
openai.WithAPIKey(dsn.Token),
openai.WithAPIKeyName(envKeyForProvider(name)),
)...), nil
}
}
// registerOpenAICompatBuiltin installs BOTH halves of an OpenAI-compat
// built-in: the eager provider under name (credential from keyEnv) and the
// matching name:// DSN scheme. Why both in one call: the two halves are a pair
// — a built-in whose scheme is missing resolves as a spec but not from an
// LLM_* DSN, and the credential rules below have to hold identically in each.
// Adding the next one is a single line rather than six lines to copy.
//
// The two credential rules, holding by construction for every caller:
// - WithAPIKey is passed UNCONDITIONALLY, even when the lookup comes back
// empty. openai.New defaults its key to OPENAI_API_KEY, so anything less
// lets an unset keyEnv silently authenticate as OpenAI.
// - WithAPIKeyName makes the synthetic-401 hint name keyEnv, so a keyless
// call tells the operator the variable that actually fixes it.
func registerOpenAICompatBuiltin(r *Registry, wrap func(...openai.Option) []openai.Option, name, baseURL, keyEnv string) {
r.providers[name] = openai.New(wrap(
openai.WithName(name),
openai.WithBaseURL(baseURL),
openai.WithAPIKey(r.envLookup(keyEnv)),
openai.WithAPIKeyName(keyEnv),
)...)
r.schemes[name] = openaiCompatScheme(wrap)
}
// registerBuiltins installs the built-in providers and env-DSN scheme
// factories into a fresh registry. httpClient, when non-nil, is used by
// every provider and factory the registry itself constructs.
@@ -68,26 +137,44 @@ func registerBuiltins(r *Registry, httpClient *http.Client) {
)...), nil
}
// Third-party endpoints that ARE the openai client at another base URL —
// no new package, mirroring llama-swap's chat path. Each gets the eager
// built-in plus its name:// DSN scheme, and the credential rules hold by
// construction (see registerOpenAICompatBuiltin).
//
// kimi (ADR-0026): Moonshot's international endpoint; China host via
// kimi://[email protected]/v1.
registerOpenAICompatBuiltin(r, openaiOpts, ProviderKimi, kimiBaseURL, "KIMI_API_KEY")
// qwen (ADR-0027): Alibaba Model Studio's international host. Model Studio
// also exposes an Anthropic-compatible endpoint; the ADR records why the
// OpenAI one is the built-in. China / workspace-scoped regional hosts via
// qwen://[email protected]/compatible-mode/v1.
registerOpenAICompatBuiltin(r, openaiOpts, ProviderQwen, qwenBaseURL, "QWEN_API_KEY")
// llama-swap: OpenAI-compatible chat + image generation + management
// endpoints over a model-swapping proxy. Chat reuses the openai client
// (provider/llamaswap delegates); the DSN builds an http:// base URL
// because llama-swap is local-first (TLS-fronted instances can use the
// openai:// scheme for chat). The no-DSN built-in errors on use with a
// clear message, mirroring foreman.
// (provider/llamaswap delegates). Two schemes: "llama-swap" builds an
// http:// base URL (local-first default), "llama-swaps" builds https://
// for a TLS-fronted instance (mirrors redis/rediss). The no-DSN built-in
// errors on use with a clear message, mirroring foreman.
llamaSwapOpts := func(extra ...llamaswap.Option) []llamaswap.Option {
if httpClient != nil {
extra = append(extra, llamaswap.WithHTTPClient(httpClient))
}
return extra
}
r.providers[ProviderLlamaSwap] = llamaswap.New(llamaSwapOpts(llamaswap.WithName(ProviderLlamaSwap))...)
r.schemes[ProviderLlamaSwap] = func(name string, dsn DSN) (llm.Provider, error) {
return llamaswap.New(llamaSwapOpts(
llamaswap.WithName(name),
llamaswap.WithBaseURL("http://"+dsn.Host),
llamaswap.WithToken(dsn.Token),
)...), nil
llamaSwapScheme := func(urlScheme string) SchemeFactory {
return func(name string, dsn DSN) (llm.Provider, error) {
return llamaswap.New(llamaSwapOpts(
llamaswap.WithName(name),
llamaswap.WithBaseURL(urlScheme+"://"+dsn.Host),
llamaswap.WithToken(dsn.Token),
)...), nil
}
}
r.providers[ProviderLlamaSwap] = llamaswap.New(llamaSwapOpts(llamaswap.WithName(ProviderLlamaSwap))...)
r.schemes[ProviderLlamaSwap] = llamaSwapScheme("http")
r.schemes[ProviderLlamaSwapTLS] = llamaSwapScheme("https")
// Anthropic and Anthropic-compatible endpoints.
anthropicOpts := func(extra ...anthropic.Option) []anthropic.Option {
+22
View File
@@ -51,6 +51,28 @@ func TestLlamaSwapScheme(t *testing.T) {
}
}
// TestLlamaSwapsScheme: the "llama-swaps" scheme builds an https base URL for a
// TLS-fronted instance (vs "llama-swap" which is http local-first).
func TestLlamaSwapsScheme(t *testing.T) {
r := newTestRegistry(t)
if err := r.LoadEnv(map[string]string{
"LLM_LST": "llama-swaps://[email protected]",
}); err != nil {
t.Fatalf("LoadEnv: %v", err)
}
p, ok := r.Provider("lst")
if !ok {
t.Fatal("provider \"lst\" not registered")
}
lp, ok := p.(*llamaswap.Provider)
if !ok {
t.Fatalf("provider is %T, want *llamaswap.Provider", p)
}
if want := "https://swap.example.com"; lp.BaseURL() != want {
t.Errorf("baseURL = %q, want %q", lp.BaseURL(), want)
}
}
// TestLlamaSwapBuiltinNoURL: the no-DSN built-in resolves but errors clearly on
// use (mirrors foreman), rather than silently hitting a wrong host.
func TestLlamaSwapBuiltinNoURL(t *testing.T) {
+240
View File
@@ -0,0 +1,240 @@
package majordomo
import (
"context"
"errors"
"io"
"net/http"
"strings"
"testing"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
// Shared fixtures and the shared contract for the built-ins that are "the
// openai client pointed somewhere else" (kimi, qwen, ...). They live here
// rather than in any one provider's test file so a new OpenAI-compat built-in
// has nothing to copy — the same reason registerOpenAICompatBuiltin exists on
// the production side.
// chatCompletionOK is a minimal valid Chat Completions body, so Generate
// returns a non-empty response (an empty one would trigger failover, not a
// clean pass).
const chatCompletionOK = `{"id":"c1","object":"chat.completion","choices":[` +
`{"index":0,"message":{"role":"assistant","content":"ok"},"finish_reason":"stop"}]}`
// captureRT records the last request (and the bytes of its body) and returns a
// canned response without touching the network, so these tests stay hermetic
// while still exercising the real openai client the built-ins reuse: base URL,
// auth header, and the JSON actually put on the wire.
type captureRT struct {
req *http.Request
reqBody []byte
body string
}
func (c *captureRT) RoundTrip(r *http.Request) (*http.Response, error) {
c.req = r
// Drain and close the request body: a RoundTripper owns it, and those
// bytes are what wire-shape assertions read.
c.reqBody = nil
if r.Body != nil {
c.reqBody, _ = io.ReadAll(r.Body)
_ = r.Body.Close()
}
return &http.Response{
StatusCode: http.StatusOK,
Body: io.NopCloser(strings.NewReader(c.body)),
Header: make(http.Header),
Request: r,
}, nil
}
// singleKeyEnv builds a WithEnvLookup function that knows exactly one variable
// and returns "" for everything else. The empty default has teeth: a built-in
// that reached for any other variable name gets nothing, so the request 401s
// and the test fails rather than quietly authenticating off the wrong key.
func singleKeyEnv(key, value string) func(string) string {
return func(k string) string {
if k == key {
return value
}
return ""
}
}
// openAICompatBuiltin describes one built-in for the shared contract below.
// Adding an OpenAI-compat built-in means adding a row here — not copying a
// test file, which is how kimi's and qwen's suites became near-identical.
type openAICompatBuiltin struct {
name string // registry name and spec prefix
keyEnv string // the credential variable this built-in reads
model string // a current model id for that endpoint
wantURL string // chat-completions URL the default endpoint must produce
// The name:// DSN case: an alternate host (regional/China endpoint)
// reached through an LLM_<dsnVar> definition.
dsnVar string
dsnHost string
wantDSNURL string
}
var openAICompatBuiltins = []openAICompatBuiltin{
{
name: ProviderKimi,
keyEnv: "KIMI_API_KEY",
model: "kimi-k2-0711-preview",
wantURL: "https://api.moonshot.ai/v1/chat/completions",
dsnVar: "LLM_KCN",
dsnHost: "api.moonshot.cn/v1",
wantDSNURL: "https://api.moonshot.cn/v1/chat/completions",
},
{
name: ProviderQwen,
keyEnv: "QWEN_API_KEY",
model: "qwen3.8-max",
wantURL: "https://dashscope-intl.aliyuncs.com/compatible-mode/v1/chat/completions",
dsnVar: "LLM_QCN",
dsnHost: "dashscope.aliyuncs.com/compatible-mode/v1",
wantDSNURL: "https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions",
},
}
// TestOpenAICompatBuiltins is the whole contract an OpenAI-compat built-in
// owes, asserted identically for every one of them: it resolves in Parse and
// targets its own endpoint with its own key; a missing key fails closed naming
// the right variable and never reaching the network; its name:// DSN reaches
// any other host on the DSN token; and a keyless DSN names the LLM_<NAME> that
// actually fixes it rather than the built-in's variable, which does nothing
// for a DSN-defined provider.
func TestOpenAICompatBuiltins(t *testing.T) {
for _, tc := range openAICompatBuiltins {
t.Run(tc.name+"/builtin", func(t *testing.T) {
rt := &captureRT{body: chatCompletionOK}
secret := tc.name + "-secret"
r := newTestRegistry(t,
WithEnvLookup(singleKeyEnv(tc.keyEnv, secret)),
WithHTTPClient(&http.Client{Transport: rt}),
)
if p, ok := r.Provider(tc.name); !ok {
t.Fatalf("built-in %q not registered", tc.name)
} else if p.Name() != tc.name {
t.Errorf("name = %q, want %q", p.Name(), tc.name)
}
spec := tc.name + "/" + tc.model
m, err := r.Parse(spec)
if err != nil {
t.Fatalf("Parse(%q): %v", spec, err)
}
if got := targetsOf(t, m); len(got) != 1 || got[0] != spec {
t.Fatalf("targets = %v, want [%q]", got, spec)
}
if _, err := m.Generate(context.Background(), llm.Request{Messages: []llm.Message{llm.UserText("hi")}}); err != nil {
t.Fatalf("Generate: %v", err)
}
if rt.req == nil {
t.Fatal("no request captured")
}
if rt.req.URL.String() != tc.wantURL {
t.Errorf("URL = %q, want %q", rt.req.URL.String(), tc.wantURL)
}
if want := "Bearer " + secret; rt.req.Header.Get("Authorization") != want {
t.Errorf("Authorization = %q, want %q", rt.req.Header.Get("Authorization"), want)
}
})
t.Run(tc.name+"/builtin missing key", func(t *testing.T) {
rt := &captureRT{body: chatCompletionOK}
r := newTestRegistry(t, WithHTTPClient(&http.Client{Transport: rt}))
m, err := r.Parse(tc.name + "/" + tc.model)
if err != nil {
t.Fatalf("Parse: %v", err)
}
_, err = m.Generate(context.Background(), llm.Request{Messages: []llm.Message{llm.UserText("hi")}})
apiErr, ok := errors.AsType[*llm.APIError](err)
if !ok {
t.Fatalf("err = %v (%T), want *llm.APIError", err, err)
}
if apiErr.Status != http.StatusUnauthorized || apiErr.Code != "missing_api_key" {
t.Errorf("Status/Code = %d/%q, want 401/missing_api_key", apiErr.Status, apiErr.Code)
}
if !strings.Contains(apiErr.Message, tc.keyEnv) {
t.Errorf("message = %q, want it to name %s", apiErr.Message, tc.keyEnv)
}
// The load-bearing half: openai.New defaults its key to
// OPENAI_API_KEY, so a built-in that stopped passing WithAPIKey
// unconditionally would authenticate as OpenAI instead of failing.
if strings.Contains(apiErr.Message, "OPENAI_API_KEY") {
t.Errorf("message = %q, must not name OPENAI_API_KEY", apiErr.Message)
}
if rt.req != nil {
t.Error("network was hit despite missing key")
}
})
t.Run(tc.name+"/dsn scheme", func(t *testing.T) {
rt := &captureRT{body: chatCompletionOK}
r := newTestRegistry(t, WithHTTPClient(&http.Client{Transport: rt}))
if err := r.LoadEnv(map[string]string{
tc.dsnVar: tc.name + "://tok@" + tc.dsnHost,
}); err != nil {
t.Fatalf("LoadEnv: %v", err)
}
dsnName := strings.ToLower(strings.TrimPrefix(tc.dsnVar, "LLM_"))
m, err := r.Parse(dsnName + "/" + tc.model)
if err != nil {
t.Fatalf("Parse: %v", err)
}
if _, err := m.Generate(context.Background(), llm.Request{Messages: []llm.Message{llm.UserText("hi")}}); err != nil {
t.Fatalf("Generate: %v", err)
}
if rt.req == nil {
t.Fatal("no request captured")
}
if rt.req.URL.String() != tc.wantDSNURL {
t.Errorf("URL = %q, want %q", rt.req.URL.String(), tc.wantDSNURL)
}
if want := "Bearer tok"; rt.req.Header.Get("Authorization") != want {
t.Errorf("Authorization = %q, want %q", rt.req.Header.Get("Authorization"), want)
}
})
t.Run(tc.name+"/dsn scheme missing token", func(t *testing.T) {
rt := &captureRT{body: chatCompletionOK}
r := newTestRegistry(t, WithHTTPClient(&http.Client{Transport: rt}))
if err := r.LoadEnv(map[string]string{
tc.dsnVar: tc.name + "://" + tc.dsnHost, // no token
}); err != nil {
t.Fatalf("LoadEnv: %v", err)
}
dsnName := strings.ToLower(strings.TrimPrefix(tc.dsnVar, "LLM_"))
m, err := r.Parse(dsnName + "/" + tc.model)
if err != nil {
t.Fatalf("Parse: %v", err)
}
_, err = m.Generate(context.Background(), llm.Request{Messages: []llm.Message{llm.UserText("hi")}})
apiErr, ok := errors.AsType[*llm.APIError](err)
if !ok {
t.Fatalf("err = %v (%T), want *llm.APIError", err, err)
}
// A keyless DSN is fixed by adding a token to that DSN, so the
// hint must name the defining variable — never the built-in's own
// key, which does nothing for a DSN-defined provider.
if !strings.Contains(apiErr.Message, tc.dsnVar) {
t.Errorf("message = %q, want it to name %s", apiErr.Message, tc.dsnVar)
}
if strings.Contains(apiErr.Message, tc.keyEnv) {
t.Errorf("message = %q, must not name %s for a DSN provider", apiErr.Message, tc.keyEnv)
}
if rt.req != nil {
t.Error("network was hit despite missing token")
}
})
}
}
+86
View File
@@ -0,0 +1,86 @@
package majordomo
import (
"context"
"encoding/json"
"net/http"
"testing"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
// The contract qwen shares with every other OpenAI-compat built-in (endpoint,
// credential isolation, its qwen:// DSN) is asserted by the table in
// builtin_openaicompat_test.go. What remains here is qwen-specific: the
// reverse-leak direction, and the wire claim ADR-0027 turns on.
// TestQwenBuiltinKeyDoesNotLeakToOpenAI: QWEN_API_KEY is the qwen built-in's
// credential and nothing else's. Why this direction too: the shared table's
// missing-key case only proves qwen never borrows OPENAI_API_KEY; this proves
// the reverse — a registry that can see QWEN_API_KEY must not hand it to the
// openai built-in, which would send an Alibaba key to api.openai.com.
func TestQwenBuiltinKeyDoesNotLeakToOpenAI(t *testing.T) {
// Set before newTestRegistry: the openai built-in reads OPENAI_API_KEY at
// construction. Giving it a real key is what keeps this test honest — a
// keyless openai target would 401 before any request, and the assertion
// below would pass without a single byte reaching the wire.
t.Setenv("OPENAI_API_KEY", "openai-secret")
rt := &captureRT{body: chatCompletionOK}
r := newTestRegistry(t,
WithEnvLookup(singleKeyEnv("QWEN_API_KEY", "qwen-secret")),
WithHTTPClient(&http.Client{Transport: rt}),
)
m, err := r.Parse("openai/gpt-4o-mini")
if err != nil {
t.Fatalf("Parse: %v", err)
}
if _, err := m.Generate(context.Background(), llm.Request{Messages: []llm.Message{llm.UserText("hi")}}); err != nil {
t.Fatalf("Generate: %v", err)
}
if rt.req == nil {
t.Fatal("no request captured")
}
if want := "Bearer openai-secret"; rt.req.Header.Get("Authorization") != want {
t.Errorf("Authorization = %q, want %q — the qwen credential must not reach the openai built-in",
rt.req.Header.Get("Authorization"), want)
}
}
// TestQwenReasoningEffortReachesWire is the load-bearing test for ADR-0027's
// central claim: Model Studio's OpenAI-compatible surface takes reasoning as a
// top-level "reasoning_effort" body field, which the openai client already
// sends — so llm.WithReasoningEffort survives the trip on qwen with no
// qwen-specific code. Routing qwen through the anthropic client instead would
// drop it silently (provider/anthropic ignores ReasoningEffort by design), and
// that difference would be invisible without asserting on the wire body.
func TestQwenReasoningEffortReachesWire(t *testing.T) {
rt := &captureRT{body: chatCompletionOK}
r := newTestRegistry(t,
WithEnvLookup(singleKeyEnv("QWEN_API_KEY", "qwen-secret")),
WithHTTPClient(&http.Client{Transport: rt}),
)
m, err := r.Parse("qwen/qwen3.8-max")
if err != nil {
t.Fatalf("Parse: %v", err)
}
_, err = m.Generate(context.Background(), llm.Request{
Messages: []llm.Message{llm.UserText("hi")},
ReasoningEffort: "high",
})
if err != nil {
t.Fatalf("Generate: %v", err)
}
if rt.reqBody == nil {
t.Fatal("no request body captured")
}
var sent map[string]any
if err := json.Unmarshal(rt.reqBody, &sent); err != nil {
t.Fatalf("decode request body: %v", err)
}
if got := sent["reasoning_effort"]; got != "high" {
t.Errorf("reasoning_effort = %v, want %q (body: %s)", got, "high", rt.reqBody)
}
}
+7 -5
View File
@@ -38,11 +38,13 @@ features leak into the canonical API).
`Unload` (POST `/api/models/unload[/:model]`). A small `doJSON` helper shares
bearer auth + error mapping; non-2xx → `*llm.APIError` (so `llm.Classify`
applies), transport errors wrapped raw.
- DSN: the `llama-swap` scheme builds an **http://** base URL from the host
(llama-swap is local-first), deliberately *not* the DSN's https-always
`BaseURL()`. A TLS-fronted instance can use the `openai://` scheme for chat.
A no-DSN built-in `llama-swap` provider registers but errors on use (mirrors
foreman).
- DSN: two schemes share one factory. `llama-swap` builds an **http://** base
URL from the host (llama-swap is local-first), deliberately *not* the DSN's
https-always `BaseURL()`; `llama-swaps` builds **https://** for a TLS-fronted
instance (mirrors redis/rediss). Why a second scheme rather than auto-detect:
a DSN carries no reliable http-vs-https signal, so the choice stays explicit.
Only `llama-swap` registers a no-DSN built-in provider (errors on use, mirrors
foreman); `llama-swaps` is a scheme only.
- Image generation is implemented here too, against the new `imagegen`
interface (see ADR-0016).
+19
View File
@@ -42,3 +42,22 @@ asked for "a new ai image interface as opposed to llm".
callers (additive fields/options).
- No health/failover for image models yet; if needed it can be added as a
separate chain type rather than retrofitting the chat chain.
## Update — optional per-request settings
`Request` gained additive optional overrides — `Steps *int`, `CFGScale *float64`,
`NegativePrompt string`, `Sampler string`, `Seed *int64` — with mirror options
(`WithSteps`, …). nil/"" means "leave the backend's per-model default", so the v1
contract is unchanged for callers that don't set them. `provider/llamaswap`
forwards them to sd-server as `steps`/`cfg_scale`/`negative_prompt`/`sample_method`/
`seed` (omitempty). This realizes the "seeds/steps … additive fields" note above;
img2img/masks/streaming remain deferred.
## Update — A1111 txt2img endpoint (seed support)
`provider/llamaswap` now POSTs to sd-server's **`/sdapi/v1/txt2img`** (A1111)
instead of the OpenAI `/v1/images/generations`. That OpenAI endpoint **ignores
`seed`** on the stable-diffusion.cpp build we run — every render of a prompt is
byte-identical, so a batch of N collapses to one image. `/sdapi/v1/txt2img`
honours `seed`, restoring real per-render variety. llama-swap still routes by
the `model` field in the body; `Size` is split into `width`/`height`.
+53
View File
@@ -0,0 +1,53 @@
# ADR-0017: audio — canonical speech synthesis + transcription interfaces
**Status:** Accepted — 2026-07-11
## Context
mort is growing agent tools that speak (TTS) and transcribe audio through a
llama-swap host whose upstreams expose the OpenAI `/v1/audio/speech` and
`/v1/audio/transcriptions` endpoints (kokoro, chatterbox, whisper.cpp). Like
image generation before it (ADR-0016), speech shares none of the chat
contract's message/tool/stream machinery, and majordomo had no speech surface
— an earlier migration doc explicitly scoped transcription out of the llm
package. The same reasoning that produced `imagegen` applies.
## Decision
- One new canonical **leaf package `audio`** holding both directions —
synthesis and transcription — rather than two packages; they are one
modality and will share future types (voice metadata, audio formats).
Root re-exports mirror imagegen (`SpeechModel`, `SpeechProvider`,
`SpeechRequest`, `SpeechResult`, `TranscriptionModel`, ...).
- Minimal v1 surface, imagegen conventions throughout (zero value = backend
default, functional options + `Apply`, `Raw any` escape hatch):
- `SpeechRequest{ Input; Voice; Format; Speed }`
`SpeechResult{ Audio []byte; MIME string; Raw }`;
`SpeechModel.Speak(ctx, req, ...opts)`;
`SpeechProvider.SpeechModel(id, ...)`.
- `TranscriptionRequest{ Audio []byte; MIME; Filename; Language; Prompt }`
`TranscriptionResult{ Text string; Raw }`;
`TranscriptionModel.Transcribe(ctx, req, ...opts)`;
`TranscriptionProvider.TranscriptionModel(id, ...)`.
- **Bytes in/out, never URLs** — mirrors `llm.ImagePart`'s bytes-only
contract; fetching is the caller's concern.
- Providers are split (`SpeechProvider` vs `TranscriptionProvider`) so a
backend can implement either half; llamaswap implements both.
- First implementation: `provider/llamaswap``/v1/audio/speech` (JSON body,
raw audio response; MIME from Content-Type with a format-based fallback),
`/v1/audio/transcriptions` (multipart, `response_format=json`), plus
`ListVoices(ctx, model)` (GET `/v1/audio/voices?model=`, tolerant of the
string-list and object-list shapes upstreams use) as a llamaswap management
method, not part of the canonical contract.
- Out of scope for v1 (designed-for, deferred): streaming synthesis,
word-level timestamps/segments, translation, voice cloning inputs, and
registry-level DSN resolution for audio models.
## Consequences
- Speech is provider-agnostic from day one; an OpenAI or Google speech
backend implements the same interfaces.
- `SpeechResult.Audio` is a plain byte slice, so results flow into any file
store or attachment pipeline without a majordomo dependency.
- The `audio` package name is the modality, not the direction; if music/sfx
generation ever lands it has a home.
+36
View File
@@ -0,0 +1,36 @@
# ADR-0018: imagegen.Editor — image-to-image as a separate optional interface
**Status:** Accepted — 2026-07-11
## Context
ADR-0016 shipped text-to-image and explicitly deferred img2img. mort's new
llama-swap media tools need "edit this image under this prompt"
(image-to-image with a denoising strength). Two shape questions: does Edit
belong on `imagegen.Model`, and which llama-swap endpoint carries it —
OpenAI-style `/v1/images/edits` (multipart) or A1111-style `/sdapi/v1/img2img`
(JSON)?
## Decision
- **`Editor` is a separate, optional interface** (`Edit(ctx, EditRequest,
...EditOption) (*Result, error)`), not a new method on `Model`. Existing
`Model` implementations keep compiling; callers type-assert
(`m.(imagegen.Editor)`) or require the capability explicitly.
- `EditRequest` = the generation knobs (prompt, N, size, steps, cfg, negative
prompt, sampler, seed) plus `Init Image` (required) and `Strength *float64`
(denoising strength in [0,1]; nil = backend default). Same option/Apply
conventions; result type is the shared `Result`.
- llama-swap implements it via **`/sdapi/v1/img2img`**, not
`/v1/images/edits`: the same sd-server build that ignores `seed` on the
OpenAI images route (the reason txt2img went SDAPI in ADR-0016's
implementation) applies; the JSON shape is txt2img's plus
`init_images: ["<b64>"]` + `denoising_strength`, so it reuses `doJSON`
verbatim, where the OpenAI route is multipart.
## Consequences
- Backends that can't edit simply don't implement `Editor`; no stub methods.
- The init image travels base64-inline in JSON (~33% overhead) — acceptable at
chat-image sizes; a future backend needing multipart can still satisfy the
same interface.
+49
View File
@@ -0,0 +1,49 @@
# ADR-0019: videogen — canonical video-generation surface
**Status:** Accepted — 2026-07-12
## Context
mort is adding local video generation (Wan 2.2 / LTX-class models on a 24GB
GPU) behind the same llama-swap instance that serves imagegen and audio. Like
those modalities, video generation shares none of the chat machinery, so it
needs its own small contract package (the ADR-0016/0017 pattern). Three shape
questions: one interface for text-to-video and image-to-video or two, batch or
single result, and which wire endpoint the llama-swap provider targets.
## Decision
- **New `videogen/` package** with the established conventions: `Request` /
`Result` / `Option` + `Apply`, `Model` / `ModelOption` / `Provider`, zero
values mean backend default, `Image = llm.ImagePart`.
- **Text-to-video and image-to-video are one surface.** `Request.InitImage
*Image` (nil = pure text-to-video) instead of an imagegen-style separate
`Editor` interface: hybrid checkpoints (Wan 2.2 TI2V) serve both modes from
the same model and endpoint, so a second interface would duplicate the
request shape for no dispatch benefit.
- **`Result` carries a single `Video`, not a batch.** The blocking sync
endpoint answers with the encoded clip as the response body — one request,
one clip. Batching multi-minute generations behind one HTTP request is the
wrong shape; if batch ever matters it arrives with an async job surface,
not by widening this one.
- **provider/llamaswap targets `POST {base}/v1/videos/sync`**
(multipart/form-data, llama-swap routes by the `model` form field; the
response body is the video). This is vLLM-Omni's blocking videos endpoint,
and the steve/llama-swap fork dispatches it as a model route. Parameter
names follow vLLM-Omni (`num_frames`, `fps`, `num_inference_steps`,
`guidance_scale`); the conditioning frame is an `input_reference` file part
per OpenAI's videos API. Optional fields stay off the wire so per-model
launch-flag defaults apply — the imagegen convention.
- **No polling in v1.** The async `POST /v1/videos` + `GET /v1/videos/{id}`
job flow is deliberately not wrapped: callers (mort's skill tools) already
run synchronous-with-generous-timeout and bound the call with a context
deadline. An async `Job` surface is a compatible later addition.
## Consequences
- A `videogen.Video` is its own type (`Data []byte`, `MIME string`); there is
no `llm.VideoPart`, and no chat-side video-input support is implied.
- Duration is expressed as `NumFrames` + `FPS` (the diffusion-native knobs),
not seconds; callers wanting seconds convert at their edge.
- Any upstream exposing the same `/v1/videos/sync` shape (e.g. a ComfyUI
shim) works unchanged; the contract does not name an engine.
@@ -0,0 +1,62 @@
# ADR-0020: Upstream-passthrough media surfaces (mask, upscale, background removal, interpolation, diarization, meshgen)
Status: Accepted (2026-07-12)
## Context
The llama-swap host is growing capabilities whose native HTTP APIs are not
OpenAI-shaped and carry no routable `model` field: rembg (`/api/remove`),
a Real-ESRGAN+RIFE shim (`/v1/upscale`, `/v1/interpolate`), a WhisperX
diarization sidecar (`/asr?diarize=true`), and Hunyuan3D (`/generate`).
llama-swap's fork already exposes a generic passthrough —
`/upstream/<model>/<any path>` — that pins the model, runs the normal
load/swap queue, and proxies the rest of the path verbatim.
Separately, sd-server's `/sdapi/v1/img2img` accepts an inpainting `mask`
field that `imagegen.EditRequest` could not express.
## Decision
1. **Route odd-shaped upstreams through `/upstream/<model>/<path>`** via a
shared `upstreamPath` helper (model-id validation identical to `Unload`:
reject `/?#`, never escape). No per-endpoint llama-swap routes; the
passthrough is the contract.
2. **Grow the existing leaf contracts instead of inventing parallel ones**,
keeping the ADR-0016→0019 conventions (functional options, zero value =
backend default, bytes-only I/O, `Raw` escape hatch, provider mints
model):
- `imagegen.EditRequest.Mask` (white = repaint, black = keep; backends
without mask support must reject, not ignore).
- `imagegen.Upscaler` / `UpscaleProvider` — super-resolution is not a
diffusion model, so it binds its own backend id.
- `imagegen.BackgroundRemover` / `BackgroundRemovalProvider` — request
`Net` selects the remover's internal network (rembg `model=` param);
`OnlyMask` returns the segmentation mask (the natural mask source for
inpainting).
- `videogen.Interpolator` / `InterpolationProvider` — fps boost or
slow-mo (`SlowMo` plays synthesized frames at the source rate; audio
stripped by the backend in that mode).
- `audio.DiarizationModel` / `DiarizationProvider` — speaker-labelled
segments; per-file `SPEAKER_NN` labels are explicitly NOT stable
identities.
3. **New `meshgen` leaf package** for image→3D (`Mesh{Data, Format, MIME}`,
formats glb/stl/obj). Image-to-3D only: text-to-3D is the caller-owned
composition imagegen → meshgen.
4. **Binary success bodies are validated before wrapping**: one-image
results reject non-image payloads, interpolation reuses the videogen
MIME rules and 512MB cap, meshes reject JSON-shaped "success" bodies
(a queue-full detail page must never become "the mesh"), and
diarization requires `output=json` because the vtt/srt shapes drop
speaker labels.
## Consequences
- provider/llamaswap gains five surfaces with no new wire machinery beyond
`upstreamPath` + a shared one-file-multipart builder.
- The passthrough couples majordomo to the fork's `/upstream` route
(upstream llama-swap has it too) and to each upstream's native API shape;
those shapes are pinned by the netherstorm image builds, not by version
negotiation — smoke tests on the host are the drift defence.
- `upstream.ignorePaths` (llama-swap config) can 409 asset-looking paths on
cold models; none of the paths used here match the default pattern, but
new surfaces must check.
+35
View File
@@ -0,0 +1,35 @@
# ADR-0021: musicgen interface (blocking Generate over an async job queue)
Status: Accepted (2026-07-12)
## Context
The llama-swap host gained ACE-Step 1.5 (full songs with vocals, seconds per
clip warm on the reference GPU). Its API is an async job queue —
`POST /release_task` → poll `POST /query_result``GET` the result file —
unlike every other majordomo media backend, which is one blocking call.
## Decision
- New `musicgen` leaf package, ADR-0016→0020 conventions: `Request{Prompt,
Lyrics, DurationSeconds, Format, Steps, Seed}`, `Result{Audio{Data, MIME},
Raw}`, functional options, zero value = backend default, bytes-only.
- **`Model.Generate` blocks, polling internally** (2s interval, ctx-bounded).
The one-call contract is the package's value: callers budget the whole job
with a context deadline exactly like imagegen/videogen, and the async
mechanics stay a provider detail. An async job surface (mirroring the
deliberately-deferred videogen one, ADR-0019) can come later if a caller
ever needs progress.
- provider/llamaswap reaches ACE-Step through the `/upstream/<model>/`
passthrough (ADR-0020). Envelope parsing is tolerant (`data` wrapper or
bare array; `result` arrives as a double-encoded JSON string) and the
result-file URL is routed back through the same upstream. `audio_duration`
is the v1 param name — an unknown field degrades to default clip length
upstream, never an error; verify at smoke.
## Consequences
- Music jobs occupy the exclusive GPU group like video; callers own the
timeout budget (mort keeps the tool timeout under its agent ceiling).
- The double-encoded `result` string and envelope shapes are pinned by the
netherstorm image build; host smoke tests are the drift defence.
@@ -0,0 +1,41 @@
# ADR-0022: embeddings + rerank interface
Status: Accepted (2026-07-12)
## Context
majordomo had no embedding or reranking surface at all. The llama-swap host
now runs two persistent CPU-only llama-server members (Qwen3-Embedding-0.6B
via `/v1/embeddings`, bge-reranker-v2-m3 via `/v1/rerank`), and mort wants a
reranking stage in memory retrieval with embedding-backed retrieval as a
later step.
## Decision
- New `embeddings` leaf package with TWO half-surfaces, split like audio's
Speech/Transcription: `EmbedModel`/`EmbedProvider` and
`RerankModel`/`RerankProvider`. They are separate mints because on the
reference host they are two DIFFERENT server instances — llama-server with
`--embeddings` and `--rerank` together returns all-zero embeddings
(llama.cpp #20085) — and because rerankers are cross-encoders, not
embedders.
- `EmbedResult.Vectors [][]float32` in input order (provider must order by
the response's `index`, never trust wire order). No `dimensions` param:
llama-server doesn't implement it; Matryoshka truncation is caller-side.
- `InstructedQuery(task, query)` helper encodes the instruction-aware
asymmetry (queries wrapped, documents bare) so call sites can't silently
degrade retrieval by forgetting the prefix.
- `RerankResult` sorted by descending score; parser reads ONLY
`results[].index` and `results[].relevance_score` because llama-server
documents the shape as subject to change. Scores are model-specific —
comparable within one response only.
## Consequences
- Callers get vectors/scores with strict validation (count mismatch, index
out of range, empty vector are hard errors — a silently missing vector is
a retrieval bug factory).
- llama-server's rerank scoring has open correctness issues for some models
(llama.cpp #16407); consumers must validate against a fixture before
trusting scores in production (mort gates its memory-rerank convar on
exactly that).
+59
View File
@@ -0,0 +1,59 @@
# ADR-0023: Wave-3 image + document surfaces (segmentation, colorize, face restore, OCR)
Status: Accepted (2026-07-16)
## Context
The llama-swap host is gaining four wave-3 capabilities (spec:
mort docs/specs/2026-07-16-llamaswap-wave3.md): promptable segmentation
(GroundingDINO + SAM 2.1), photo colorization (DDColor), face restoration
(GFPGAN), and document OCR (Surya). All ride the `/upstream/<model>/<path>`
passthrough (ADR-0020); none of their native APIs are OpenAI-shaped.
## Decision
1. **Grow `imagegen` with three optional interfaces**, ADR-0016→0022
conventions (functional options, zero value = backend default, bytes-only
I/O, provider mints model):
- `imagegen.Segmenter` / `SegmentationProvider` — `SegmentationRequest
{Image, Prompt, Threshold}` → one grayscale mask image, WHITE = the
prompted region. Same polarity as `EditRequest.Mask` (white = repaint),
so the mask feeds inpainting directly; cutouts are derived client-side
(mask-as-alpha), one host call serving both. The client always sends
`output=mask` — the shim's `cutout`/`boxes` modes are not exposed.
- `imagegen.Colorizer` / `ColorizeProvider` — `ColorizeRequest{Image}`,
no knobs (the reference backend takes none; options reserved).
- `imagegen.FaceRestorer` / `FaceRestoreProvider` —
`FaceRestoreRequest{Image, Upscale}` (1 or 2, 0 = backend default).
2. **New `ocr` leaf package** rather than a method on an existing surface:
OCR is document-shaped (multi-page, PDFs), not image-generation-shaped.
`Request{Document, MIME, Filename, Languages, MaxPages}` →
`Result{Text, Pages []Page{Number, Text}, Raw}`. `Result.Text` is the
pages joined with a blank line; the per-line bbox/confidence/layout
detail stays in `Raw` (json.RawMessage) — exact text is the contract,
geometry is the escape hatch.
3. **provider/llamaswap wire shapes** (pinned by the netherstorm image
builds; host smoke tests are the drift defence):
- `POST /upstream/<id>/v1/segment` multipart `file`,`prompt`
[,`threshold`],`output=mask` → mask PNG.
- `POST /upstream/<id>/v1/colorize` multipart `file` → PNG.
- `POST /upstream/<id>/v1/restore_faces` multipart `file`[,`upscale`] → PNG.
- `POST /upstream/<id>/v1/ocr` multipart `file`[,`langs` (comma-joined),
`max_pages`] → JSON `{pages:[{number,text,lines,layout}]}`. The decode
is tolerant: a page without aggregate `text` joins its line texts; a
missing page `number` defaults to position. Zero pages is an error
(a blank page still arrives as a page), mirroring the "no transcript"
honesty rule.
4. **Binary success bodies are validated before wrapping** (ADR-0020 rule):
the three image surfaces reuse `singleImageResult` (positive evidence of
image-ness required), OCR requires decodable JSON.
## Consequences
- imagegen grows from five to eight optional surfaces; consumers
type-assert or use the provider methods directly, as before.
- `ocr` is the seventh leaf media package (imagegen, audio, videogen,
meshgen, musicgen, embeddings, ocr); the conventions have held across all
of them.
- PDF handling lives host-side (the shim rasterizes via pypdfium2);
majordomo ships bytes and never needs a PDF dependency.
+65
View File
@@ -0,0 +1,65 @@
# ADR-0024: Wave-3 audio surfaces (stems, SFX, speech enhance, voice clone, translate)
Status: Accepted (2026-07-16)
## Context
The llama-swap host is gaining wave-3 audio capabilities (spec: mort
docs/specs/2026-07-16-llamaswap-wave3.md): Demucs stem separation and
DeepFilterNet speech enhancement (both on the CPU `audioutils` shim), Stable
Audio Open sound effects (`sfxgen`), plus two upgrades to existing models —
chatterbox's stateless voice-clone route and whisper.cpp's per-request
translate flag (both verified against the live images 2026-07-16).
## Decision
1. **`audio.StemSeparator` / `StemSeparationProvider`** —
`StemSeparationRequest{Audio, MIME, Filename, Mode, Model, Format}`
`StemSeparationResult{Stems []Stem{Name, Audio, MIME}}`.
- `Mode` is the caller-facing split: `"two"` (vocals + accompaniment,
sent as Demucs' `two_stems=vocals`) or `"four"`; `""` = backend
default. `Model` selects the Demucs variant (`htdemucs`/`htdemucs_ft`),
mirroring `BackgroundRemovalRequest.Net`.
- **The wire format is a ZIP** (`POST /upstream/<id>/v1/stems`): four WAV
stems would blow any JSON-of-base64 budget. Entry name → stem name,
extension → MIME; entries may sit under a per-model directory. Unpacking
is bounded per entry (zip-bomb guard) and a non-zip 2xx body fails loud.
2. **`SFXModel` reuses `musicgen`** — a sound effect is a short audio clip
from a text prompt; only the provider method differs. The sfxgen route
(`POST /upstream/<id>/v1/sfx`, JSON `{prompt, seconds, steps?, cfg_scale?,
seed?}`) is SYNCHRONOUS (WAV body), unlike ACE-Step's job queue.
`musicgen.Request` gains `CFGScale *float64` for it; lyrics and non-wav
formats are rejected (the model can't honor them). The ~11s model ceiling
is the backend's to enforce.
3. **`audio.SpeechEnhancer` / `SpeechEnhancementProvider`** —
`EnhancementRequest{Audio, MIME, Filename}`
`POST /upstream/<id>/v1/enhance` → WAV. The result reuses `SpeechResult`;
audio-in/audio-out needs no new result type.
4. **Voice cloning is a `SpeechRequest` field, not a new surface**
`ReferenceAudio []byte` + `ReferenceMIME`. When set, the llamaswap
speech model switches from JSON `/v1/audio/speech` to multipart
`POST /upstream/<id>/v1/audio/speech/upload` (fields `input` +
`voice_file`; verified live: stateless per-request cloning, no voice
library). Voice/format/speed still ride when set; the clone route's MIME
fallback is wav (not the JSON route's mp3). Backends without cloning must
reject a reference-carrying request rather than silently ignore it.
5. **Translation is a `TranscriptionRequest` bool**`Translate` maps to
whisper.cpp's `translate=true` form field (server.cpp:534). Because that
server's language DEFAULT is `en` (which silently skips translation), the
provider forces `language=auto` when translating without an explicit
language hint; an explicit hint wins.
6. **Binary success bodies are validated before wrapping** (ADR-0020 rule):
sfx/enhance require positive evidence of audio-ness (declared audio/*
Content-Type or sniffed RIFF/WAVE), stems require a parseable zip with
at least one stem.
## Consequences
- `audio` grows from three surfaces to five; the SFX surface adds zero new
types (musicgen reuse) — one format→MIME table and one multipart builder
keep serving every audio endpoint.
- The clone-route switch means one `SpeechModel` can answer over two wire
shapes; tests pin both routes so a regression can't silently drop cloning.
- `two_stems` is hardwired to vocals: "isolate X vs the rest" for other
sources is a backend capability not exposed in v1 (add a field when a
caller needs it, not before).
+67
View File
@@ -0,0 +1,67 @@
# ADR-0025: Wave-3 video surfaces (lipsync, video matte, video upscale, chain jobs)
Status: Accepted (2026-07-16)
## Context
The llama-swap host is gaining wave-3 video capabilities (spec: mort
docs/specs/2026-07-16-llamaswap-wave3.md): SadTalker talking heads, Robust
Video Matting and per-frame Real-ESRGAN on the mediautils shim, and a
videoutils orchestrator that generates LONG videos as a chain of i2v
segments (generate → extract last frame → continue → concat → optional RIFE
smoothing), exposed as an async job API following the ACE-Step precedent.
## Decision
1. **Three new optional `videogen` interfaces**, ADR-0016→0024 conventions:
- `videogen.LipSyncer` / `LipsyncProvider` — `LipsyncRequest{Image,
Audio, AudioMIME, AudioFilename, Still, Enhance, Preprocess}` →
`POST /upstream/<id>/v1/talking_head` (multipart `image` + `audio`
file parts + optional `still`/`enhance`/`preprocess` fields) → mp4.
Sync and minutes-slow (Hunyuan precedent); context deadline is the
budget.
- `videogen.VideoBackgroundRemover` / `VideoBackgroundRemovalProvider` —
`VideoBackgroundRemovalRequest{Video, MIME, Filename, Output}` →
`POST /upstream/<id>/v1/video/matte`. `Output` ∈
`greenscreen_mp4` (universally playable) | `alpha_webm` (true
transparency); "" = backend default.
- `videogen.VideoUpscaler` / `VideoUpscaleProvider` —
`VideoUpscaleRequest{Video, MIME, Filename, Scale}` (2 or 4) →
`POST /upstream/<id>/v1/video/upscale` → mp4.
2. **The chain client is deliberately ASYNC** (`videogen.Chainer` /
`ChainerProvider`), unlike musicgen's blocking Generate (ADR-0021): a
chain holds the GPU through multiple model swaps for many minutes, and
the caller must be able to poll progress AND fetch completed segments
after a mid-chain failure — partial delivery is mandatory (never discard
multi-minute GPU output), which a blocking one-call contract cannot
express.
- `SubmitChain(ctx, ChainRequest{Segments[{Prompt,Seconds}], InitImage,
SmoothJoins, Size}) (jobID, error)` — JSON
`POST /upstream/<id>/v1/video/chain`, init image as base64
`init_image_b64` (JSON submit, not multipart, per the pinned host
contract).
- `ChainStatus(ctx, jobID) (*ChainJob{Status, Segment, Total,
SegmentIDs, Raw})` — `GET /v1/jobs/{id}`; polling doubles as a
liveness signal for the shim's idle TTL. Segment entries are tolerated
as strings or `{id|segment_id}` objects.
- `ChainResult(ctx, jobID)` / `ChainSegmentResult(ctx, jobID, n)` —
`GET /v1/jobs/{id}/result` and `/v1/jobs/{id}/segments/{n}`.
- Job ids are echoed server input: job paths reject ids carrying path
structure (`/?#`, `..`), the upstreamPath smuggling rule.
3. **Binary success bodies are validated before wrapping** (ADR-0020 rule):
all four clip-returning calls go through a shared `singleVideoResult`
(positive evidence of video-ness — declared video/* Content-Type or
sniffed mp4/webm magic — so a JSON status page or proxy error can never
become "the clip").
## Consequences
- videogen grows from two surfaces (Model, Interpolator) to six; the
chain client is the package's first async surface — the job-API shape
deferred in ADR-0019/0021 now exists where the workload actually
demands it.
- The wire shapes are pinned by the netherstorm videoutils/mediautils/
SadTalker image builds; host smoke tests are the drift defence.
- mort's long-video tool owns the poll loop, timeout budget, and
partial-result envelope; majordomo only guarantees the artifacts stay
fetchable.
+60
View File
@@ -0,0 +1,60 @@
# ADR-0026: Kimi (Moonshot AI) built-in provider
**Status:** Accepted — 2026-07-18
## Context
Moonshot AI's Kimi models (Kimi K2, `moonshot-v1-*`, and the vision variants)
are served over an OpenAI-compatible Chat Completions API at
`https://api.moonshot.ai/v1` (`https://api.moonshot.cn/v1` for China),
authenticated with a bearer key. mort wants Kimi as a first-class failover
tier, so `kimi/kimi-k2-...` should parse, chain, and alias out of the box with
a dedicated `KIMI_API_KEY` env var — the same ergonomics as `openai`,
`anthropic`, and `google`.
Two tensions:
- The wire protocol is byte-for-byte OpenAI Chat Completions, so a hand-rolled
client would duplicate `provider/openai` for zero gain (ADR-0007 forbids it),
exactly as ADR-0015 found for llama-swap.
- The README's current stance is that arbitrary OpenAI-compatible endpoints
(Groq, Together, …) are *consumer-registered*, not baked in. Blessing Kimi as
a built-in is a deliberate, narrow exception justified by the north star:
mort names Kimi directly in its tiers, and a built-in with `KIMI_API_KEY`
keeps mort's config free of boilerplate `openai.New(WithName/WithBaseURL)`
wiring.
## Decision
- **No new package.** The `kimi` built-in and `kimi://` DSN scheme both
construct `provider/openai` pointed at the Moonshot base URL — the chat path
inherits every openai feature/fix automatically (like llama-swap's chat).
- The built-in reads its key through the registry's injected `envLookup`
(`KIMI_API_KEY` only — no `MOONSHOT_API_KEY` alias, per the project owner) so
it stays hermetically testable via `WithEnvLookup`.
- **`WithAPIKey` is passed unconditionally, even when empty.** `openai.New`
defaults its key to `OPENAI_API_KEY`; without an explicit override an unset
`KIMI_API_KEY` would silently authenticate Kimi with the OpenAI key. Passing
the (possibly empty) lookup result severs that fallthrough.
- New `openai.WithAPIKeyName("KIMI_API_KEY")` option customizes only the
synthetic-401 missing-key hint (default `OPENAI_API_KEY`), so a keyless kimi
call tells the operator the *right* variable to set.
- The default endpoint is the international host (`kimiBaseURL`). The China
endpoint (or any other host) is reachable with a `kimi://` DSN, e.g.
`LLM_KCN=kimi://[email protected]/v1`. The `kimi://` scheme is an
OpenAI-compatible target labeled `kimi` with the same key-name hint; it is
intentionally near-identical to `openai://` — its value is a clear name in
specs and error reporting.
## Consequences
- `kimi/<model>` is first-class in Parse, chains, aliases, and health/failover
with no consumer wiring; model ids pass through verbatim (no catalog).
- Chat, streaming, tools, and structured output ride the openai client. Image
*inputs* work at the client level but only the Moonshot vision models accept
them (matrix footnote ³).
- `WithAPIKeyName` is a small, generally useful addition to `provider/openai`;
the default preserves existing behavior for every other openai-compat target.
- Blessing one third-party endpoint as a built-in sets a precedent; future ones
should clear the same bar (a named consumer needs it in-config), not be added
reflexively — `RegisterProvider`/`LLM_*` remain the path for the rest.
+102
View File
@@ -0,0 +1,102 @@
# ADR-0027: Qwen (Alibaba) built-in provider — OpenAI-compat, not Anthropic-compat
**Status:** Accepted — 2026-08-12
## Context
Alibaba's Qwen models (`qwen3.8-max`, `qwen3.7-plus`, the `qwen3-vl-*` vision
variants, …) are served from Model Studio / DashScope, and mort wants them as a
first-class failover tier with a dedicated `QWEN_API_KEY` — the same ergonomics
ADR-0026 gave Kimi.
Unlike Kimi, Model Studio exposes the same models over **two** protocols:
| | OpenAI-compatible | Anthropic-compatible |
|---|---|---|
| Base URL | `https://dashscope-intl.aliyuncs.com/compatible-mode/v1` | `https://dashscope-intl.aliyuncs.com/apps/anthropic` |
| Endpoints | full Chat Completions surface | `/v1/messages` only (no `/v1/models`) |
| Purpose | the documented developer API | a shim, documented around hosting Claude Code |
So the question this ADR answers is not "which client do we reuse" but
"which of Alibaba's two wire protocols does the built-in speak".
## Decision
**The `qwen` built-in and the `qwen://` DSN scheme speak OpenAI-compat**, over
`provider/openai` — no new package, mirroring ADR-0026 (kimi) and ADR-0015
(llama-swap chat). Default base URL is the international host; the China host
(`dashscope.aliyuncs.com/compatible-mode/v1`) and workspace-scoped regional
hosts are reachable with a `qwen://` DSN.
Credential handling is copied from kimi verbatim, because both of its rules
are load-bearing: `WithAPIKey` is passed unconditionally (even empty) so an
unset `QWEN_API_KEY` can never fall through to `openai.New`'s `OPENAI_API_KEY`
default, and `WithAPIKeyName("QWEN_API_KEY")` makes the synthetic-401 hint name
the variable the operator actually has to set.
The kimi and qwen DSN factories were identical, so they now share one
`openaiCompatScheme` helper — the next OpenAI-compat built-in gets the
credential and key-hint rules by construction rather than by copy.
### Why not the Anthropic-compatible endpoint
Every concrete difference favors OpenAI-compat *for this codebase*:
- **Reasoning survives the trip.** Model Studio takes `reasoning_effort` as a
top-level field on the OpenAI surface, which `provider/openai` already sends
`llm.WithReasoningEffort` works on qwen with zero qwen-specific code
(`TestQwenReasoningEffortReachesWire` asserts it on the wire). Down the
anthropic client it would be dropped in silence: `provider/anthropic`
deliberately ignores `Request.ReasoningEffort`, because first-party Claude
has no such knob.
- **Structured output would regress.** `provider/anthropic` implements
`Request.Schema` with the first-party GA `output_config.format` mechanism.
Alibaba's shim does not implement it; a compat endpoint that ignores an
unknown field returns unconstrained prose while still reporting success.
The OpenAI path sends `response_format: json_schema`, which Model Studio
supports natively on the Max/Plus families.
- **Cache accounting already lands.** Model Studio's implicit prefix cache
reports hits in `usage.prompt_tokens_details.cached_tokens`, which the openai
client already maps to `llm.Usage.CacheReadTokens`. The anthropic client
reads `cache_read_input_tokens`, a field the shim has no reason to emit.
- **Thinking content is discarded on the anthropic path anyway.**
`provider/anthropic` skips `thinking` blocks in both the buffered and
streaming decoders, so the shim's headline feature — first-class
`thinking: {type: "enabled", budget_tokens: N}` — buys majordomo nothing
today.
- **Smaller blast radius.** The anthropic client has no `WithAPIKeyName`
option, so a keyless qwen would tell the operator to set `ANTHROPIC_API_KEY`;
fixing that means changing the first-party Anthropic client to serve a
third-party shim.
- **It is the less-exercised surface.** The Anthropic endpoint is documented as
Messages-only, with a temperature range that differs from Anthropic's own
([0, 2) vs [0.0, 1.0]) — i.e. it is Qwen semantics wearing an Anthropic
envelope, not an Anthropic-equivalent target.
The one thing the Anthropic surface offers that OpenAI-compat does not is
explicit `cache_control` breakpoints reached through `Request.PromptCache`.
That is not a reason to route Qwen through it: Model Studio's implicit cache is
automatic and already metered, and if explicit breakpoints ever matter they
belong in `provider/openai` (Model Studio accepts `cache_control` on content
blocks there too), where every OpenAI-compat target would get them.
## Consequences
- `qwen/<model>` is first-class in Parse, chains, aliases, and health/failover
with no consumer wiring; model ids pass through verbatim (no catalog).
- Chat, streaming, tools, structured output, reasoning effort, and cached-token
accounting all ride the openai client and inherit its fixes.
- Image *inputs* work at the client level, but only the `qwen-vl-*` /
`qwen3-vl-*` models accept them (matrix footnote ⁴; ³ is kimi's).
- Two model-side quirks are Alibaba's, not majordomo's, and are left to the
caller rather than papered over: thinking is **on by default** on some models
(e.g. `qwen3.7-plus`), and Qwen3 *open-source* models require streaming when
thinking is enabled — a buffered `Generate` against one of those needs a
model that supports non-streaming thinking (the Max/Plus families do).
- If a future consumer genuinely needs the Anthropic surface, it is reachable
today without library changes:
`LLM_QWEN_ANTHROPIC=anthropic://[email protected]/apps/anthropic`
— with the reasoning/structured-output caveats above.
- Second third-party built-in after kimi. The ADR-0026 bar still holds: a named
consumer needs it in-config. `RegisterProvider`/`LLM_*` remain the path for
everything else.
+11
View File
@@ -20,3 +20,14 @@ One decision per file, append-only; supersede rather than rewrite.
| [0014](0014-conversion-driven-extensions.md) | Conversion-driven extensions (resolvers, typed tools, hooks, ops controls) | Accepted |
| [0015](0015-llama-swap-provider.md) | llama-swap provider — reuse openai for chat, tailored management + image | Accepted |
| [0016](0016-imagegen-interface.md) | imagegen — a canonical text-to-image interface | Accepted |
| [0017](0017-audio-interfaces.md) | audio — canonical speech synthesis + transcription interfaces | Accepted |
| [0018](0018-imagegen-editor.md) | imagegen.Editor — image-to-image as a separate optional interface | Accepted |
| [0019](0019-videogen-interface.md) | videogen — canonical video-generation surface | Accepted |
| [0020](0020-upstream-passthrough-media-surfaces.md) | Upstream-passthrough media surfaces (mask, upscale, background removal, interpolation, diarization, meshgen) | Accepted |
| [0021](0021-musicgen-interface.md) | musicgen — blocking Generate over an async job queue | Accepted |
| [0022](0022-embeddings-rerank-interface.md) | embeddings + rerank interface | Accepted |
| [0023](0023-image-doc-surfaces.md) | Wave-3 image + document surfaces (segmentation, colorize, face restore, OCR) | Accepted |
| [0024](0024-audio-wave3-surfaces.md) | Wave-3 audio surfaces (stems, SFX, speech enhance, voice clone, translate) | Accepted |
| [0025](0025-videogen-wave3-surfaces.md) | Wave-3 video surfaces (lipsync, video matte, video upscale, chain jobs) | Accepted |
| [0026](0026-kimi-builtin.md) | Kimi (Moonshot AI) built-in provider — reuse openai client, KIMI_API_KEY | Accepted |
| [0027](0027-qwen-builtin.md) | Qwen (Alibaba) built-in provider — OpenAI-compat, not Model Studio's Anthropic-compat endpoint | Accepted |
+166
View File
@@ -0,0 +1,166 @@
// Package embeddings is majordomo's canonical text-embedding and reranking
// surface (ADR-0022, following the ADR-0016→0021 leaf-contract lineage).
// Two small halves, split like audio's Speech/Transcription so backends can
// implement either:
//
// - EmbedModel turns texts into dense vectors (/v1/embeddings-style).
// - RerankModel scores documents against a query with a cross-encoder
// (/v1/rerank-style) — usually a DIFFERENT backend model than the
// embedder, hence a separate mint.
//
// Instruction-aware embedders (Qwen3-Embedding et al.) want queries wrapped
// as "Instruct: {task}\nQuery: {query}" while documents go in bare;
// InstructedQuery encodes that so callers don't hand-roll (and silently
// degrade retrieval) at each site.
package embeddings
import "context"
// EmbedRequest is a batch embedding request. Inputs are embedded
// independently; the result vector order matches the input order.
type EmbedRequest struct {
// Inputs are the texts to embed. Required (at least one).
Inputs []string
}
// EmbedResult is the canonical embedding result.
type EmbedResult struct {
// Vectors holds one embedding per input, in input order. Backends
// normalize per their own convention (llama-server: Euclidean-normalized).
Vectors [][]float32
// Raw is the provider-native response object. May be nil.
Raw any
}
// EmbedOption mutates an EmbedRequest before it is sent. Reserved: the
// request shape is deliberately minimal today.
type EmbedOption func(*EmbedRequest)
// Apply returns a copy of the request with all options applied.
func (r EmbedRequest) Apply(opts ...EmbedOption) EmbedRequest {
for _, opt := range opts {
opt(&r)
}
return r
}
// InstructedQuery wraps a retrieval QUERY for instruction-aware embedding
// models. Documents must NOT be wrapped — the asymmetry is the point, and
// getting it wrong silently costs retrieval quality. An empty task uses the
// generic web-search instruction the reference model was trained with.
func InstructedQuery(task, query string) string {
if task == "" {
task = "Given a web search query, retrieve relevant passages that answer the query"
}
return "Instruct: " + task + "\nQuery: " + query
}
// EmbedModel embeds texts as dense vectors.
type EmbedModel interface {
// Embed returns one vector per input, in input order.
Embed(ctx context.Context, req EmbedRequest, opts ...EmbedOption) (*EmbedResult, error)
}
// EmbedModelOption configures an EmbedModel at construction time. Reserved
// for future per-model settings.
type EmbedModelOption func(*EmbedModelConfig)
// EmbedModelConfig carries per-model construction settings.
type EmbedModelConfig struct{}
// ApplyEmbedModelOptions folds options into a config.
func ApplyEmbedModelOptions(opts []EmbedModelOption) EmbedModelConfig {
var cfg EmbedModelConfig
for _, opt := range opts {
opt(&cfg)
}
return cfg
}
// EmbedProvider mints embedding models bound to one backend.
type EmbedProvider interface {
// Name is the registry identifier for the provider.
Name() string
// EmbedModel returns an EmbedModel bound to the given id (passed through
// to the backend verbatim; no catalog validation).
EmbedModel(id string, opts ...EmbedModelOption) (EmbedModel, error)
}
// RerankRequest scores documents against a query.
type RerankRequest struct {
// Query is the search query. Required.
Query string
// Documents are the candidate texts to score. Required (at least one).
Documents []string
// TopN limits how many results the backend returns; 0 = all.
TopN int
}
// RerankItem is one scored document.
type RerankItem struct {
// Index is the document's position in the request's Documents slice.
Index int
// Score is the backend's relevance score (higher = more relevant).
// Scales are model-specific — compare within one response only.
Score float64
}
// RerankResult is the canonical rerank result, sorted by descending Score.
type RerankResult struct {
// Results are the scored documents (top-N when the request bounded it).
Results []RerankItem
// Raw is the provider-native response object. May be nil.
Raw any
}
// RerankOption mutates a RerankRequest before it is sent.
type RerankOption func(*RerankRequest)
// WithTopN limits how many results the backend returns.
func WithTopN(n int) RerankOption { return func(r *RerankRequest) { r.TopN = n } }
// Apply returns a copy of the request with all options applied.
func (r RerankRequest) Apply(opts ...RerankOption) RerankRequest {
for _, opt := range opts {
opt(&r)
}
return r
}
// RerankModel scores documents against a query with a cross-encoder.
type RerankModel interface {
// Rerank returns scored documents sorted by descending relevance.
Rerank(ctx context.Context, req RerankRequest, opts ...RerankOption) (*RerankResult, error)
}
// RerankModelOption configures a RerankModel at construction time. Reserved
// for future per-model settings.
type RerankModelOption func(*RerankModelConfig)
// RerankModelConfig carries per-model construction settings.
type RerankModelConfig struct{}
// ApplyRerankModelOptions folds options into a config.
func ApplyRerankModelOptions(opts []RerankModelOption) RerankModelConfig {
var cfg RerankModelConfig
for _, opt := range opts {
opt(&cfg)
}
return cfg
}
// RerankProvider mints rerank models bound to one backend.
type RerankProvider interface {
// Name is the registry identifier for the provider.
Name() string
// RerankModel returns a RerankModel bound to the given id (passed
// through to the backend verbatim; no catalog validation).
RerankModel(id string, opts ...RerankModelOption) (RerankModel, error)
}
+16 -2
View File
@@ -26,8 +26,9 @@ var ErrUnknownProvider = errors.New("unknown provider")
// authenticated with the bearer token "test-token".
type DSN struct {
// Scheme selects the provider implementation: "foreman", "ollama",
// "ollama-cloud", "openai", "anthropic", "google"/"gemini", or any
// custom scheme registered with RegisterScheme.
// "ollama-cloud", "openai", "kimi", "qwen", "anthropic",
// "google"/"gemini", "llama-swap"/"llama-swaps", or any custom scheme
// registered with RegisterScheme.
Scheme string
// Token is the provider secret (bearer token or API key); empty = none.
Token string
@@ -40,6 +41,19 @@ type DSN struct {
// env-defined providers always speak TLS).
func (d DSN) BaseURL() string { return "https://" + d.Host }
// envKeyForProvider returns the LLM_* variable that defines the provider named
// name: "m1" → LLM_M1, "my-prov" → LLM_MY_PROV.
//
// This is the single definition on purpose. Two call sites need byte-identical
// output and would drift apart in silence: lazy resolution reads this variable
// to find an unregistered provider, and openaiCompatScheme names it in the
// missing-key hint so a keyless DSN target tells the operator which variable to
// set. Those two were separate copies with a comment asserting they matched —
// a comment is not enforcement, this function is.
func envKeyForProvider(name string) string {
return "LLM_" + strings.ToUpper(strings.ReplaceAll(name, "-", "_"))
}
// ParseDSN parses a raw DSN string. The algorithm matches go-llm exactly:
// split on "://", then an optional "@" separates the token from the host;
// trailing slashes on the host are trimmed.
+76
View File
@@ -0,0 +1,76 @@
package imagegen
import "context"
// BackgroundRemovalRequest asks a matting/segmentation backend to cut the
// subject out of an image. Zero values mean "backend default" (ADR-0020).
type BackgroundRemovalRequest struct {
// Image is the image to process. Required.
Image Image
// Net selects the remover's internal network where the backend offers
// several (rembg: "u2net", "birefnet-general", "isnet-general-use", ...);
// "" = backend default. This is NOT the provider model id — that is fixed
// when the BackgroundRemover is minted.
Net string
// OnlyMask returns the black/white segmentation mask instead of the
// cutout — WHITE marks the SUBJECT. To inpaint (repaint) the subject,
// use it directly as EditRequest.Mask; to repaint the BACKGROUND,
// invert it first (EditRequest.Mask is white-means-repaint).
OnlyMask bool
}
// BackgroundRemovalOption mutates a BackgroundRemovalRequest before it is sent.
type BackgroundRemovalOption func(*BackgroundRemovalRequest)
// WithBackgroundNet selects the remover's internal network.
func WithBackgroundNet(n string) BackgroundRemovalOption {
return func(r *BackgroundRemovalRequest) { r.Net = n }
}
// WithOnlyMask requests the segmentation mask instead of the cutout.
func WithOnlyMask() BackgroundRemovalOption {
return func(r *BackgroundRemovalRequest) { r.OnlyMask = true }
}
// Apply returns a copy of the request with all options applied.
func (r BackgroundRemovalRequest) Apply(opts ...BackgroundRemovalOption) BackgroundRemovalRequest {
for _, opt := range opts {
opt(&r)
}
return r
}
// BackgroundRemover cuts subjects out of images (RGBA PNG with transparent
// background, or the mask alone with OnlyMask).
type BackgroundRemover interface {
// RemoveBackground returns the cutout (or mask) as a one-image Result.
RemoveBackground(ctx context.Context, req BackgroundRemovalRequest, opts ...BackgroundRemovalOption) (*Result, error)
}
// BackgroundRemovalModelOption configures a BackgroundRemover at construction
// time. Reserved for future per-model settings.
type BackgroundRemovalModelOption func(*BackgroundRemovalModelConfig)
// BackgroundRemovalModelConfig carries per-model construction settings.
type BackgroundRemovalModelConfig struct{}
// ApplyBackgroundRemovalModelOptions folds options into a config.
func ApplyBackgroundRemovalModelOptions(opts []BackgroundRemovalModelOption) BackgroundRemovalModelConfig {
var cfg BackgroundRemovalModelConfig
for _, opt := range opts {
opt(&cfg)
}
return cfg
}
// BackgroundRemovalProvider mints BackgroundRemovers bound to one backend.
type BackgroundRemovalProvider interface {
// Name is the registry identifier for the provider.
Name() string
// BackgroundRemovalModel returns a BackgroundRemover bound to the given
// id (passed through to the backend verbatim; no catalog validation).
BackgroundRemovalModel(id string, opts ...BackgroundRemovalModelOption) (BackgroundRemover, error)
}
+54
View File
@@ -0,0 +1,54 @@
package imagegen
import "context"
// ColorizeRequest asks a colorization backend (DDColor style) to add color to
// a grayscale/faded photo (ADR-0023).
type ColorizeRequest struct {
// Image is the image to colorize. Required.
Image Image
}
// ColorizeOption mutates a ColorizeRequest before it is sent. Reserved for
// future request settings (the reference backend takes no parameters).
type ColorizeOption func(*ColorizeRequest)
// Apply returns a copy of the request with all options applied.
func (r ColorizeRequest) Apply(opts ...ColorizeOption) ColorizeRequest {
for _, opt := range opts {
opt(&r)
}
return r
}
// Colorizer adds color to grayscale images.
type Colorizer interface {
// Colorize returns the colorized image as a one-image Result.
Colorize(ctx context.Context, req ColorizeRequest, opts ...ColorizeOption) (*Result, error)
}
// ColorizeModelOption configures a Colorizer at construction time. Reserved
// for future per-model settings.
type ColorizeModelOption func(*ColorizeModelConfig)
// ColorizeModelConfig carries per-model construction settings.
type ColorizeModelConfig struct{}
// ApplyColorizeModelOptions folds options into a config.
func ApplyColorizeModelOptions(opts []ColorizeModelOption) ColorizeModelConfig {
var cfg ColorizeModelConfig
for _, opt := range opts {
opt(&cfg)
}
return cfg
}
// ColorizeProvider mints Colorizers bound to one backend.
type ColorizeProvider interface {
// Name is the registry identifier for the provider.
Name() string
// ColorizeModel returns a Colorizer bound to the given id (passed through
// to the backend verbatim; no catalog validation).
ColorizeModel(id string, opts ...ColorizeModelOption) (Colorizer, error)
}
+130
View File
@@ -0,0 +1,130 @@
package imagegen
import "context"
// EditRequest is an image-to-image (edit) request: a prompt applied to an
// initial image. As with Request, zero values mean "backend default"
// (ADR-0018).
type EditRequest struct {
// Prompt is the text description of the desired edit.
Prompt string
// Init is the initial image the edit starts from. Required, EXCEPT when
// RefImages is set — see there.
Init Image
// RefImages carries reference images for INSTRUCTION-EDIT models
// (FLUX.1 Kontext, Qwen-Image-Edit), which are a different kind of edit
// from img2img and reach the model by a different path.
//
// img2img noises Init and denoises it back under the prompt: the prompt
// describes the DESIRED IMAGE, and how much of the original survives is a
// function of Strength. An instruction-edit model instead takes the
// picture as conditioning and the prompt as an INSTRUCTION about it
// ("change the sign to read OPEN"), leaving everything it was not asked
// to touch bit-for-bit intact — no mask, no strength, no compositing.
//
// Sending one of these models an Init instead of a RefImage does not
// degrade gracefully, it silently does the wrong thing: measured against
// FLUX.1-Kontext on 2026-07-30, "change the blue rectangle to green" via
// init_images left the rectangle blue and drifted every other region,
// while the same prompt via a reference image turned it green and left
// the rest of the frame numerically unchanged.
//
// When RefImages is non-empty, Init/Mask/Strength are IGNORED: they
// describe a pipeline this model does not run.
RefImages []Image
// Mask restricts the edit to a region (inpainting): a single-channel or
// RGB image the same size as Init where WHITE pixels are repainted and
// BLACK pixels are kept. Empty = whole-image edit. Backends without mask
// support must reject a masked request rather than silently ignoring it.
Mask Image
// Strength is the denoising strength in [0,1] — how far the result may
// depart from Init (0 = return the input, 1 = ignore it); nil = backend
// default.
Strength *float64
// N is the number of images to generate; 0 = provider default.
N int
// Size is the requested resolution, e.g. "1024x1024"; "" = provider
// default (usually the init image's own resolution).
Size string
// Steps is the number of diffusion steps; nil = backend default.
Steps *int
// CFGScale is the classifier-free-guidance scale; nil = backend default.
CFGScale *float64
// NegativePrompt steers generation away from concepts; "" = none.
NegativePrompt string
// Sampler selects the sampling method (e.g. "euler", "euler_a");
// "" = backend default.
Sampler string
// Seed fixes the RNG seed for reproducible output; nil = random.
Seed *int64
}
// EditOption mutates an EditRequest before it is sent. Options passed to Edit
// are applied to a copy of the request, so an EditRequest value can be reused.
type EditOption func(*EditRequest)
// WithEditMask restricts the edit to a region (white = repaint, black = keep).
func WithEditMask(m Image) EditOption { return func(r *EditRequest) { r.Mask = m } }
// WithEditRefImages supplies reference images for an instruction-edit model
// (Kontext / Qwen-Image-Edit). See EditRequest.RefImages — this selects a
// different edit path, not a variation on img2img.
func WithEditRefImages(imgs ...Image) EditOption {
return func(r *EditRequest) { r.RefImages = imgs }
}
// WithEditStrength sets the denoising strength in [0,1].
func WithEditStrength(s float64) EditOption { return func(r *EditRequest) { r.Strength = &s } }
// WithEditN sets the number of images to generate.
func WithEditN(n int) EditOption { return func(r *EditRequest) { r.N = n } }
// WithEditSize sets the requested resolution (e.g. "1024x1024").
func WithEditSize(size string) EditOption { return func(r *EditRequest) { r.Size = size } }
// WithEditSteps overrides the number of diffusion steps.
func WithEditSteps(n int) EditOption { return func(r *EditRequest) { r.Steps = &n } }
// WithEditCFGScale overrides the classifier-free-guidance scale.
func WithEditCFGScale(s float64) EditOption { return func(r *EditRequest) { r.CFGScale = &s } }
// WithEditNegativePrompt sets a negative prompt.
func WithEditNegativePrompt(s string) EditOption {
return func(r *EditRequest) { r.NegativePrompt = s }
}
// WithEditSampler overrides the sampling method.
func WithEditSampler(s string) EditOption { return func(r *EditRequest) { r.Sampler = s } }
// WithEditSeed fixes the RNG seed for reproducible output.
func WithEditSeed(seed int64) EditOption { return func(r *EditRequest) { r.Seed = &seed } }
// Apply returns a copy of the request with all options applied. Providers
// call this once at the top of Edit.
func (r EditRequest) Apply(opts ...EditOption) EditRequest {
for _, opt := range opts {
opt(&r)
}
return r
}
// Editor is the image-to-image surface. It is a separate, optional interface
// rather than a method on Model so existing Model implementations keep
// compiling; callers type-assert (`m.(imagegen.Editor)`) or require it
// explicitly.
type Editor interface {
// Edit produces one or more images derived from the request's init image
// under the request's prompt.
Edit(ctx context.Context, req EditRequest, opts ...EditOption) (*Result, error)
}
+26
View File
@@ -0,0 +1,26 @@
package imagegen
import "testing"
func TestEditRequestApply(t *testing.T) {
base := EditRequest{Prompt: "make it night", Init: Image{MIME: "image/png", Data: []byte{1}}}
got := base.Apply(WithEditStrength(0.7), WithEditN(2), WithEditSeed(42))
if got.Prompt != "make it night" || len(got.Init.Data) != 1 {
t.Errorf("got = %+v", got)
}
if got.Strength == nil || *got.Strength != 0.7 {
t.Errorf("Strength = %v, want 0.7", got.Strength)
}
if got.N != 2 {
t.Errorf("N = %d, want 2", got.N)
}
if got.Seed == nil || *got.Seed != 42 {
t.Errorf("Seed = %v, want 42", got.Seed)
}
// Apply must not mutate the receiver (options apply to a copy).
if base.Strength != nil || base.N != 0 || base.Seed != nil {
t.Errorf("base mutated: %+v", base)
}
}
+62
View File
@@ -0,0 +1,62 @@
package imagegen
import "context"
// FaceRestoreRequest asks a face-restoration backend (GFPGAN style) to repair
// degraded faces in a photo. Zero values mean "backend default" (ADR-0023).
type FaceRestoreRequest struct {
// Image is the image to restore. Required.
Image Image
// Upscale is the output enlargement factor (1 or 2 on the reference
// backend); 0 = backend default.
Upscale int
}
// FaceRestoreOption mutates a FaceRestoreRequest before it is sent.
type FaceRestoreOption func(*FaceRestoreRequest)
// WithFaceRestoreUpscale sets the output enlargement factor.
func WithFaceRestoreUpscale(n int) FaceRestoreOption {
return func(r *FaceRestoreRequest) { r.Upscale = n }
}
// Apply returns a copy of the request with all options applied.
func (r FaceRestoreRequest) Apply(opts ...FaceRestoreOption) FaceRestoreRequest {
for _, opt := range opts {
opt(&r)
}
return r
}
// FaceRestorer repairs degraded/blurry faces in photos.
type FaceRestorer interface {
// RestoreFaces returns the restored image as a one-image Result.
RestoreFaces(ctx context.Context, req FaceRestoreRequest, opts ...FaceRestoreOption) (*Result, error)
}
// FaceRestoreModelOption configures a FaceRestorer at construction time.
// Reserved for future per-model settings.
type FaceRestoreModelOption func(*FaceRestoreModelConfig)
// FaceRestoreModelConfig carries per-model construction settings.
type FaceRestoreModelConfig struct{}
// ApplyFaceRestoreModelOptions folds options into a config.
func ApplyFaceRestoreModelOptions(opts []FaceRestoreModelOption) FaceRestoreModelConfig {
var cfg FaceRestoreModelConfig
for _, opt := range opts {
opt(&cfg)
}
return cfg
}
// FaceRestoreProvider mints FaceRestorers bound to one backend.
type FaceRestoreProvider interface {
// Name is the registry identifier for the provider.
Name() string
// FaceRestoreModel returns a FaceRestorer bound to the given id (passed
// through to the backend verbatim; no catalog validation).
FaceRestoreModel(id string, opts ...FaceRestoreModelOption) (FaceRestorer, error)
}
+137
View File
@@ -0,0 +1,137 @@
package imagegen
import "context"
// FaceSwapRequest transfers an identity from Source into Target.
//
// This is a DIFFERENT OPERATION from Edit, not a better-tuned one. Measured
// against the instruction-edit models on 2026-07-31, asking a diffusion model
// to put a specific person's face into a photo does not work by any route —
// by name, by description, or by supplying the portrait as a reference image.
// Face swapping is a dedicated detect/align/blend pipeline; a provider that
// cannot do it should not pretend Edit is a substitute.
type FaceSwapRequest struct {
// Target is the photo to edit — the pose, expression, lighting and
// everything outside the face are preserved from it.
Target Image
// Source is a photo of the face to put in. Only the identity travels;
// the source's own pose and expression do not.
Source Image
// Index selects WHICH face in Target, in the provider's documented
// ordering (llamaswap: left to right by box centre, as reported by
// ListFaces). nil = the largest face, which is right for a portrait and
// wrong for a group — enumerate first when it matters.
Index *int
// All swaps every detected face and ignores Index.
All bool
}
// FaceSwapOption mutates a FaceSwapRequest before it is sent.
type FaceSwapOption func(*FaceSwapRequest)
// WithFaceIndex selects which face in the target to replace.
func WithFaceIndex(i int) FaceSwapOption { return func(r *FaceSwapRequest) { r.Index = &i } }
// WithAllFaces swaps every detected face.
func WithAllFaces() FaceSwapOption { return func(r *FaceSwapRequest) { r.All = true } }
// Apply returns a copy of the request with all options applied.
func (r FaceSwapRequest) Apply(opts ...FaceSwapOption) FaceSwapRequest {
for _, opt := range opts {
opt(&r)
}
return r
}
// DetectedFace is one face located in an image, in PIXEL coordinates.
type DetectedFace struct {
// Index is the face's position in the provider's stable ordering, and
// the value FaceSwapRequest.Index expects.
Index int
// Box is [x0, y0, x1, y1].
Box [4]int
// Score is the detector's confidence, 0-1.
Score float64
// Yaw is how far the head is turned from camera, in degrees, or nil when
// the provider does not report pose. Exposed on ENUMERATION, not just
// after the fact, because it is how a caller picks a face a swap will
// actually work on: past roughly ±45° the features carrying identity are
// edge-on, and the result reads as a generic person however good the
// transfer is. A bounding box cannot show this.
Yaw *float64
}
// Size returns the box dimensions. Derived rather than stored: carrying
// width/height alongside Box is two sources of truth for one fact, and the
// pair can disagree after any transform.
func (f DetectedFace) Size() (w, h int) {
return f.Box[2] - f.Box[0], f.Box[3] - f.Box[1]
}
// SwappedFace is the MEASURED outcome for one face the provider replaced.
//
// It exists because "the call returned an image" and "the likeness
// transferred" are different claims that look identical from outside, and a
// caller that cannot tell them apart will go looking for another way to
// check. The one it reaches for — asking a vision model who the result looks
// like — is wrong in exactly the cases that matter: a VLM shown a jogger in a
// Georgetown cap holding McDonald's cups answers "Bill Clinton" whoever's
// face is on him, so it reports failure on a correct swap.
type SwappedFace struct {
// Index is the face's position in the provider's left-to-right ordering.
Index int
// Width, Height are the replaced face's pixel size in the TARGET.
// Meaningful only against ImageWidth/ImageHeight: a 138px face is large
// in a 400px picture and nearly invisible in a 2000px one, and it is the
// ratio, not the absolute size, that decides whether a person notices.
Width, Height int
// ImageWidth, ImageHeight are the target image's dimensions, repeated on
// every entry so a single face is self-describing without the caller
// holding onto the rest of the response.
ImageWidth, ImageHeight int
// Yaw is how far the head is turned from camera, in degrees, or nil when
// the provider does not report pose. The best single predictor of whether
// a swap will READ as the source person: past roughly ±45° the features
// carrying identity are edge-on and the result looks like a generic
// person rather than a specific one.
Yaw *float64
// IdentitySimilarity is cosine similarity between the source face and the
// face actually present in the result, 0-1, or nil when the provider
// could not measure it. Above ~0.5 the identity transferred; a LOW value
// is the only evidence that a swap genuinely failed.
IdentitySimilarity *float64
}
// FractionOfImage is the swapped face's width as a share of the image's, 0-1.
// The number that predicts whether a person will SEE the change: the swap
// that prompted all this replaced a 138px face in a 1010px-wide photo — 14%,
// correct by every measure and invisible at a glance — while the same code on
// a 168px face in a 385px picture (44%) is unmistakable. Returns 0 when the
// dimensions are unknown.
func (f SwappedFace) FractionOfImage() float64 {
if f.ImageWidth <= 0 || f.Width <= 0 {
return 0
}
return float64(f.Width) / float64(f.ImageWidth)
}
// FaceSwapper is the optional face-transfer surface. Separate interface so
// existing providers keep compiling; callers type-assert.
type FaceSwapper interface {
// ListFaces enumerates the faces in an image, in the SAME ordering
// FaceSwapRequest.Index uses. Exposed because a caller asked to change
// "the man on the right" needs a way to name one face and to check its
// own choice against pixel boxes.
ListFaces(ctx context.Context, img Image) ([]DetectedFace, error)
// FaceSwap transfers Source's identity into Target.
FaceSwap(ctx context.Context, req FaceSwapRequest, opts ...FaceSwapOption) (*Result, error)
}
+47
View File
@@ -38,6 +38,29 @@ type Request struct {
// Size is the requested resolution, e.g. "512x512" or "1024x1024";
// "" = provider default.
Size string
// The fields below are optional per-request overrides. Their zero value
// (nil pointer or empty string) means "leave the backend's own default" —
// for stable-diffusion.cpp that is the per-model default baked into the
// llama-swap launch flags. A caller overrides only what it explicitly sets.
// Steps is the number of diffusion steps; nil = backend default.
Steps *int
// CFGScale is the classifier-free-guidance scale; nil = backend default.
// Architecture-sensitive (SDXL likes ~7, Flux wants 1), so prefer leaving
// it nil unless the caller knows the target model.
CFGScale *float64
// NegativePrompt steers generation away from concepts; "" = none.
NegativePrompt string
// Sampler selects the sampling method (e.g. "euler", "euler_a");
// "" = backend default.
Sampler string
// Seed fixes the RNG seed for reproducible output; nil = random.
Seed *int64
}
// Result is the canonical image-generation result.
@@ -45,6 +68,15 @@ type Result struct {
// Images are the generated images, in the order the backend returned them.
Images []Image
// SwappedFaces is the measured outcome of a FaceSwap, one entry per face
// replaced. Empty for every other operation, and empty from a provider
// that does not measure. TYPED rather than tucked into Raw: a caller has
// to act on this — it is the only way to distinguish a swap that
// transferred the likeness from one that returned an image and nothing
// more — and a value reachable solely by type-asserting an `any` is one
// nobody discovers in time to use.
SwappedFaces []SwappedFace
// Raw is the provider-native response object, an escape hatch for
// provider-specific fields. May be nil; never required for normal use.
Raw any
@@ -60,6 +92,21 @@ func WithN(n int) Option { return func(r *Request) { r.N = n } }
// WithSize sets the requested resolution (e.g. "1024x1024").
func WithSize(size string) Option { return func(r *Request) { r.Size = size } }
// WithSteps overrides the number of diffusion steps.
func WithSteps(n int) Option { return func(r *Request) { r.Steps = &n } }
// WithCFGScale overrides the classifier-free-guidance scale.
func WithCFGScale(s float64) Option { return func(r *Request) { r.CFGScale = &s } }
// WithNegativePrompt sets a negative prompt.
func WithNegativePrompt(s string) Option { return func(r *Request) { r.NegativePrompt = s } }
// WithSampler overrides the sampling method (e.g. "euler", "euler_a").
func WithSampler(s string) Option { return func(r *Request) { r.Sampler = s } }
// WithSeed fixes the RNG seed for reproducible output.
func WithSeed(seed int64) Option { return func(r *Request) { r.Seed = &seed } }
// Apply returns a copy of the request with all options applied. Providers call
// this once at the top of Generate.
func (r Request) Apply(opts ...Option) Request {
+70
View File
@@ -0,0 +1,70 @@
package imagegen
import "context"
// SegmentationRequest asks a promptable-segmentation backend (GroundingDINO +
// SAM style) for the mask of a text-described region. Zero values mean
// "backend default" (ADR-0023).
type SegmentationRequest struct {
// Image is the image to segment. Required.
Image Image
// Prompt is the text description of the region to segment (e.g. "the red
// car"). Required — promptless segmentation is a different capability.
Prompt string
// Threshold is the detection confidence threshold in (0,1]; 0 = backend
// default.
Threshold float64
}
// SegmentationOption mutates a SegmentationRequest before it is sent.
type SegmentationOption func(*SegmentationRequest)
// WithSegmentationThreshold sets the detection confidence threshold.
func WithSegmentationThreshold(t float64) SegmentationOption {
return func(r *SegmentationRequest) { r.Threshold = t }
}
// Apply returns a copy of the request with all options applied.
func (r SegmentationRequest) Apply(opts ...SegmentationOption) SegmentationRequest {
for _, opt := range opts {
opt(&r)
}
return r
}
// Segmenter extracts a prompted region's mask from an image. The result is a
// single grayscale mask image where WHITE marks the prompted region — the
// same polarity as EditRequest.Mask (white = repaint), so it feeds inpainting
// directly; derive a cutout client-side by applying the mask as alpha.
type Segmenter interface {
// Segment returns the region mask as a one-image Result.
Segment(ctx context.Context, req SegmentationRequest, opts ...SegmentationOption) (*Result, error)
}
// SegmentationModelOption configures a Segmenter at construction time.
// Reserved for future per-model settings (mirrors ModelOption).
type SegmentationModelOption func(*SegmentationModelConfig)
// SegmentationModelConfig carries per-model construction settings.
type SegmentationModelConfig struct{}
// ApplySegmentationModelOptions folds options into a config.
func ApplySegmentationModelOptions(opts []SegmentationModelOption) SegmentationModelConfig {
var cfg SegmentationModelConfig
for _, opt := range opts {
opt(&cfg)
}
return cfg
}
// SegmentationProvider mints Segmenters bound to one backend.
type SegmentationProvider interface {
// Name is the registry identifier for the provider.
Name() string
// SegmentationModel returns a Segmenter bound to the given id (passed
// through to the backend verbatim; no catalog validation).
SegmentationModel(id string, opts ...SegmentationModelOption) (Segmenter, error)
}
+62
View File
@@ -0,0 +1,62 @@
package imagegen
import "context"
// UpscaleRequest asks a super-resolution backend to enlarge an image. Zero
// values mean "backend default", mirroring EditRequest (ADR-0020).
type UpscaleRequest struct {
// Image is the image to upscale. Required.
Image Image
// Scale is the enlargement factor (2 or 4 on the reference backend);
// 0 = backend default.
Scale int
}
// UpscaleOption mutates an UpscaleRequest before it is sent.
type UpscaleOption func(*UpscaleRequest)
// WithUpscaleScale sets the enlargement factor.
func WithUpscaleScale(s int) UpscaleOption { return func(r *UpscaleRequest) { r.Scale = s } }
// Apply returns a copy of the request with all options applied.
func (r UpscaleRequest) Apply(opts ...UpscaleOption) UpscaleRequest {
for _, opt := range opts {
opt(&r)
}
return r
}
// Upscaler enlarges images with a super-resolution model. It is its own
// small interface (not a method on Model) because upscalers are not
// diffusion models — they bind to a different backend id entirely.
type Upscaler interface {
// Upscale returns the enlarged image (a one-image Result).
Upscale(ctx context.Context, req UpscaleRequest, opts ...UpscaleOption) (*Result, error)
}
// UpscaleModelOption configures an Upscaler at construction time. Reserved
// for future per-model settings (mirrors ModelOption).
type UpscaleModelOption func(*UpscaleModelConfig)
// UpscaleModelConfig carries per-model construction settings.
type UpscaleModelConfig struct{}
// ApplyUpscaleModelOptions folds options into a config.
func ApplyUpscaleModelOptions(opts []UpscaleModelOption) UpscaleModelConfig {
var cfg UpscaleModelConfig
for _, opt := range opts {
opt(&cfg)
}
return cfg
}
// UpscaleProvider mints Upscalers bound to one backend.
type UpscaleProvider interface {
// Name is the registry identifier for the provider.
Name() string
// UpscaleModel returns an Upscaler bound to the given id (passed through
// to the backend verbatim; no catalog validation).
UpscaleModel(id string, opts ...UpscaleModelOption) (Upscaler, error)
}
+21
View File
@@ -26,6 +26,7 @@ import (
"encoding/json"
"sync"
"gitea.stevedudenhoeffer.com/steve/majordomo/audio"
"gitea.stevedudenhoeffer.com/steve/majordomo/imagegen"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
@@ -67,6 +68,26 @@ type (
ImageResult = imagegen.Result
ImageOption = imagegen.Option
ImageModelOption = imagegen.ModelOption
ImageEditor = imagegen.Editor
ImageEditRequest = imagegen.EditRequest
ImageEditOption = imagegen.EditOption
)
// Re-exported canonical speech types. See the audio package for
// documentation. Speech synthesis and transcription are separate contracts
// from llm, mirroring imagegen (ADR-0017); the first backend is
// provider/llamaswap.
type (
SpeechModel = audio.SpeechModel
SpeechProvider = audio.SpeechProvider
SpeechRequest = audio.SpeechRequest
SpeechResult = audio.SpeechResult
SpeechOption = audio.SpeechOption
TranscriptionModel = audio.TranscriptionModel
TranscriptionProvider = audio.TranscriptionProvider
TranscriptionRequest = audio.TranscriptionRequest
TranscriptionResult = audio.TranscriptionResult
TranscriptionOption = audio.TranscriptionOption
)
// Re-exported role and finish-reason constants.
+41 -15
View File
@@ -5,10 +5,16 @@
// already satisfies the target's llm.Capabilities. Images that do not fit
// are decoded, downscaled (never upscaled), and re-encoded into an allowed
// format and byte budget. Anything that cannot honestly be made to fit —
// undecodable formats, impossible byte budgets, too many images, images for
// a text-only target — fails with an error wrapping llm.ErrUnsupported so a
// failover chain can advance to a more capable target without a health
// penalty.
// undecodable formats, impossible byte budgets, images for a text-only
// target — fails with an error wrapping llm.ErrUnsupported so a failover
// chain can advance to a more capable target without a health penalty.
//
// Over-count is the exception: a request carrying more images than
// MaxImagesPerReq does NOT fail — the oldest images are replaced with a short
// text placeholder and the most-recent MaxImagesPerReq are kept, because a hard
// refuse exhausts a chain whose targets share the same cap (e.g. an agent loop
// accumulating a preview image per iteration). MaxImagesPerReq remains the
// per-model knob (0 = no image support).
//
// Why a separate package: every provider would otherwise duplicate the same
// decode/scale/encode pipeline. Providers keep only a cheap capability
@@ -52,15 +58,21 @@ func Normalize(req llm.Request, caps llm.Capabilities) (llm.Request, error) {
if !caps.SupportsImages() {
return llm.Request{}, fmt.Errorf("media: %w: target does not accept image input (request carries %d image(s))", llm.ErrUnsupported, total)
}
// Why error instead of dropping the overflow: silently removing an image
// changes the question the caller asked; the honest move is to refuse and
// let a chain try a roomier target.
// Over-cap images are elided in the same copy-on-write pass below: the
// OLDEST excess are replaced with a placeholder and the most-recent
// MaxImagesPerReq kept (see the package doc for why we elide rather than
// refuse). toElide is how many of the first images, front-to-back, to drop.
toElide := 0
if total > caps.MaxImagesPerReq {
return llm.Request{}, fmt.Errorf("media: %w: request carries %d images, target allows at most %d per request", llm.ErrUnsupported, total, caps.MaxImagesPerReq)
toElide = total - caps.MaxImagesPerReq
}
// Single copy-on-write pass: for each image, the first toElide become a text
// placeholder; the rest are size-normalized against caps. The Messages slice
// and an affected message's Parts slice are copied at most once.
out := req
copiedMessages := false
seen := 0
for mi := range req.Messages {
copiedParts := false
for pi, part := range req.Messages[mi].Parts {
@@ -68,13 +80,22 @@ func Normalize(req llm.Request, caps llm.Capabilities) (llm.Request, error) {
if !ok {
continue
}
norm, changed, err := normalizeImage(ip, caps)
if err != nil {
return llm.Request{}, fmt.Errorf("media: message %d, part %d: %w", mi, pi, err)
}
if !changed {
continue
seen++
var replacement llm.Part
if seen <= toElide {
replacement = llm.Text(imageOverflowPlaceholder)
} else {
norm, changed, err := normalizeImage(ip, caps)
if err != nil {
return llm.Request{}, fmt.Errorf("media: message %d, part %d: %w", mi, pi, err)
}
if !changed {
continue
}
replacement = norm
}
if !copiedMessages {
out.Messages = make([]llm.Message, len(req.Messages))
copy(out.Messages, req.Messages)
@@ -86,12 +107,17 @@ func Normalize(req llm.Request, caps llm.Capabilities) (llm.Request, error) {
out.Messages[mi].Parts = parts
copiedParts = true
}
out.Messages[mi].Parts[pi] = norm
out.Messages[mi].Parts[pi] = replacement
}
}
return out, nil
}
// imageOverflowPlaceholder replaces an image elided to fit a target's
// per-request image cap. It keeps the message turn intact and tells the model
// an earlier image was omitted rather than silently changing the conversation.
const imageOverflowPlaceholder = "[earlier image omitted to fit this model's per-request image limit]"
// Info reports an image part's sniffed format ("jpeg", "png", "gif", or
// "webp") and pixel dimensions. It is a cheap metadata read — the pixels are
// never decoded. webp is recognized by signature but not decodable with the
+39 -9
View File
@@ -149,18 +149,48 @@ func TestNormalizeImagesUnsupported(t *testing.T) {
}
}
func TestNormalizeTooManyImages(t *testing.T) {
img := llm.Image("image/png", encPNG(t, gradient(4, 4)))
func TestNormalizeOverCount(t *testing.T) {
// 3 distinguishable images across 2 messages; cap = 2. Over-count no longer
// errors — the OLDEST image is replaced with a placeholder and the most-recent
// two (the relevant ones in an iterative run) are kept, in order.
a := llm.Image("image/png", encPNG(t, gradient(2, 2))).(llm.ImagePart)
b := llm.Image("image/png", encPNG(t, gradient(4, 4))).(llm.ImagePart)
c := llm.Image("image/png", encPNG(t, gradient(8, 8))).(llm.ImagePart)
req := llm.Request{Messages: []llm.Message{
llm.UserParts(img, img),
llm.UserParts(img),
llm.UserParts(a, b),
llm.UserParts(c),
}}
_, err := Normalize(req, llm.Capabilities{MaxImagesPerReq: 2})
if !errors.Is(err, llm.ErrUnsupported) {
t.Fatalf("err = %v, want ErrUnsupported", err)
caps := llm.Capabilities{MaxImagesPerReq: 2, MaxImageDimension: 64, MaxImageBytes: 1 << 20, AllowedImageMIME: []string{"image/png"}}
out, err := Normalize(req, caps)
if err != nil {
t.Fatalf("over-count should not error: %v", err)
}
if !strings.Contains(err.Error(), "3 images") || !strings.Contains(err.Error(), "at most 2") {
t.Errorf("err message %q lacks the counts", err)
var imgs []llm.ImagePart
placeholders := 0
for _, m := range out.Messages {
for _, p := range m.Parts {
switch v := p.(type) {
case llm.ImagePart:
imgs = append(imgs, v)
case llm.TextPart:
if v.Text == imageOverflowPlaceholder {
placeholders++
}
}
}
}
// The exact survivors are the most-recent two, in order: b then c (a elided).
if len(imgs) != 2 || !bytes.Equal(imgs[0].Data, b.Data) || !bytes.Equal(imgs[1].Data, c.Data) {
t.Fatalf("kept %d images; want exactly [b, c] (the most-recent two)", len(imgs))
}
if placeholders != 1 {
t.Errorf("placeholders = %d, want 1 for the elided oldest image", placeholders)
}
// Input request untouched (copy-on-write): the first part is still image a,
// not a placeholder — a len check alone wouldn't catch in-place substitution.
first, ok := req.Messages[0].Parts[0].(llm.ImagePart)
if !ok || !bytes.Equal(first.Data, a.Data) {
t.Errorf("input request was mutated; first part = %+v", req.Messages[0].Parts[0])
}
}
+172
View File
@@ -0,0 +1,172 @@
// Package meshgen is majordomo's canonical image-to-3D surface. Like
// imagegen/audio/videogen, it is a deliberately separate leaf contract from
// the llm package: mesh generation shares none of the chat machinery, so it
// gets its own small Provider/Model interfaces (ADR-0020, following the
// ADR-0016→0019 lineage: functional options, zero values = backend default,
// bytes-only I/O, Raw escape hatch).
//
// The first implementation is provider/llamaswap, which targets a
// Hunyuan3D-2.1-style api_server reached through the /upstream passthrough.
// The surface is image-to-3D only: text-to-3D is the composition
// imagegen.Generate → meshgen.Generate, owned by the caller.
package meshgen
import (
"context"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
// Image is the conditioning input (bytes + MIME), aliased to llm.ImagePart so
// chat- or imagegen-sourced images feed mesh generation without conversion.
type Image = llm.ImagePart
// Mesh is one generated 3D asset: raw encoded bytes plus its format.
type Mesh struct {
// Data is the encoded mesh (binary GLB/STL/OBJ container).
Data []byte
// Format is the lowercase container name ("glb", "stl", "obj").
Format string
// MIME is the corresponding MIME type, e.g. "model/gltf-binary",
// "model/stl".
MIME string
}
// Request is an image-to-3D generation request. Zero values mean "backend
// default".
type Request struct {
// Image is the source image the mesh is reconstructed from. Required.
Image Image
// Format is the requested output container ("glb", "stl", "obj");
// "" = backend default (glb).
Format string
// Texture asks the backend to also synthesize surface textures. Much
// slower and much more VRAM-hungry than shape-only on the reference
// backend; leave false for print-pipeline geometry.
Texture bool
// RemoveBackground lets the backend cut the subject out first; nil =
// backend default (true on Hunyuan3D). Set explicitly to false when the
// input is already a clean cutout.
RemoveBackground *bool
// OctreeResolution is the shape-decoder grid resolution; 0 = backend
// default.
OctreeResolution int
// Steps is the number of diffusion steps; nil = backend default.
Steps *int
// GuidanceScale is the guidance strength; nil = backend default.
GuidanceScale *float64
// Seed fixes the RNG seed for reproducible output; nil = backend default.
Seed *int64
// FaceCount caps the output mesh's face count; 0 = backend default.
FaceCount int
}
// Result is the canonical mesh-generation result.
type Result struct {
// Mesh is the generated asset.
Mesh Mesh
// Raw is the provider-native response object. May be nil.
Raw any
}
// Option mutates a Request before it is sent. Options passed to Generate are
// applied to a copy of the request, so a Request value can be reused.
type Option func(*Request)
// WithFormat sets the output container ("glb", "stl", "obj").
func WithFormat(f string) Option { return func(r *Request) { r.Format = f } }
// WithTexture enables texture synthesis.
func WithTexture() Option { return func(r *Request) { r.Texture = true } }
// WithRemoveBackground sets the backend's background-removal preprocessing.
func WithRemoveBackground(v bool) Option {
return func(r *Request) { r.RemoveBackground = &v }
}
// WithOctreeResolution sets the shape-decoder grid resolution.
func WithOctreeResolution(n int) Option { return func(r *Request) { r.OctreeResolution = n } }
// WithSteps overrides the number of diffusion steps.
func WithSteps(n int) Option { return func(r *Request) { r.Steps = &n } }
// WithGuidanceScale overrides the guidance strength.
func WithGuidanceScale(s float64) Option { return func(r *Request) { r.GuidanceScale = &s } }
// WithSeed fixes the RNG seed.
func WithSeed(seed int64) Option { return func(r *Request) { r.Seed = &seed } }
// WithFaceCount caps the output mesh's face count.
func WithFaceCount(n int) Option { return func(r *Request) { r.FaceCount = n } }
// Apply returns a copy of the request with all options applied. Providers
// call this once at the top of Generate.
func (r Request) Apply(opts ...Option) Request {
for _, opt := range opts {
opt(&r)
}
return r
}
// Model generates 3D meshes from images.
type Model interface {
// Generate reconstructs a mesh from the request's image.
Generate(ctx context.Context, req Request, opts ...Option) (*Result, error)
}
// ModelOption configures a Model at construction time. Reserved for future
// per-model settings.
type ModelOption func(*ModelConfig)
// ModelConfig carries per-model construction settings.
type ModelConfig struct{}
// ApplyModelOptions folds options into a config.
func ApplyModelOptions(opts []ModelOption) ModelConfig {
var cfg ModelConfig
for _, opt := range opts {
opt(&cfg)
}
return cfg
}
// Provider mints mesh models bound to one backend.
type Provider interface {
// Name is the registry identifier for the provider.
Name() string
// MeshModel returns a Model bound to the given id (passed through to the
// backend verbatim; no catalog validation).
MeshModel(id string, opts ...ModelOption) (Model, error)
}
// Converter re-encodes a mesh between containers (GLB/STL/OBJ). A separate
// optional surface because conversion typically runs on a DIFFERENT backend
// than generation (the reference host converts on its mediautils shim —
// Hunyuan3D's live server always emits GLB regardless of the requested
// format).
type Converter interface {
// Convert returns the mesh re-encoded in the given format.
Convert(ctx context.Context, mesh Mesh, format string) (*Result, error)
}
// ConverterProvider mints Converters bound to one backend.
type ConverterProvider interface {
// Name is the registry identifier for the provider.
Name() string
// MeshConverter returns a Converter bound to the given id (passed
// through to the backend verbatim; no catalog validation).
MeshConverter(id string) (Converter, error)
}
+128
View File
@@ -0,0 +1,128 @@
// Package musicgen is majordomo's canonical music-generation surface (full
// songs from a text prompt, optionally with lyrics). Like imagegen/audio/
// videogen/meshgen, it is a deliberately separate leaf contract from the llm
// package (ADR-0021, following the ADR-0016→0020 lineage: functional
// options, zero values = backend default, bytes-only I/O, Raw escape hatch).
//
// The first implementation is provider/llamaswap, which targets an
// ACE-Step-1.5-style async job API reached through the /upstream
// passthrough; Generate blocks (polling internally) so callers get the
// imagegen-style one-call contract — bound it with a context deadline.
package musicgen
import "context"
// Audio is one generated piece: raw encoded bytes plus a MIME type.
type Audio struct {
// Data is the encoded audio container.
Data []byte
// MIME is the audio MIME type, e.g. "audio/mpeg".
MIME string
}
// Request is a music-generation request. Zero values mean "backend default".
type Request struct {
// Prompt describes the music (genre, mood, instrumentation, tempo...).
Prompt string
// Lyrics are optional song lyrics for models that sing; "" =
// instrumental or model-written lyrics, per backend behavior.
Lyrics string
// DurationSeconds is the requested clip length; 0 = backend default.
DurationSeconds int
// Format is the audio container ("mp3", "wav", "flac", "opus", "aac");
// "" = backend default (mp3 on ACE-Step).
Format string
// Steps is the number of inference steps; nil = backend default.
Steps *int
// CFGScale is the classifier-free-guidance scale; nil = backend default.
// Architecture-sensitive, so prefer leaving it nil unless the caller
// knows the target model. Backends without the knob ignore it.
CFGScale *float64
// Seed fixes the RNG seed for reproducible output; nil = backend
// default (random).
Seed *int64
}
// Result is the canonical music-generation result.
type Result struct {
// Audio is the generated piece.
Audio Audio
// Raw is the provider-native response object (e.g. the job result with
// bpm/keyscale metadata). May be nil.
Raw any
}
// Option mutates a Request before it is sent. Options passed to Generate are
// applied to a copy of the request, so a Request value can be reused.
type Option func(*Request)
// WithLyrics sets song lyrics.
func WithLyrics(l string) Option { return func(r *Request) { r.Lyrics = l } }
// WithDuration sets the requested clip length in seconds.
func WithDuration(seconds int) Option {
return func(r *Request) { r.DurationSeconds = seconds }
}
// WithFormat sets the audio container format.
func WithFormat(f string) Option { return func(r *Request) { r.Format = f } }
// WithSteps overrides the number of inference steps.
func WithSteps(n int) Option { return func(r *Request) { r.Steps = &n } }
// WithCFGScale overrides the classifier-free-guidance scale.
func WithCFGScale(s float64) Option { return func(r *Request) { r.CFGScale = &s } }
// WithSeed fixes the RNG seed.
func WithSeed(seed int64) Option { return func(r *Request) { r.Seed = &seed } }
// Apply returns a copy of the request with all options applied. Providers
// call this once at the top of Generate.
func (r Request) Apply(opts ...Option) Request {
for _, opt := range opts {
opt(&r)
}
return r
}
// Model generates music from text.
type Model interface {
// Generate renders the request as one audio clip. It blocks until the
// backend finishes (or ctx expires) even when the backend is an async
// job queue — polling is the provider's business, not the caller's.
Generate(ctx context.Context, req Request, opts ...Option) (*Result, error)
}
// ModelOption configures a Model at construction time. Reserved for future
// per-model settings.
type ModelOption func(*ModelConfig)
// ModelConfig carries per-model construction settings.
type ModelConfig struct{}
// ApplyModelOptions folds options into a config.
func ApplyModelOptions(opts []ModelOption) ModelConfig {
var cfg ModelConfig
for _, opt := range opts {
opt(&cfg)
}
return cfg
}
// Provider mints music models bound to one backend.
type Provider interface {
// Name is the registry identifier for the provider.
Name() string
// MusicModel returns a Model bound to the given id (passed through to
// the backend verbatim; no catalog validation).
MusicModel(id string, opts ...ModelOption) (Model, error)
}
+114
View File
@@ -0,0 +1,114 @@
// Package ocr is majordomo's canonical document text-recognition surface.
// Like imagegen/audio/videogen/musicgen, it is a deliberately separate leaf
// contract from the llm package (ADR-0023, following the ADR-0016→0022
// lineage: functional options, zero values = backend default, bytes-only
// I/O, Raw escape hatch). OCR is not chat-vision: it targets dedicated
// detection+recognition models (Surya style) that return exact per-page,
// per-line text rather than a model's paraphrase.
//
// The first implementation is provider/llamaswap, which posts the document
// to a Surya shim through the /upstream passthrough; the shim rasterizes
// PDFs itself, so Document may be an image (png/jpg/webp) or a PDF.
package ocr
import "context"
// Request is a text-recognition request. Zero values mean "backend default".
type Request struct {
// Document is the encoded document to recognize: an image (png/jpg/webp)
// or a PDF. Required. Carried as bytes, never a URL.
Document []byte
// MIME is the document MIME type (e.g. "image/png", "application/pdf");
// "" = let the backend sniff it.
MIME string
// Filename is the multipart filename hint some backends key their format
// detection on; "" derives one from MIME ("document.pdf") or falls back
// to "document".
Filename string
// Languages are ISO-639 hints for the recognizer (e.g. "en", "de");
// nil = backend default (auto/multilingual).
Languages []string
// MaxPages caps how many pages of a multi-page document are recognized;
// 0 = backend default (all pages, up to the backend's own ceiling).
MaxPages int
}
// Page is the recognized text of one page.
type Page struct {
// Number is the 1-based page number.
Number int
// Text is the page's recognized text.
Text string
}
// Result is the canonical text-recognition result.
type Result struct {
// Text is the full recognized text, pages joined in order.
Text string
// Pages are the per-page results in page order.
Pages []Page
// Raw is the provider-native response object (e.g. the per-line
// bbox/confidence/layout detail), an escape hatch for provider-specific
// fields. May be nil; never required for normal use.
Raw any
}
// Option mutates a Request before it is sent. Options passed to Recognize are
// applied to a copy of the request, so a Request value can be reused.
type Option func(*Request)
// WithLanguages sets the recognizer language hints.
func WithLanguages(langs ...string) Option {
return func(r *Request) { r.Languages = langs }
}
// WithMaxPages caps how many pages are recognized.
func WithMaxPages(n int) Option { return func(r *Request) { r.MaxPages = n } }
// Apply returns a copy of the request with all options applied. Providers
// call this once at the top of Recognize.
func (r Request) Apply(opts ...Option) Request {
for _, opt := range opts {
opt(&r)
}
return r
}
// Model recognizes text in documents.
type Model interface {
// Recognize extracts the document's text, page by page.
Recognize(ctx context.Context, req Request, opts ...Option) (*Result, error)
}
// ModelOption configures a Model at construction time. Reserved for future
// per-model settings.
type ModelOption func(*ModelConfig)
// ModelConfig carries per-model construction settings.
type ModelConfig struct{}
// ApplyModelOptions folds options into a config.
func ApplyModelOptions(opts []ModelOption) ModelConfig {
var cfg ModelConfig
for _, opt := range opts {
opt(&cfg)
}
return cfg
}
// Provider mints OCR models bound to one backend.
type Provider interface {
// Name is the registry identifier for the provider.
Name() string
// OCRModel returns a Model bound to the given id (passed through to the
// backend verbatim; no catalog validation).
OCRModel(id string, opts ...ModelOption) (Model, error)
}
+3 -1
View File
@@ -213,7 +213,9 @@ func TestBuiltinsResolve(t *testing.T) {
r := newTestRegistry(t)
// All built-in provider names resolve even before their client
// implementations land (stub providers error only on use).
for _, name := range []string{"openai", "anthropic", "google", "ollama", "ollama-cloud", "foreman"} {
// Note: llama-swap is intentionally excluded — its no-URL built-in errors
// at Model() construction (not just on use), so it can't resolve here.
for _, name := range []string{"openai", "kimi", "anthropic", "google", "ollama", "ollama-cloud", "foreman"} {
if _, err := r.Parse(name + "/anything"); err != nil {
t.Errorf("Parse(%s/anything): %v", name, err)
}
+80
View File
@@ -1,5 +1,12 @@
# progress
## 2026-06-27 — llama-swaps (TLS) DSN scheme
Follow-up to the llama-swap provider: added the `llama-swaps` DSN scheme (https
base URL) alongside `llama-swap` (http, local-first), mirroring redis/rediss, so
a TLS-fronted instance is first-class instead of being pushed to the `openai://`
scheme. Scheme-only (no default built-in); shares one factory in builtin.go.
## 2026-06-27 — llama-swap provider + canonical image-gen interface
**Landed (ADR-0015, ADR-0016).** New `provider/llamaswap`: chat **delegates to
@@ -241,3 +248,76 @@ alias-in-chain failover, permanent-policy override) and wires anything the
tests flush out.
**Next:** Phase 2 — exhaustive health/chain test matrix.
## 2026-07-11 — audio surfaces + image editing (ADR-0017, ADR-0018)
- New leaf package `audio`: `SpeechModel`/`SpeechProvider` (TTS) and
`TranscriptionModel`/`TranscriptionProvider` (STT), imagegen conventions
(zero value = backend default, options + Apply, bytes in/out). Root
re-exports added.
- `imagegen.Editor` — optional img2img interface (`EditRequest` with
`Init Image` + `Strength`); shared `Result`.
- provider/llamaswap implements all of it: `/v1/audio/speech`,
`/v1/audio/transcriptions` (multipart), `ListVoices`
(`/v1/audio/voices?model=`, shape-tolerant), `/sdapi/v1/img2img`
(txt2img wire + init_images/denoising_strength, shared decode), and a
cheap `Health(ctx)` probe (GET /health) for often-offline hosts.
- Hermetic httptest coverage for every new wire shape + validation errors.
- Consumer: mort's llamaswap media tool cluster (status/image/TTS/STT tools).
## 2026-07-18 — Kimi (Moonshot AI) built-in provider (ADR-0026)
- New built-in `kimi` provider + `kimi://` DSN scheme: Moonshot's
OpenAI-compatible Chat Completions endpoint, so both reuse `provider/openai`
(no new client, mirrors llama-swap's chat path). Default base URL
`https://api.moonshot.ai/v1`; China endpoint via
`LLM_KCN=kimi://[email protected]/v1`.
- Credential is `KIMI_API_KEY` (read through the registry's injected envLookup,
so it's hermetically testable). `WithAPIKey` is passed unconditionally so an
unset `KIMI_API_KEY` can never fall through to the openai client's
`OPENAI_API_KEY` default.
- New `openai.WithAPIKeyName` option customizes the missing-key error hint
(default `OPENAI_API_KEY`); the kimi built-in/scheme name `KIMI_API_KEY`.
- Hermetic tests (capturing RoundTripper): built-in base URL + bearer, missing
key names KIMI_API_KEY with no OPENAI fallthrough and no network hit, and the
kimi:// scheme round-trips against the China host.
- Docs kept in sync: README built-in table + DSN scheme list + support matrix
(footnote ³), `.env.example`, ADR-0026 (+ index; also backfilled the missing
0024/0025 index rows).
- Consumer: mort names Kimi as a failover tier.
## 2026-08-12 — Qwen (Alibaba) built-in provider (ADR-0027)
- New built-in `qwen` provider + `qwen://` DSN scheme over Alibaba Model
Studio's OpenAI-compatible mode, reusing `provider/openai` (no new client,
mirrors kimi/llama-swap). Default base URL
`https://dashscope-intl.aliyuncs.com/compatible-mode/v1`; China/regional
hosts via `LLM_QCN=qwen://[email protected]/compatible-mode/v1`.
- Credential is `QWEN_API_KEY` (via the registry's injected envLookup).
`WithAPIKey` passed unconditionally so an unset key cannot fall through to
`OPENAI_API_KEY`; `WithAPIKeyName` names `QWEN_API_KEY` in the 401 hint.
- **Chose OpenAI-compat over Model Studio's Anthropic-compatible
`/apps/anthropic` shim** (ADR-0027): on the anthropic client
`ReasoningEffort` is ignored by design, `Request.Schema` rides
`output_config.format` (which the shim does not implement), and cached-token
accounting reads Anthropic-only usage fields. The shim is still reachable
ad hoc via an `anthropic://` DSN.
- `registerOpenAICompatBuiltin` installs BOTH halves of an OpenAI-compat
built-in (eager provider + `name://` DSN scheme via the shared
`openaiCompatScheme`), so the two credential rules — unconditional
`WithAPIKey`, and `WithAPIKeyName` naming that same variable — hold by
construction. kimi and qwen are one line each.
- `envKeyForProvider` is the single definition of the `LLM_<NAME>` form,
shared by lazy resolution (`registry.go`) and the DSN missing-key hint. They
were separate copies with a comment asserting they matched.
- The shared contract is ONE table (`builtin_openaicompat_test.go`), run
identically for every OpenAI-compat built-in: endpoint + bearer, missing key
fails closed naming its own variable with no network hit, the `name://` DSN
reaching another host, and a keyless DSN naming `LLM_<NAME>` rather than the
built-in's key. Adding a built-in is a table row that immediately owes all
four; `builtin_kimi_test.go` was retired into it. Qwen-only tests: the
reverse credential leak, and `reasoning_effort` asserted on the wire body
(the ADR's load-bearing claim).
- Docs in sync: README built-in table + Qwen paragraph + DSN scheme list +
support matrix (footnote ⁴), `.env.example`, ADR-0027 (+ index).
- Consumer: mort wants Qwen as a failover tier.
+378
View File
@@ -0,0 +1,378 @@
package llamaswap
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"mime"
"net/http"
"net/url"
"strconv"
"strings"
"gitea.stevedudenhoeffer.com/steve/majordomo/audio"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
// SpeechModel implements audio.SpeechProvider, binding a text-to-speech model
// served by llama-swap (routed to a kokoro/chatterbox-style OpenAI-compatible
// upstream). The id is passed through verbatim and selects which upstream
// llama-swap loads.
func (p *Provider) SpeechModel(id string, opts ...audio.SpeechModelOption) (audio.SpeechModel, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
_ = audio.ApplySpeechModelOptions(opts)
return &speechModel{p: p, id: id}, nil
}
type speechModel struct {
p *Provider
id string
}
// speechRequest is the OpenAI /v1/audio/speech request shape. llama-swap
// routes by the `model` field in the body.
type speechRequest struct {
Model string `json:"model"`
Input string `json:"input"`
Voice string `json:"voice,omitempty"`
ResponseFormat string `json:"response_format,omitempty"`
Speed float64 `json:"speed,omitempty"`
}
// Speak implements audio.SpeechModel via POST {base}/v1/audio/speech, or —
// when the request carries reference audio for voice cloning — via multipart
// POST {base}/upstream/<id>/v1/audio/speech/upload (the chatterbox clone
// route; verified live 2026-07-16: stateless per-request cloning, fields
// `input` + `voice_file`).
func (m *speechModel) Speak(ctx context.Context, req audio.SpeechRequest, opts ...audio.SpeechOption) (*audio.SpeechResult, error) {
req = req.Apply(opts...)
if strings.TrimSpace(req.Input) == "" {
return nil, fmt.Errorf("%w: speech synthesis requires input text", llm.ErrUnsupported)
}
if req.Speed < 0 {
return nil, fmt.Errorf("%w: speech speed must be >= 0, got %g", llm.ErrUnsupported, req.Speed)
}
if len(req.ReferenceAudio) > 0 {
return m.speakWithReference(ctx, req)
}
wire := speechRequest{
Model: m.id,
Input: req.Input,
Voice: req.Voice,
ResponseFormat: req.Format,
Speed: req.Speed,
}
body, err := json.Marshal(wire)
if err != nil {
return nil, fmt.Errorf("llama-swap: encode speech request: %w", err)
}
audioBytes, contentType, err := m.p.doRaw(ctx, http.MethodPost, "/v1/audio/speech", m.id, "application/json", bytes.NewReader(body), maxResponseBytes)
if err != nil {
return nil, err
}
if len(audioBytes) == 0 {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id, Message: "speech response contained no audio"}
}
return &audio.SpeechResult{Audio: audioBytes, MIME: speechMIME(contentType, req.Format)}, nil
}
// speakWithReference performs zero-shot voice cloning via the upstream
// passthrough clone route. The reference sample rides as the `voice_file`
// part and the text as an `input` field; voice/format/speed still go on the
// wire when set (upstreams ignore fields they don't understand). The result
// MIME comes from the response header or content sniffing (audioResultMIME,
// same validation as the sfx/enhance surfaces) — a JSON soft error or an
// HTML proxy page must never be wrapped up as audio bytes.
func (m *speechModel) speakWithReference(ctx context.Context, req audio.SpeechRequest) (*audio.SpeechResult, error) {
upPath, err := upstreamPath(m.id, "/v1/audio/speech/upload")
if err != nil {
return nil, err
}
speed := ""
if req.Speed != 0 {
speed = strconv.FormatFloat(req.Speed, 'g', -1, 64)
}
body, formType, err := buildMultipart("build speech clone form",
filePart{field: "voice_file", filename: transcriptionFilename("", req.ReferenceMIME), data: req.ReferenceAudio},
[]formField{
{"input", req.Input, true},
{"voice", req.Voice, false},
{"response_format", req.Format, false},
{"speed", speed, false},
})
if err != nil {
return nil, err
}
audioBytes, contentType, err := m.p.doRaw(ctx, http.MethodPost, upPath, m.id, formType, body, maxAudioResponseBytes)
if err != nil {
return nil, err
}
if len(audioBytes) == 0 {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id, Message: "speech clone response contained no audio"}
}
mimeType := audioResultMIME(contentType, audioBytes)
if mimeType == "" {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: fmt.Sprintf("speech clone response is not audio (Content-Type %q): %s", contentType, truncateForError(audioBytes))}
}
return &audio.SpeechResult{Audio: audioBytes, MIME: mimeType}, nil
}
// speechMIME resolves the result MIME type: the response Content-Type when it
// is a concrete audio type, else a mapping from the requested format, else
// audio/mpeg (the OpenAI endpoint's default container is mp3).
func speechMIME(contentType, format string) string {
if mt := mimeFromContentType(contentType, "audio/"); mt != "" {
return mt
}
switch strings.ToLower(strings.TrimSpace(format)) {
case "wav":
return "audio/wav"
case "opus":
return "audio/ogg"
case "aac":
return "audio/aac"
case "flac":
return "audio/flac"
default: // "", "mp3", and anything unrecognized
return "audio/mpeg"
}
}
// audioResultMIME resolves an audio body's MIME type: the response
// Content-Type when it is a concrete audio type, else content sniffing
// (RIFF/WAVE, Ogg and friends), else "" — the caller treats undetectable as
// an upstream error, mirroring videoMIME. Sniffed "audio/wave" is normalized
// to the conventional "audio/wav", and Ogg's container type
// "application/ogg" to "audio/ogg". Shared by the clone, sfx, and enhance
// surfaces, which all answer raw audio bodies.
func audioResultMIME(contentType string, data []byte) string {
if mt := mimeFromContentType(contentType, "audio/"); mt != "" {
return mt
}
switch mt := http.DetectContentType(data); {
case mt == "audio/wave":
return "audio/wav"
case mt == "application/ogg":
return "audio/ogg"
case strings.HasPrefix(mt, "audio/"):
return mt
}
return ""
}
// TranscriptionModel implements audio.TranscriptionProvider, binding a
// speech-to-text model served by llama-swap (routed to a whisper.cpp-style
// upstream).
func (p *Provider) TranscriptionModel(id string, opts ...audio.TranscriptionModelOption) (audio.TranscriptionModel, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
_ = audio.ApplyTranscriptionModelOptions(opts)
return &transcriptionModel{p: p, id: id}, nil
}
type transcriptionModel struct {
p *Provider
id string
}
// Transcribe implements audio.TranscriptionModel via POST
// {base}/v1/audio/transcriptions (multipart/form-data — llama-swap routes by
// the `model` form field).
func (m *transcriptionModel) Transcribe(ctx context.Context, req audio.TranscriptionRequest, opts ...audio.TranscriptionOption) (*audio.TranscriptionResult, error) {
req = req.Apply(opts...)
if len(req.Audio) == 0 {
return nil, fmt.Errorf("%w: transcription requires audio bytes", llm.ErrUnsupported)
}
// Translation is a per-request bool form field on whisper.cpp's server
// (server.cpp:534, verified 2026-07-16). Its language DEFAULT is "en",
// which would make translate a no-op — so an explicit language hint
// wins, but an unset one is forced to auto-detection.
translate := ""
language := req.Language
if req.Translate {
translate = "true"
if language == "" {
language = "auto"
}
}
buf, formType, err := buildMultipart("build transcription form",
filePart{field: "file", filename: transcriptionFilename(req.Filename, req.MIME), data: req.Audio},
[]formField{
{"model", m.id, true},
{"response_format", "json", true},
{"language", language, false},
{"prompt", req.Prompt, false},
{"translate", translate, false},
})
if err != nil {
return nil, err
}
raw, _, err := m.p.doRaw(ctx, http.MethodPost, "/v1/audio/transcriptions", m.id, formType, buf, maxResponseBytes)
if err != nil {
return nil, err
}
var out struct {
Text string `json:"text"`
}
if err := json.Unmarshal(raw, &out); err != nil {
return nil, fmt.Errorf("llama-swap: decode transcription response: %w", err)
}
return &audio.TranscriptionResult{Text: out.Text, Raw: json.RawMessage(raw)}, nil
}
// transcriptionFilename picks the multipart filename hint: the caller's
// (sanitized — upload metadata is untrusted and CR/LF would inject multipart
// headers), else one derived from the MIME subtype ("audio.mp3"), else
// "audio". MIME parameters ("audio/ogg; codecs=opus") are stripped before
// matching.
func transcriptionFilename(filename, mimeType string) string {
if name := sanitizeFilename(filename); name != "" {
return name
}
mt := strings.ToLower(strings.TrimSpace(mimeType))
if parsed, _, err := mime.ParseMediaType(mt); err == nil {
mt = parsed
}
switch mt {
case "audio/mpeg", "audio/mp3":
return "audio.mp3"
case "audio/wav", "audio/x-wav", "audio/wave":
return "audio.wav"
case "audio/ogg":
return "audio.ogg"
case "audio/opus":
return "audio.opus"
case "audio/flac", "audio/x-flac":
return "audio.flac"
case "audio/mp4", "audio/m4a", "audio/x-m4a":
return "audio.m4a"
case "audio/webm", "video/webm":
return "audio.webm"
default:
return "audio"
}
}
// sanitizeFilename strips characters that would corrupt or inject into the
// multipart Content-Disposition header, or smuggle directory structure to
// the receiving side. Quotes and backslashes are escaped by mime/multipart
// itself, but a file-writing shim decodes them right back — so CR/LF/NUL
// and both path separators are dropped: an upload-metadata filename must
// never traverse ("../x", "a/b", "C:\x").
func sanitizeFilename(name string) string {
name = strings.NewReplacer("\r", "", "\n", "", "\x00", "", "/", "", "\\", "").Replace(name)
return strings.TrimSpace(name)
}
// ListVoices returns the voices a TTS model offers (GET
// /v1/audio/voices?model=...). The decode is tolerant: upstreams answer with
// either a bare string list or a list of {id|name} objects.
func (p *Provider) ListVoices(ctx context.Context, model string) ([]string, error) {
if model == "" {
return nil, fmt.Errorf("llama-swap: ListVoices requires a model id")
}
raw, _, err := p.doRaw(ctx, http.MethodGet, "/v1/audio/voices?model="+url.QueryEscape(model), model, "", nil, maxResponseBytes)
if err != nil {
return nil, err
}
return parseVoices(raw)
}
// parseVoices extracts voice names from the various shapes upstreams use:
// {"voices":[...]} or {"data":[...]} envelopes (or a bare array), holding
// either strings or objects keyed by id/name/voice_id.
func parseVoices(raw []byte) ([]string, error) {
var env struct {
Voices json.RawMessage `json:"voices"`
Data json.RawMessage `json:"data"`
}
list := json.RawMessage(raw)
if err := json.Unmarshal(raw, &env); err == nil {
if len(env.Voices) > 0 {
list = env.Voices
} else if len(env.Data) > 0 {
list = env.Data
}
}
var names []string
if err := json.Unmarshal(list, &names); err == nil {
return names, nil
}
// Not a plain string list. The failed decode above may have partially
// populated names (Unmarshal appends zero values before erroring on an
// element type mismatch) — start fresh for the object shape.
names = nil
var objs []struct {
ID string `json:"id"`
Name string `json:"name"`
VoiceID string `json:"voice_id"`
}
if err := json.Unmarshal(list, &objs); err == nil {
for _, o := range objs {
switch {
case o.ID != "":
names = append(names, o.ID)
case o.Name != "":
names = append(names, o.Name)
case o.VoiceID != "":
names = append(names, o.VoiceID)
}
}
return names, nil
}
return nil, fmt.Errorf("llama-swap: unrecognized voices payload shape")
}
// doRaw performs a request to a llama-swap endpoint and returns the raw
// response body and its Content-Type — the sibling of doJSON for endpoints
// whose success payload is not JSON (audio/video bytes) or whose shape
// varies. contentType sets the request Content-Type when body is non-nil.
// A response larger than maxBytes is an error, never a silent truncation.
func (p *Provider) doRaw(ctx context.Context, method, path, model, contentType string, body io.Reader, maxBytes int64) ([]byte, string, error) {
data, hdr, err := p.doRawHeaders(ctx, method, path, model, contentType, body, maxBytes)
if err != nil {
return nil, "", err
}
return data, hdr.Get("Content-Type"), nil
}
// doRawHeaders is doRaw with the WHOLE response header rather than just
// Content-Type. Only the face swap needs it — the shim reports whether the
// likeness actually transferred in X-Swap-Report, and that answer would be
// thrown away by a function that keeps one header — so doRaw stays the
// signature 25 other call sites use and delegates here. Two bodies would be
// two places for the size cap and the status check to drift apart.
func (p *Provider) doRawHeaders(ctx context.Context, method, path, model, contentType string, body io.Reader, maxBytes int64) ([]byte, http.Header, error) {
if err := p.requireBaseURL(); err != nil {
return nil, nil, err
}
req, err := p.newRequest(ctx, method, path, contentType, body)
if err != nil {
return nil, nil, err
}
resp, err := p.client.Do(req)
if err != nil {
return nil, nil, fmt.Errorf("llama-swap: do request: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode/100 != 2 {
return nil, nil, p.apiError(resp, model)
}
data, err := io.ReadAll(io.LimitReader(resp.Body, maxBytes+1))
if err != nil {
return nil, nil, fmt.Errorf("llama-swap: read response: %w", err)
}
if int64(len(data)) > maxBytes {
return nil, nil, fmt.Errorf("llama-swap: response exceeds %d bytes", maxBytes)
}
return data, resp.Header, nil
}
+247
View File
@@ -0,0 +1,247 @@
package llamaswap
import (
"context"
"encoding/json"
"errors"
"io"
"net/http"
"net/http/httptest"
"reflect"
"testing"
"gitea.stevedudenhoeffer.com/steve/majordomo/audio"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
func TestSpeak(t *testing.T) {
var gotBody map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v1/audio/speech" {
t.Errorf("path = %q", r.URL.Path)
}
if r.Header.Get("Authorization") != "Bearer tok" {
t.Errorf("auth = %q", r.Header.Get("Authorization"))
}
_ = json.NewDecoder(r.Body).Decode(&gotBody)
w.Header().Set("Content-Type", "audio/mpeg")
_, _ = w.Write([]byte("MP3BYTES"))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithToken("tok"), WithHTTPClient(srv.Client()))
sm, err := p.SpeechModel("kokoro")
if err != nil {
t.Fatalf("SpeechModel: %v", err)
}
res, err := sm.Speak(context.Background(),
audio.SpeechRequest{Input: "hello world"},
audio.WithVoice("af_heart"), audio.WithFormat("mp3"), audio.WithSpeed(1.2),
)
if err != nil {
t.Fatalf("Speak: %v", err)
}
if string(res.Audio) != "MP3BYTES" || res.MIME != "audio/mpeg" {
t.Errorf("result = %q %q", res.Audio, res.MIME)
}
want := map[string]any{"model": "kokoro", "input": "hello world", "voice": "af_heart", "response_format": "mp3", "speed": 1.2}
for k, w := range want {
if gotBody[k] != w {
t.Errorf("%s = %v, want %v", k, gotBody[k], w)
}
}
}
func TestSpeakDefaultsOmitted(t *testing.T) {
var gotBody map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_ = json.NewDecoder(r.Body).Decode(&gotBody)
_, _ = w.Write([]byte("x"))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
sm, _ := p.SpeechModel("kokoro")
res, err := sm.Speak(context.Background(), audio.SpeechRequest{Input: "hi"})
if err != nil {
t.Fatalf("Speak: %v", err)
}
for _, k := range []string{"voice", "response_format", "speed"} {
if v, ok := gotBody[k]; ok {
t.Errorf("unset request sent %q = %v, want omitted", k, v)
}
}
// No usable Content-Type and no requested format → mp3 default.
if res.MIME != "audio/mpeg" {
t.Errorf("MIME = %q, want audio/mpeg fallback", res.MIME)
}
}
func TestSpeakEmptyInput(t *testing.T) {
p := New(WithBaseURL("http://example.invalid"))
sm, _ := p.SpeechModel("kokoro")
if _, err := sm.Speak(context.Background(), audio.SpeechRequest{Input: " "}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("err = %v, want ErrUnsupported", err)
}
}
func TestSpeechMIME(t *testing.T) {
cases := []struct {
contentType, format, want string
}{
{"audio/ogg; codecs=opus", "mp3", "audio/ogg"}, // concrete header wins
{"application/octet-stream", "wav", "audio/wav"},
{"", "", "audio/mpeg"},
{"", "opus", "audio/ogg"},
{"text/plain", "flac", "audio/flac"},
}
for _, tc := range cases {
if got := speechMIME(tc.contentType, tc.format); got != tc.want {
t.Errorf("speechMIME(%q, %q) = %q, want %q", tc.contentType, tc.format, got, tc.want)
}
}
}
func TestTranscribe(t *testing.T) {
var gotFields map[string]string
var gotFile []byte
var gotFilename string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v1/audio/transcriptions" {
t.Errorf("path = %q", r.URL.Path)
}
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Fatalf("parse multipart: %v", err)
}
gotFields = map[string]string{}
for k, v := range r.MultipartForm.Value {
gotFields[k] = v[0]
}
f, hdr, err := r.FormFile("file")
if err != nil {
t.Fatalf("form file: %v", err)
}
defer f.Close()
gotFile, _ = io.ReadAll(f)
gotFilename = hdr.Filename
_, _ = w.Write([]byte(`{"text":"hello there"}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
tm, err := p.TranscriptionModel("whisper")
if err != nil {
t.Fatalf("TranscriptionModel: %v", err)
}
res, err := tm.Transcribe(context.Background(),
audio.TranscriptionRequest{Audio: []byte("AUDIO"), MIME: "audio/mpeg"},
audio.WithLanguage("en"), audio.WithPrompt("robot names"),
)
if err != nil {
t.Fatalf("Transcribe: %v", err)
}
if res.Text != "hello there" {
t.Errorf("text = %q", res.Text)
}
if string(gotFile) != "AUDIO" || gotFilename != "audio.mp3" {
t.Errorf("file = %q name = %q", gotFile, gotFilename)
}
want := map[string]string{"model": "whisper", "language": "en", "prompt": "robot names", "response_format": "json"}
for k, w := range want {
if gotFields[k] != w {
t.Errorf("%s = %q, want %q", k, gotFields[k], w)
}
}
}
func TestTranscribeEmptyAudio(t *testing.T) {
p := New(WithBaseURL("http://example.invalid"))
tm, _ := p.TranscriptionModel("whisper")
if _, err := tm.Transcribe(context.Background(), audio.TranscriptionRequest{}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("err = %v, want ErrUnsupported", err)
}
}
func TestListVoices(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v1/audio/voices" {
t.Errorf("path = %q", r.URL.Path)
}
if got := r.URL.Query().Get("model"); got != "kokoro" {
t.Errorf("model = %q", got)
}
_, _ = w.Write([]byte(`{"voices":["af_heart","af_bella"]}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
voices, err := p.ListVoices(context.Background(), "kokoro")
if err != nil {
t.Fatalf("ListVoices: %v", err)
}
if !reflect.DeepEqual(voices, []string{"af_heart", "af_bella"}) {
t.Errorf("voices = %v", voices)
}
if _, err := p.ListVoices(context.Background(), ""); err == nil {
t.Error("empty model: want error")
}
}
func TestParseVoicesShapes(t *testing.T) {
cases := []struct {
name string
raw string
want []string
}{
{"envelope strings", `{"voices":["a","b"]}`, []string{"a", "b"}},
{"bare array", `["a","b"]`, []string{"a", "b"}},
{"objects by id", `{"voices":[{"id":"a"},{"id":"b"}]}`, []string{"a", "b"}},
{"objects by name", `{"data":[{"name":"a"},{"voice_id":"b"}]}`, []string{"a", "b"}},
}
for _, tc := range cases {
got, err := parseVoices([]byte(tc.raw))
if err != nil {
t.Errorf("%s: %v", tc.name, err)
continue
}
if !reflect.DeepEqual(got, tc.want) {
t.Errorf("%s: got %v, want %v", tc.name, got, tc.want)
}
}
if _, err := parseVoices([]byte(`"just a string"`)); err == nil {
t.Error("unparseable shape: want error")
}
}
func TestAudioAPIError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusServiceUnavailable)
_, _ = w.Write([]byte(`{"error":{"message":"model is loading"}}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
sm, _ := p.SpeechModel("kokoro")
_, err := sm.Speak(context.Background(), audio.SpeechRequest{Input: "x"})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %T %v, want *llm.APIError", err, err)
}
if apiErr.Status != http.StatusServiceUnavailable || apiErr.Message != "model is loading" || apiErr.Model != "kokoro" {
t.Errorf("apiErr = %+v", apiErr)
}
}
func TestAudioNoBaseURL(t *testing.T) {
p := New()
if _, err := p.SpeechModel("kokoro"); err == nil {
t.Error("SpeechModel: want error without base URL")
}
if _, err := p.TranscriptionModel("whisper"); err == nil {
t.Error("TranscriptionModel: want error without base URL")
}
if _, err := p.ListVoices(context.Background(), "kokoro"); err == nil {
t.Error("ListVoices: want error without base URL")
}
}
+215
View File
@@ -0,0 +1,215 @@
package llamaswap
import (
"context"
"errors"
"io"
"net/http"
"net/http/httptest"
"testing"
"gitea.stevedudenhoeffer.com/steve/majordomo/audio"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
func TestSpeakWithReferenceUsesCloneRoute(t *testing.T) {
var gotPath, gotInput, gotVoice, gotFilename string
var gotRef []byte
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Fatalf("parse multipart: %v", err)
}
gotInput = r.FormValue("input")
gotVoice = r.FormValue("voice")
f, hdr, err := r.FormFile("voice_file")
if err != nil {
t.Fatalf("voice_file part: %v", err)
}
defer f.Close()
gotRef, _ = io.ReadAll(f)
gotFilename = hdr.Filename
w.Header().Set("Content-Type", "audio/wav")
_, _ = w.Write(wavFixture())
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
sm, _ := p.SpeechModel("chatterbox")
res, err := sm.Speak(context.Background(),
audio.SpeechRequest{Input: "hello in my voice", Voice: "narrator"},
audio.WithReferenceAudio([]byte("REFWAV"), "audio/wav"))
if err != nil {
t.Fatalf("Speak: %v", err)
}
if gotPath != "/upstream/chatterbox/v1/audio/speech/upload" {
t.Errorf("path = %q, want clone route", gotPath)
}
if gotInput != "hello in my voice" || gotVoice != "narrator" {
t.Errorf("input/voice = %q/%q", gotInput, gotVoice)
}
if string(gotRef) != "REFWAV" || gotFilename != "audio.wav" {
t.Errorf("ref = %q name = %q", gotRef, gotFilename)
}
if res.MIME != "audio/wav" || len(res.Audio) == 0 {
t.Errorf("result = %q/%d bytes", res.MIME, len(res.Audio))
}
}
func TestSpeakWithReferenceSniffsHeaderlessWav(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Fatalf("parse multipart: %v", err)
}
for _, k := range []string{"voice", "response_format", "speed"} {
if v, ok := r.MultipartForm.Value[k]; ok {
t.Errorf("unset request sent %q = %v, want omitted", k, v)
}
}
w.Header()["Content-Type"] = nil // no declared type at all
_, _ = w.Write(wavFixture())
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
sm, _ := p.SpeechModel("chatterbox")
res, err := sm.Speak(context.Background(),
audio.SpeechRequest{Input: "hi"},
audio.WithReferenceAudio([]byte("REF"), ""))
if err != nil {
t.Fatalf("Speak: %v", err)
}
// Headerless RIFF sniffs audio/wave, normalized to the conventional wav.
if res.MIME != "audio/wav" {
t.Errorf("MIME = %q, want sniffed audio/wav", res.MIME)
}
}
func TestSpeakWithReferenceRejectsNonAudioResponse(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// A FastAPI-style 2xx soft error must not be wrapped up as audio.
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"detail":"reference audio too short"}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
sm, _ := p.SpeechModel("chatterbox")
_, err := sm.Speak(context.Background(),
audio.SpeechRequest{Input: "hi"},
audio.WithReferenceAudio([]byte("REF"), "audio/wav"))
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want APIError for non-audio clone body", err)
}
}
func TestSanitizeFilenameStripsPathAndControlBytes(t *testing.T) {
for in, want := range map[string]string{
"song.mp3": "song.mp3",
"../../etc/passwd": "....etcpasswd",
"a\r\nContent-Type: evil": "aContent-Type: evil",
"..\\..\\boot.ini": "....boot.ini",
"nul\x00byte.wav": "nulbyte.wav",
" / ": "",
} {
if got := sanitizeFilename(in); got != want {
t.Errorf("sanitizeFilename(%q) = %q, want %q", in, got, want)
}
}
}
func TestSpeakWithoutReferenceKeepsJSONRoute(t *testing.T) {
var gotPath, gotContentType string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
gotContentType = r.Header.Get("Content-Type")
_, _ = w.Write([]byte("MP3"))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
sm, _ := p.SpeechModel("chatterbox")
if _, err := sm.Speak(context.Background(), audio.SpeechRequest{Input: "hi"}); err != nil {
t.Fatalf("Speak: %v", err)
}
if gotPath != "/v1/audio/speech" || gotContentType != "application/json" {
t.Errorf("path/content-type = %q/%q, want JSON route", gotPath, gotContentType)
}
}
func TestTranscribeTranslateForcesAutoLanguage(t *testing.T) {
var gotTranslate, gotLanguage string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Fatalf("parse multipart: %v", err)
}
gotTranslate = r.FormValue("translate")
gotLanguage = r.FormValue("language")
_, _ = w.Write([]byte(`{"text":"hello"}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
tm, _ := p.TranscriptionModel("whisper-large-v3-turbo")
res, err := tm.Transcribe(context.Background(),
audio.TranscriptionRequest{Audio: []byte("AUDIO")},
audio.WithTranslate())
if err != nil {
t.Fatalf("Transcribe: %v", err)
}
// whisper.cpp's server default language is "en", which would skip
// translation — translate must force auto-detection when no explicit
// language hint was given.
if gotTranslate != "true" || gotLanguage != "auto" {
t.Errorf("translate/language = %q/%q, want true/auto", gotTranslate, gotLanguage)
}
if res.Text != "hello" {
t.Errorf("text = %q", res.Text)
}
}
func TestTranscribeTranslateKeepsExplicitLanguage(t *testing.T) {
var gotTranslate, gotLanguage string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Fatalf("parse multipart: %v", err)
}
gotTranslate = r.FormValue("translate")
gotLanguage = r.FormValue("language")
_, _ = w.Write([]byte(`{"text":"hello"}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
tm, _ := p.TranscriptionModel("whisper-large-v3-turbo")
if _, err := tm.Transcribe(context.Background(),
audio.TranscriptionRequest{Audio: []byte("AUDIO"), Language: "de", Translate: true}); err != nil {
t.Fatalf("Transcribe: %v", err)
}
if gotTranslate != "true" || gotLanguage != "de" {
t.Errorf("translate/language = %q/%q, want true/de", gotTranslate, gotLanguage)
}
}
func TestTranscribeWithoutTranslateOmitsFields(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Fatalf("parse multipart: %v", err)
}
for _, k := range []string{"translate", "language"} {
if v, ok := r.MultipartForm.Value[k]; ok {
t.Errorf("unset request sent %q = %v, want omitted", k, v)
}
}
_, _ = w.Write([]byte(`{"text":"hi"}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
tm, _ := p.TranscriptionModel("whisper-large-v3-turbo")
if _, err := tm.Transcribe(context.Background(),
audio.TranscriptionRequest{Audio: []byte("AUDIO")}); err != nil {
t.Fatalf("Transcribe: %v", err)
}
}
+113
View File
@@ -0,0 +1,113 @@
// diarize.go implements audio.DiarizationProvider against a
// whisper-asr-webservice upstream (WhisperX engine) reached through
// llama-swap's /upstream passthrough (ADR-0020):
//
// POST /upstream/<id>/asr?output=json&diarize=true[&language&min_speakers&max_speakers]
//
// output=json is load-bearing: the vtt/srt output formats DROP the speaker
// labels; only the json shape carries per-segment `speaker` fields.
package llamaswap
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/url"
"strconv"
"gitea.stevedudenhoeffer.com/steve/majordomo/audio"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
// DiarizationModel implements audio.DiarizationProvider. The id selects
// which upstream llama-swap loads (the pyannote-equipped WhisperX sidecar,
// not the plain whisper.cpp model).
func (p *Provider) DiarizationModel(id string, opts ...audio.DiarizationModelOption) (audio.DiarizationModel, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
_ = audio.ApplyDiarizationModelOptions(opts)
return &diarizationModel{p: p, id: id}, nil
}
type diarizationModel struct {
p *Provider
id string
}
// asrResponse is whisper-asr-webservice's output=json shape (the subset this
// package relies on; per-word entries are ignored — segment granularity is
// the contract).
type asrResponse struct {
Text string `json:"text"`
Language string `json:"language"`
Segments []struct {
Start float64 `json:"start"`
End float64 `json:"end"`
Text string `json:"text"`
Speaker string `json:"speaker"`
} `json:"segments"`
}
// Diarize implements audio.DiarizationModel.
func (m *diarizationModel) Diarize(ctx context.Context, req audio.DiarizationRequest, opts ...audio.DiarizationOption) (*audio.DiarizationResult, error) {
req = req.Apply(opts...)
if len(req.Audio) == 0 {
return nil, fmt.Errorf("%w: diarization requires audio bytes", llm.ErrUnsupported)
}
if req.MinSpeakers < 0 || req.MaxSpeakers < 0 ||
(req.MaxSpeakers > 0 && req.MinSpeakers > req.MaxSpeakers) {
return nil, fmt.Errorf("%w: invalid speaker bounds [%d,%d]", llm.ErrUnsupported, req.MinSpeakers, req.MaxSpeakers)
}
q := url.Values{}
q.Set("output", "json")
q.Set("diarize", "true")
if req.Language != "" {
q.Set("language", req.Language)
}
if req.MinSpeakers > 0 {
q.Set("min_speakers", strconv.Itoa(req.MinSpeakers))
}
if req.MaxSpeakers > 0 {
q.Set("max_speakers", strconv.Itoa(req.MaxSpeakers))
}
path, err := upstreamPath(m.id, "/asr?"+q.Encode())
if err != nil {
return nil, err
}
body, contentType, err := buildMultipart("build diarization form",
filePart{field: "audio_file", filename: transcriptionFilename(req.Filename, req.MIME), data: req.Audio},
nil)
if err != nil {
return nil, err
}
raw, _, err := m.p.doRaw(ctx, http.MethodPost, path, m.id, contentType, body, maxResponseBytes)
if err != nil {
return nil, err
}
var out asrResponse
if err := json.Unmarshal(raw, &out); err != nil {
return nil, fmt.Errorf("llama-swap: decode diarization response: %w", err)
}
res := &audio.DiarizationResult{
Text: out.Text,
Language: out.Language,
Raw: json.RawMessage(raw),
}
for _, s := range out.Segments {
res.Segments = append(res.Segments, audio.DiarizationSegment{
Start: s.Start,
End: s.End,
Speaker: s.Speaker,
Text: s.Text,
})
}
if len(res.Segments) == 0 && res.Text == "" {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id, Message: "diarization response contained no transcript"}
}
return res, nil
}
+227
View File
@@ -0,0 +1,227 @@
package llamaswap
import (
"context"
"encoding/base64"
"encoding/json"
"errors"
"net/http"
"net/http/httptest"
"testing"
"gitea.stevedudenhoeffer.com/steve/majordomo/imagegen"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
func editInit(t *testing.T) imagegen.Image {
t.Helper()
raw, err := base64.StdEncoding.DecodeString(onePixelPNG)
if err != nil {
t.Fatalf("decode fixture: %v", err)
}
return imagegen.Image{MIME: "image/png", Data: raw}
}
func TestImageEdit(t *testing.T) {
var gotBody map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/sdapi/v1/img2img" {
t.Errorf("path = %q", r.URL.Path)
}
_ = json.NewDecoder(r.Body).Decode(&gotBody)
_, _ = w.Write([]byte(`{"images":["` + onePixelPNG + `"]}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
im, _ := p.ImageModel("sd")
ed, ok := im.(imagegen.Editor)
if !ok {
t.Fatal("imageModel does not implement imagegen.Editor")
}
res, err := ed.Edit(context.Background(),
imagegen.EditRequest{Prompt: "make it night", Init: editInit(t)},
imagegen.WithEditStrength(0.6),
)
if err != nil {
t.Fatalf("Edit: %v", err)
}
if len(res.Images) != 1 || res.Images[0].MIME != "image/png" {
t.Fatalf("images = %+v", res.Images)
}
if gotBody["model"] != "sd" || gotBody["prompt"] != "make it night" {
t.Errorf("model/prompt = %v/%v", gotBody["model"], gotBody["prompt"])
}
inits, ok := gotBody["init_images"].([]any)
if !ok || len(inits) != 1 || inits[0] != onePixelPNG {
t.Errorf("init_images = %v, want the b64 fixture", gotBody["init_images"])
}
if gotBody["denoising_strength"] != 0.6 {
t.Errorf("denoising_strength = %v, want 0.6", gotBody["denoising_strength"])
}
}
func TestImageEditOmitsUnsetOverrides(t *testing.T) {
var gotBody map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_ = json.NewDecoder(r.Body).Decode(&gotBody)
_, _ = w.Write([]byte(`{"images":["` + onePixelPNG + `"]}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
im, _ := p.ImageModel("sd")
ed := im.(imagegen.Editor)
if _, err := ed.Edit(context.Background(), imagegen.EditRequest{Prompt: "x", Init: editInit(t)}); err != nil {
t.Fatalf("Edit: %v", err)
}
for _, k := range []string{"denoising_strength", "steps", "cfg_scale", "negative_prompt", "sample_method", "seed", "width", "height"} {
if v, ok := gotBody[k]; ok {
t.Errorf("unset request sent %q = %v, want omitted", k, v)
}
}
}
func TestImageEditValidation(t *testing.T) {
p := New(WithBaseURL("http://example.invalid"))
im, _ := p.ImageModel("sd")
ed := im.(imagegen.Editor)
cases := []struct {
name string
req imagegen.EditRequest
}{
{"empty prompt", imagegen.EditRequest{Prompt: " ", Init: imagegen.Image{Data: []byte{1}}}},
{"missing init", imagegen.EditRequest{Prompt: "x"}},
{"negative N", imagegen.EditRequest{Prompt: "x", Init: imagegen.Image{Data: []byte{1}}, N: -1}},
}
for _, tc := range cases {
if _, err := ed.Edit(context.Background(), tc.req); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("%s: err = %v, want ErrUnsupported", tc.name, err)
}
}
bad := 1.5
if _, err := ed.Edit(context.Background(), imagegen.EditRequest{Prompt: "x", Init: imagegen.Image{Data: []byte{1}}, Strength: &bad}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("out-of-range strength: err = %v, want ErrUnsupported", err)
}
}
func TestImageEditWithMask(t *testing.T) {
var gotBody map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_ = json.NewDecoder(r.Body).Decode(&gotBody)
_, _ = w.Write([]byte(`{"images":["` + onePixelPNG + `"]}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
im, _ := p.ImageModel("sd")
ed := im.(imagegen.Editor)
mask := imagegen.Image{MIME: "image/png", Data: []byte{0xDE, 0xAD}}
if _, err := ed.Edit(context.Background(),
imagegen.EditRequest{Prompt: "replace the sky", Init: editInit(t)},
imagegen.WithEditMask(mask),
); err != nil {
t.Fatalf("Edit: %v", err)
}
if got := gotBody["mask"]; got != base64.StdEncoding.EncodeToString(mask.Data) {
t.Errorf("mask = %v, want the b64 mask", got)
}
}
func TestImageEditWithoutMaskOmitsField(t *testing.T) {
var gotBody map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_ = json.NewDecoder(r.Body).Decode(&gotBody)
_, _ = w.Write([]byte(`{"images":["` + onePixelPNG + `"]}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
im, _ := p.ImageModel("sd")
ed := im.(imagegen.Editor)
if _, err := ed.Edit(context.Background(),
imagegen.EditRequest{Prompt: "p", Init: editInit(t)}); err != nil {
t.Fatalf("Edit: %v", err)
}
if _, ok := gotBody["mask"]; ok {
t.Error("mask field sent for unmasked edit; want omitted")
}
}
// TestImageEditByReferenceUsesTxt2ImgExtraImages pins the instruction-edit
// wire shape. It is a DIFFERENT endpoint and a DIFFERENT field from img2img,
// and the difference is not cosmetic: measured against FLUX.1-Kontext on
// 2026-07-30, the same prompt sent as init_images left the thing it was told
// to change untouched and drifted everything else, while extra_images changed
// exactly what was asked and left the rest of the frame numerically
// unchanged. Routing a reference edit down the img2img path would look like
// a working call and silently produce the wrong picture.
func TestImageEditByReferenceUsesTxt2ImgExtraImages(t *testing.T) {
var gotPath string
var gotBody map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
_ = json.NewDecoder(r.Body).Decode(&gotBody)
_, _ = w.Write([]byte(`{"images":["` + onePixelPNG + `"]}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
im, _ := p.ImageModel("imagegen-flux-kontext")
ed := im.(imagegen.Editor)
ref := editInit(t)
if _, err := ed.Edit(context.Background(), imagegen.EditRequest{
Prompt: "make the sign read OPEN",
// Init/Mask/Strength are set and must be IGNORED — they describe a
// pipeline this model does not run.
Init: ref,
Mask: ref,
Strength: func() *float64 { s := 0.75; return &s }(),
}, imagegen.WithEditRefImages(ref)); err != nil {
t.Fatalf("reference edit: %v", err)
}
if gotPath != "/sdapi/v1/txt2img" {
t.Errorf("path = %q, want /sdapi/v1/txt2img (there is no init latent to denoise)", gotPath)
}
extra, ok := gotBody["extra_images"].([]any)
if !ok || len(extra) != 1 {
t.Fatalf("extra_images = %v, want the one reference image", gotBody["extra_images"])
}
if _, present := gotBody["init_images"]; present {
t.Error("init_images must NOT be sent on the reference path — it re-noises the picture")
}
if _, present := gotBody["denoising_strength"]; present {
t.Error("denoising_strength must NOT be sent on the reference path")
}
if _, present := gotBody["mask"]; present {
t.Error("mask must NOT be sent on the reference path")
}
}
// TestImageEditByReferenceRejectsEmptyRefs guards the case that would
// otherwise silently become a plain txt2img: a reference edit whose only
// reference carries no bytes has nothing to condition on, and rendering the
// prompt from scratch is not what the caller asked for.
func TestImageEditByReferenceRejectsEmptyRefs(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte(`{"images":["` + onePixelPNG + `"]}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
im, _ := p.ImageModel("imagegen-flux-kontext")
ed := im.(imagegen.Editor)
_, err := ed.Edit(context.Background(), imagegen.EditRequest{Prompt: "anything"},
imagegen.WithEditRefImages(imagegen.Image{MIME: "image/png"}))
if !errors.Is(err, llm.ErrUnsupported) {
t.Fatalf("err = %v, want ErrUnsupported for an all-empty reference set", err)
}
}
+150
View File
@@ -0,0 +1,150 @@
// embed.go implements embeddings.EmbedProvider and embeddings.RerankProvider
// against llama-server instances behind llama-swap (ADR-0022):
//
// POST /v1/embeddings {model, input: [...]} (OpenAI shape)
// POST /v1/rerank {model, query, documents, top_n} (Jina-ish shape)
//
// Both paths are in llama-swap's normal model-routed tables — no /upstream
// needed. The two surfaces are minted separately because they are two
// DIFFERENT server instances on the host: llama-server with --embeddings
// and --rerank enabled together returns all-zero embeddings (llama.cpp
// #20085), so the host runs one of each and the ids differ.
package llamaswap
import (
"context"
"fmt"
"net/http"
"sort"
"strings"
"gitea.stevedudenhoeffer.com/steve/majordomo/embeddings"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
// EmbedModel implements embeddings.EmbedProvider. The id selects which
// upstream llama-swap loads (a persistent CPU member on the reference host,
// so calls are cheap and never evict GPU models).
func (p *Provider) EmbedModel(id string, opts ...embeddings.EmbedModelOption) (embeddings.EmbedModel, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
_ = embeddings.ApplyEmbedModelOptions(opts)
return &embedModel{p: p, id: id}, nil
}
type embedModel struct {
p *Provider
id string
}
// Embed implements embeddings.EmbedModel via POST {base}/v1/embeddings.
func (m *embedModel) Embed(ctx context.Context, req embeddings.EmbedRequest, opts ...embeddings.EmbedOption) (*embeddings.EmbedResult, error) {
req = req.Apply(opts...)
if len(req.Inputs) == 0 {
return nil, fmt.Errorf("%w: embedding requires at least one input", llm.ErrUnsupported)
}
for i, in := range req.Inputs {
if strings.TrimSpace(in) == "" {
return nil, fmt.Errorf("%w: embedding input %d is empty", llm.ErrUnsupported, i)
}
}
wire := struct {
Model string `json:"model"`
Input []string `json:"input"`
}{Model: m.id, Input: req.Inputs}
var resp struct {
Data []struct {
Index int `json:"index"`
Embedding []float32 `json:"embedding"`
} `json:"data"`
}
if err := m.p.doJSON(ctx, http.MethodPost, "/v1/embeddings", m.id, &wire, &resp); err != nil {
return nil, err
}
if len(resp.Data) != len(req.Inputs) {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: fmt.Sprintf("embeddings response has %d vectors for %d inputs", len(resp.Data), len(req.Inputs))}
}
// The OpenAI shape carries an index per entry; order by it rather than
// trusting response order.
vectors := make([][]float32, len(req.Inputs))
for _, d := range resp.Data {
if d.Index < 0 || d.Index >= len(vectors) || len(d.Embedding) == 0 {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: fmt.Sprintf("embeddings response entry index %d invalid or empty", d.Index)}
}
if vectors[d.Index] != nil {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: fmt.Sprintf("embeddings response repeats index %d", d.Index)}
}
vectors[d.Index] = d.Embedding
}
for i, v := range vectors {
if v == nil {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: fmt.Sprintf("embeddings response missing vector for input %d", i)}
}
}
return &embeddings.EmbedResult{Vectors: vectors, Raw: &resp}, nil
}
// RerankModel implements embeddings.RerankProvider.
func (p *Provider) RerankModel(id string, opts ...embeddings.RerankModelOption) (embeddings.RerankModel, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
_ = embeddings.ApplyRerankModelOptions(opts)
return &rerankModel{p: p, id: id}, nil
}
type rerankModel struct {
p *Provider
id string
}
// Rerank implements embeddings.RerankModel via POST {base}/v1/rerank. The
// response parser reads only results[].index and results[].relevance_score —
// llama-server documents the shape as "might change", so stay minimal.
func (m *rerankModel) Rerank(ctx context.Context, req embeddings.RerankRequest, opts ...embeddings.RerankOption) (*embeddings.RerankResult, error) {
req = req.Apply(opts...)
if strings.TrimSpace(req.Query) == "" {
return nil, fmt.Errorf("%w: rerank requires a query", llm.ErrUnsupported)
}
if len(req.Documents) == 0 {
return nil, fmt.Errorf("%w: rerank requires at least one document", llm.ErrUnsupported)
}
if req.TopN < 0 {
return nil, fmt.Errorf("%w: rerank top_n must be >= 0, got %d", llm.ErrUnsupported, req.TopN)
}
wire := struct {
Model string `json:"model"`
Query string `json:"query"`
Documents []string `json:"documents"`
TopN int `json:"top_n,omitempty"`
}{Model: m.id, Query: req.Query, Documents: req.Documents, TopN: req.TopN}
var resp struct {
Results []struct {
Index int `json:"index"`
RelevanceScore float64 `json:"relevance_score"`
} `json:"results"`
}
if err := m.p.doJSON(ctx, http.MethodPost, "/v1/rerank", m.id, &wire, &resp); err != nil {
return nil, err
}
if len(resp.Results) == 0 {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id, Message: "rerank response contained no results"}
}
out := &embeddings.RerankResult{Raw: &resp}
for _, r := range resp.Results {
if r.Index < 0 || r.Index >= len(req.Documents) {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: fmt.Sprintf("rerank result index %d out of range", r.Index)}
}
out.Results = append(out.Results, embeddings.RerankItem{Index: r.Index, Score: r.RelevanceScore})
}
sort.SliceStable(out.Results, func(i, j int) bool { return out.Results[i].Score > out.Results[j].Score })
return out, nil
}
+61
View File
@@ -0,0 +1,61 @@
// enhance.go implements audio.SpeechEnhancementProvider against a
// DeepFilterNet shim (audioutils) reached through llama-swap's /upstream
// passthrough (ADR-0024):
//
// POST /upstream/<id>/v1/enhance multipart file -> WAV
package llamaswap
import (
"context"
"fmt"
"net/http"
"gitea.stevedudenhoeffer.com/steve/majordomo/audio"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
// SpeechEnhancerModel implements audio.SpeechEnhancementProvider. The id
// selects which upstream llama-swap loads.
func (p *Provider) SpeechEnhancerModel(id string, opts ...audio.SpeechEnhancerModelOption) (audio.SpeechEnhancer, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
_ = audio.ApplySpeechEnhancerModelOptions(opts)
return &speechEnhancerModel{p: p, id: id}, nil
}
type speechEnhancerModel struct {
p *Provider
id string
}
// Enhance implements audio.SpeechEnhancer. The endpoint always answers WAV.
func (m *speechEnhancerModel) Enhance(ctx context.Context, req audio.EnhancementRequest, opts ...audio.EnhancementOption) (*audio.SpeechResult, error) {
req = req.Apply(opts...)
if len(req.Audio) == 0 {
return nil, fmt.Errorf("%w: speech enhancement requires audio bytes", llm.ErrUnsupported)
}
path, err := upstreamPath(m.id, "/v1/enhance")
if err != nil {
return nil, err
}
body, contentType, err := buildMultipart("build enhance form",
filePart{field: "file", filename: transcriptionFilename(req.Filename, req.MIME), data: req.Audio},
nil)
if err != nil {
return nil, err
}
raw, respType, err := m.p.doRaw(ctx, http.MethodPost, path, m.id, contentType, body, maxAudioResponseBytes)
if err != nil {
return nil, err
}
if len(raw) == 0 {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id, Message: "enhance response contained no audio"}
}
mimeType := audioResultMIME(respType, raw)
if mimeType == "" {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: fmt.Sprintf("enhance response is not audio (Content-Type %q): %s", respType, truncateForError(raw))}
}
return &audio.SpeechResult{Audio: raw, MIME: mimeType}, nil
}
+238
View File
@@ -0,0 +1,238 @@
// faceswap.go implements imagegen.FaceSwapper against the InsightFace shim
// (buffalo_l + inswapper_128) reached through llama-swap's /upstream
// passthrough (ADR-0024):
//
// POST /upstream/<id>/v1/faces multipart file -> JSON
// POST /upstream/<id>/v1/faceswap multipart target, source -> PNG
package llamaswap
import (
"context"
"encoding/json"
"fmt"
"mime"
"net/http"
"strconv"
"strings"
"gitea.stevedudenhoeffer.com/steve/majordomo/imagegen"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
// maxFaceSwapResponseBytes bounds the returned PNG. Generous: the shim echoes
// the target's dimensions, and a 4K photo round-trips as a large lossless PNG.
const maxFaceSwapResponseBytes = 64 << 20
// FaceSwapModel implements the face-transfer surface. The id selects which
// upstream llama-swap loads.
func (p *Provider) FaceSwapModel(id string) (imagegen.FaceSwapper, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
return &faceSwapModel{p: p, id: id}, nil
}
type faceSwapModel struct {
p *Provider
id string
}
// facesResponse mirrors the shim's /v1/faces body.
type facesResponse struct {
Count int `json:"count"`
Faces []struct {
Index int `json:"index"`
Box []int `json:"box"`
Score float64 `json:"score"`
Width int `json:"width"`
Height int `json:"height"`
Yaw *float64 `json:"yaw"`
} `json:"faces"`
}
// ListFaces implements imagegen.FaceSwapper.
func (m *faceSwapModel) ListFaces(ctx context.Context, img imagegen.Image) ([]imagegen.DetectedFace, error) {
if len(img.Data) == 0 {
return nil, fmt.Errorf("%w: face detection requires image bytes", llm.ErrUnsupported)
}
path, err := upstreamPath(m.id, "/v1/faces")
if err != nil {
return nil, err
}
body, contentType, err := buildMultipart("build faces form",
filePart{field: "file", filename: imageFilename(img.MIME, "image"), data: img.Data}, nil)
if err != nil {
return nil, err
}
raw, _, err := m.p.doRaw(ctx, http.MethodPost, path, m.id, contentType, body, maxFaceSwapResponseBytes)
if err != nil {
return nil, err
}
var parsed facesResponse
if err := json.Unmarshal(raw, &parsed); err != nil {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: fmt.Sprintf("faces response is not JSON: %s", truncateForError(raw))}
}
out := make([]imagegen.DetectedFace, 0, len(parsed.Faces))
for _, f := range parsed.Faces {
df := imagegen.DetectedFace{Index: f.Index, Score: f.Score, Yaw: f.Yaw}
// A short box would silently index out of range below; treat a
// malformed entry as a protocol error rather than zero-filling it,
// because a wrong box sends the caller at the wrong face.
if len(f.Box) != 4 {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: fmt.Sprintf("face %d has a %d-element box, want 4", f.Index, len(f.Box))}
}
copy(df.Box[:], f.Box)
out = append(out, df)
}
return out, nil
}
// FaceSwap implements imagegen.FaceSwapper. The endpoint always answers PNG.
func (m *faceSwapModel) FaceSwap(ctx context.Context, req imagegen.FaceSwapRequest, opts ...imagegen.FaceSwapOption) (*imagegen.Result, error) {
req = req.Apply(opts...)
if len(req.Target.Data) == 0 {
return nil, fmt.Errorf("%w: face swap requires a target image", llm.ErrUnsupported)
}
if len(req.Source.Data) == 0 {
return nil, fmt.Errorf("%w: face swap requires a source image", llm.ErrUnsupported)
}
// Only when it will actually be sent: under All the index is documented
// as ignored, so rejecting a negative one there would fail a request that
// is perfectly well formed.
if !req.All && req.Index != nil && *req.Index < 0 {
return nil, fmt.Errorf("%w: face index must be >= 0, got %d", llm.ErrUnsupported, *req.Index)
}
path, err := upstreamPath(m.id, "/v1/faceswap")
if err != nil {
return nil, err
}
var fields []formField
if req.All {
fields = append(fields, formField{key: "all", value: "true", required: true})
} else if req.Index != nil {
// Only sent when NOT swapping all: the shim ignores index under
// all=true, and sending both would imply a precedence the caller
// cannot see.
fields = append(fields, formField{key: "index", value: strconv.Itoa(*req.Index), required: true})
}
body, contentType, err := buildMultipartFiles("build faceswap form",
[]filePart{
{field: "target", filename: imageFilename(req.Target.MIME, "target"), data: req.Target.Data},
{field: "source", filename: imageFilename(req.Source.MIME, "source"), data: req.Source.Data},
}, fields)
if err != nil {
return nil, err
}
raw, respHdr, err := m.p.doRawHeaders(ctx, http.MethodPost, path, m.id, contentType, body, maxFaceSwapResponseBytes)
if err != nil {
return nil, err
}
respType := respHdr.Get("Content-Type")
if len(raw) == 0 {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id, Message: "face swap response contained no image"}
}
// Validate the BYTES, not the header. sniffImageMIME falls back to
// image/png when detection is inconclusive, so trusting it here would
// label a JSON error body as a PNG and return it as a successful image —
// and a header check alone misses the case where the response carries no
// Content-Type at all. The shim answers JSON on a semantic miss (no face
// found), which is exactly the body that would sail through.
detected := http.DetectContentType(raw)
if !strings.HasPrefix(detected, "image/") {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: fmt.Sprintf("face swap response is not an image (sniffed %q, Content-Type %q): %s",
detected, respType, truncateForError(raw))}
}
// Prefer the server's own label when it is an image type (it knows
// subtypes the sniffer does not), else what the bytes actually are.
mimeType := detected
if hdr := mimeFromContentType(respType, "image/"); hdr != "" {
mimeType = hdr
}
return &imagegen.Result{
Images: []llm.ImagePart{{MIME: mimeType, Data: raw}},
SwappedFaces: parseSwapReport(respHdr.Get("X-Swap-Report")),
}, nil
}
// swapReport mirrors the shim's X-Swap-Report header.
type swapReport struct {
Image []int `json:"image"`
Faces []struct {
Index int `json:"index"`
Size []int `json:"size"`
Yaw *float64 `json:"yaw"`
IdentitySimilarity *float64 `json:"identity_similarity"`
} `json:"faces"`
}
// parseSwapReport decodes the measured outcome. A missing or malformed header
// yields nil rather than an error: an older shim does not send it, and a swap
// that produced a good image must not fail because the diagnostics alongside
// it were unreadable.
func parseSwapReport(header string) []imagegen.SwappedFace {
header = strings.TrimSpace(header)
if header == "" {
return nil
}
var rep swapReport
if err := json.Unmarshal([]byte(header), &rep); err != nil {
return nil
}
out := make([]imagegen.SwappedFace, 0, len(rep.Faces))
for _, f := range rep.Faces {
sf := imagegen.SwappedFace{
Index: f.Index,
Yaw: f.Yaw,
IdentitySimilarity: f.IdentitySimilarity,
}
if len(f.Size) == 2 {
sf.Width, sf.Height = f.Size[0], f.Size[1]
}
if len(rep.Image) == 2 {
sf.ImageWidth, sf.ImageHeight = rep.Image[0], rep.Image[1]
}
out = append(out, sf)
}
if len(out) == 0 {
return nil
}
return out
}
// imageFilename picks a multipart filename for an image part. The shim reads
// bytes, not names, but a plausible extension keeps server-side sniffing and
// request logs honest. base distinguishes the parts of a multi-file form
// ("target"/"source", "frame"/"frame_last") so a log line says which one was
// malformed — and, for the video keyframes, so a backend that stages uploads
// by filename cannot have the second overwrite the first.
//
// Every caller routes through here: two copies of one extension table is how
// they drift.
func imageFilename(mimeType, base string) string {
if base == "" {
base = "image"
}
mt := strings.ToLower(strings.TrimSpace(mimeType))
if parsed, _, err := mime.ParseMediaType(mt); err == nil {
mt = parsed
}
switch mt {
case "image/jpeg", "image/jpg":
return base + ".jpg"
case "image/webp":
return base + ".webp"
case "image/gif":
return base + ".gif"
case "image/bmp":
return base + ".bmp"
default:
// PNG is the safe default: every caller in this repo either sends PNG
// or sends something the decoder identifies by magic bytes anyway.
return base + ".png"
}
}
+344
View File
@@ -0,0 +1,344 @@
package llamaswap
import (
"context"
"encoding/base64"
"errors"
"io"
"mime"
"mime/multipart"
"net/http"
"net/http/httptest"
"strings"
"testing"
"gitea.stevedudenhoeffer.com/steve/majordomo/imagegen"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
// parseParts pulls the multipart form a handler received.
func parseParts(t *testing.T, r *http.Request) (files map[string][]byte, fields map[string]string) {
t.Helper()
_, params, err := mime.ParseMediaType(r.Header.Get("Content-Type"))
if err != nil {
t.Fatalf("content type: %v", err)
}
mr := multipart.NewReader(r.Body, params["boundary"])
files, fields = map[string][]byte{}, map[string]string{}
for {
p, err := mr.NextPart()
if err == io.EOF {
break
}
if err != nil {
t.Fatalf("next part: %v", err)
}
body, _ := io.ReadAll(p)
if p.FileName() != "" {
files[p.FormName()] = body
} else {
fields[p.FormName()] = string(body)
}
}
return files, fields
}
// TestFaceSwapSendsBothFiles pins the two-file wire shape. A face swap is the
// first endpoint in this provider taking more than one file, so buildMultipart
// grew a sibling; getting the field NAMES wrong would reach the shim as a
// missing-argument 422 rather than anything self-explanatory.
func TestFaceSwapSendsBothFiles(t *testing.T) {
var gotPath string
var files map[string][]byte
var fields map[string]string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
files, fields = parseParts(t, r)
w.Header().Set("Content-Type", "image/png")
raw, _ := base64.StdEncoding.DecodeString(onePixelPNG)
_, _ = w.Write(raw)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
m, err := p.FaceSwapModel("faceswap")
if err != nil {
t.Fatalf("model: %v", err)
}
img := editInit(t)
res, err := m.FaceSwap(context.Background(), imagegen.FaceSwapRequest{Target: img, Source: img},
imagegen.WithFaceIndex(2))
if err != nil {
t.Fatalf("faceswap: %v", err)
}
if len(res.Images) != 1 {
t.Fatalf("images = %d, want 1", len(res.Images))
}
if !strings.HasSuffix(gotPath, "/upstream/faceswap/v1/faceswap") {
t.Errorf("path = %q", gotPath)
}
for _, want := range []string{"target", "source"} {
if len(files[want]) == 0 {
t.Errorf("no %q file part — the shim requires both", want)
}
}
if fields["index"] != "2" {
t.Errorf("index = %q, want 2", fields["index"])
}
if _, ok := fields["all"]; ok {
t.Error("all sent alongside index — the shim ignores index under all=true, so sending both implies a precedence the caller cannot see")
}
}
// TestFaceSwapAllSuppressesIndex: same reasoning from the other side.
func TestFaceSwapAllSuppressesIndex(t *testing.T) {
var fields map[string]string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, fields = parseParts(t, r)
w.Header().Set("Content-Type", "image/png")
raw, _ := base64.StdEncoding.DecodeString(onePixelPNG)
_, _ = w.Write(raw)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
m, _ := p.FaceSwapModel("faceswap")
img := editInit(t)
if _, err := m.FaceSwap(context.Background(),
imagegen.FaceSwapRequest{Target: img, Source: img, Index: new(int), All: true}); err != nil {
t.Fatalf("faceswap: %v", err)
}
if fields["all"] != "true" {
t.Errorf("all = %q, want true", fields["all"])
}
if _, ok := fields["index"]; ok {
t.Error("index sent under all=true")
}
}
// TestFaceSwapRejectsMissingImages: both are required, and the error should
// name which one rather than surfacing a shim 422.
func TestFaceSwapRejectsMissingImages(t *testing.T) {
p := New(WithBaseURL("http://example.invalid"))
m, _ := p.FaceSwapModel("faceswap")
img := editInit(t)
for _, tc := range []struct {
name string
req imagegen.FaceSwapRequest
want string
}{
{"no target", imagegen.FaceSwapRequest{Source: img}, "target"},
{"no source", imagegen.FaceSwapRequest{Target: img}, "source"},
} {
_, err := m.FaceSwap(context.Background(), tc.req)
if err == nil || !strings.Contains(err.Error(), tc.want) {
t.Errorf("%s: err = %v, want one naming %q", tc.name, err, tc.want)
}
}
}
// TestFaceSwapRejectsNonImageResponse: the shim answers JSON on a semantic
// miss (no face found). Returning those bytes as an "image" would hand the
// caller a file that is not a picture and call it success.
func TestFaceSwapRejectsNonImageResponse(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"detail":{"error":"no_face_in_source"}}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
m, _ := p.FaceSwapModel("faceswap")
img := editInit(t)
_, err := m.FaceSwap(context.Background(), imagegen.FaceSwapRequest{Target: img, Source: img})
if err == nil {
t.Fatal("a JSON body was accepted as an image")
}
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Errorf("err = %T, want *llm.APIError so callers can classify it", err)
}
if !strings.Contains(err.Error(), "no_face_in_source") {
t.Errorf("err = %v, want it to relay the shim's reason", err)
}
}
// TestListFacesParsesOrdering: the shim's left-to-right index is the contract
// callers select against, so it must survive decoding intact.
func TestListFacesParsesOrdering(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if !strings.HasSuffix(r.URL.Path, "/upstream/faceswap/v1/faces") {
t.Errorf("path = %q", r.URL.Path)
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"count":2,"faces":[
{"index":0,"box":[10,20,30,40],"score":0.9,"width":20,"height":20},
{"index":1,"box":[50,20,90,60],"score":0.8,"width":40,"height":40}]}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
m, _ := p.FaceSwapModel("faceswap")
faces, err := m.ListFaces(context.Background(), editInit(t))
if err != nil {
t.Fatalf("list: %v", err)
}
if len(faces) != 2 || faces[0].Index != 0 || faces[1].Index != 1 {
t.Fatalf("faces = %+v", faces)
}
if faces[1].Box != [4]int{50, 20, 90, 60} {
t.Errorf("box = %v", faces[1].Box)
}
}
// TestListFacesRejectsShortBox: a malformed box would send a caller at the
// wrong face, which is worse than an error.
func TestListFacesRejectsShortBox(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"count":1,"faces":[{"index":0,"box":[1,2],"score":0.9}]}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
m, _ := p.FaceSwapModel("faceswap")
if _, err := m.ListFaces(context.Background(), editInit(t)); err == nil {
t.Fatal("a 2-element box was accepted")
}
}
// TestFaceSwapRejectsHeaderlessNonImage is the regression for gadfly's
// blocking finding on #23, agreed by both models. sniffImageMIME falls back
// to image/png when detection is inconclusive, and the original guard only
// looked at Content-Type — so a JSON error body sent WITHOUT a Content-Type
// header was labelled a PNG and returned as a successful image. The shim
// answers JSON on a semantic miss, which is precisely the body that would
// have sailed through.
func TestFaceSwapRejectsHeaderlessNonImage(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
// Explicitly no Content-Type — Go only sets one if we write before
// deleting it, so clear it to model a bare upstream response.
w.Header()["Content-Type"] = nil
_, _ = w.Write([]byte(`{"detail":{"error":"no_face_in_target"}}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
m, _ := p.FaceSwapModel("faceswap")
img := editInit(t)
_, err := m.FaceSwap(context.Background(), imagegen.FaceSwapRequest{Target: img, Source: img})
if err == nil {
t.Fatal("a headerless JSON body was accepted and would have been returned as image/png")
}
if !strings.Contains(err.Error(), "no_face_in_target") {
t.Errorf("err = %v, want it to relay what actually came back", err)
}
}
// TestFaceSwapAllowsNegativeIndexUnderAll: index is documented as ignored
// when all=true, so validating it there would reject a well-formed request.
func TestFaceSwapAllowsNegativeIndexUnderAll(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set("Content-Type", "image/png")
raw, _ := base64.StdEncoding.DecodeString(onePixelPNG)
_, _ = w.Write(raw)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
m, _ := p.FaceSwapModel("faceswap")
img := editInit(t)
neg := -1
if _, err := m.FaceSwap(context.Background(),
imagegen.FaceSwapRequest{Target: img, Source: img, Index: &neg, All: true}); err != nil {
t.Fatalf("negative index rejected under all=true, where it is ignored: %v", err)
}
// ...but still rejected when it WOULD be sent.
if _, err := m.FaceSwap(context.Background(),
imagegen.FaceSwapRequest{Target: img, Source: img, Index: &neg}); err == nil {
t.Error("negative index accepted when it would actually be sent")
}
}
// TestFaceSwapParsesSwapReport: the measured outcome is the whole reason the
// header exists — a caller that cannot tell "the likeness transferred" from
// "an image came back" goes and asks a vision model, which is wrong in
// exactly the cases that matter.
func TestFaceSwapParsesSwapReport(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "image/png")
w.Header().Set("X-Swap-Report",
`{"image":[1010,1200],"faces":[{"index":2,"size":[138,172],"yaw":-82.2,"identity_similarity":0.791}]}`)
raw, _ := base64.StdEncoding.DecodeString(onePixelPNG)
_, _ = w.Write(raw)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
m, err := p.FaceSwapModel("faceswap")
if err != nil {
t.Fatalf("model: %v", err)
}
img := editInit(t)
res, err := m.FaceSwap(context.Background(), imagegen.FaceSwapRequest{Target: img, Source: img})
if err != nil {
t.Fatalf("FaceSwap: %v", err)
}
if len(res.SwappedFaces) != 1 {
t.Fatalf("SwappedFaces = %d, want 1 — the measurement was dropped", len(res.SwappedFaces))
}
f := res.SwappedFaces[0]
if f.Index != 2 || f.Width != 138 || f.Height != 172 {
t.Errorf("face = %+v, want index 2 at 138x172", f)
}
if f.Yaw == nil || *f.Yaw != -82.2 {
t.Errorf("yaw = %v, want -82.2 — the pose signal is how a caller knows a profile swap will not read", f.Yaw)
}
if f.IdentitySimilarity == nil || *f.IdentitySimilarity != 0.791 {
t.Errorf("identity_similarity = %v, want 0.791", f.IdentitySimilarity)
}
// 138/1010 — the number that says "correct, and invisible at a glance".
if got := f.FractionOfImage(); got < 0.13 || got > 0.14 {
t.Errorf("FractionOfImage = %.3f, want ~0.137", got)
}
}
// TestFaceSwapSurvivesMissingReport: an older shim sends no header at all. A
// swap that produced a good image must not fail because the diagnostics
// beside it were absent or malformed.
func TestFaceSwapSurvivesMissingReport(t *testing.T) {
for name, hdr := range map[string]string{
"absent": "",
"garbage": "not json at all",
"wrongtype": `{"faces":"nope"}`,
} {
t.Run(name, func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "image/png")
if hdr != "" {
w.Header().Set("X-Swap-Report", hdr)
}
raw, _ := base64.StdEncoding.DecodeString(onePixelPNG)
_, _ = w.Write(raw)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
m, err := p.FaceSwapModel("faceswap")
if err != nil {
t.Fatalf("model: %v", err)
}
img := editInit(t)
res, err := m.FaceSwap(context.Background(), imagegen.FaceSwapRequest{Target: img, Source: img})
if err != nil {
t.Fatalf("a %s report failed the whole swap: %v", name, err)
}
if len(res.Images) != 1 {
t.Fatal("image lost")
}
if res.SwappedFaces != nil {
t.Errorf("SwappedFaces = %+v, want nil for a %s report", res.SwappedFaces, name)
}
})
}
}
+44
View File
@@ -0,0 +1,44 @@
package llamaswap
import (
"context"
"fmt"
"io"
"net/http"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
// Health reports whether the llama-swap instance is reachable (GET
// {base}/health, which answers "OK" without touching any model). A nil error
// means the proxy itself is up — it says nothing about how long a subsequent
// request will take (a cold model swap can still block for minutes). Bound
// the probe with a short context deadline; the client has no timeout by
// design.
//
// Failure taxonomy matches the rest of the package: transport failures wrap
// the raw net error; a reachable-but-unhealthy status comes back as
// *llm.APIError, so callers can errors.As-distinguish the two.
func (p *Provider) Health(ctx context.Context) error {
if err := p.requireBaseURL(); err != nil {
return err
}
req, err := p.newRequest(ctx, http.MethodGet, "/health", "", nil)
if err != nil {
return err
}
resp, err := p.client.Do(req)
if err != nil {
return fmt.Errorf("llama-swap: health probe: %w", err)
}
defer resp.Body.Close()
_, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 1<<10))
if resp.StatusCode/100 != 2 {
return &llm.APIError{
Provider: p.name,
Status: resp.StatusCode,
Message: "health endpoint returned a non-2xx status",
}
}
return nil
}
+59
View File
@@ -0,0 +1,59 @@
package llamaswap
import (
"context"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
func TestHealth(t *testing.T) {
var gotPath, gotAuth string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
gotAuth = r.Header.Get("Authorization")
_, _ = w.Write([]byte("OK"))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithToken("tok"), WithHTTPClient(srv.Client()))
if err := p.Health(context.Background()); err != nil {
t.Fatalf("Health: %v", err)
}
if gotPath != "/health" || gotAuth != "Bearer tok" {
t.Errorf("path/auth = %q/%q", gotPath, gotAuth)
}
}
func TestHealthUnhealthyStatus(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusBadGateway)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
err := p.Health(context.Background())
if err == nil || !strings.Contains(err.Error(), "502") {
t.Errorf("err = %v, want status-502 error", err)
}
}
func TestHealthUnreachable(t *testing.T) {
// A closed server: transport error, not an HTTP status.
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {}))
url := srv.URL
srv.Close()
p := New(WithBaseURL(url))
if err := p.Health(context.Background()); err == nil {
t.Error("want error for unreachable host")
}
}
func TestHealthNoBaseURL(t *testing.T) {
p := New()
if err := p.Health(context.Background()); err == nil {
t.Error("want error without base URL")
}
}
+184 -48
View File
@@ -5,6 +5,7 @@ import (
"encoding/base64"
"fmt"
"net/http"
"strconv"
"strings"
"gitea.stevedudenhoeffer.com/steve/majordomo/imagegen"
@@ -15,8 +16,8 @@ import (
// served by llama-swap (routed to a stable-diffusion.cpp upstream). The id is
// passed through verbatim and selects which upstream llama-swap loads.
func (p *Provider) ImageModel(id string, opts ...imagegen.ModelOption) (imagegen.Model, error) {
if p.baseURL == "" {
return nil, fmt.Errorf("llama-swap provider %q: no base URL configured (set one via WithBaseURL or an LLM_* env DSN)", p.name)
if err := p.requireBaseURL(); err != nil {
return nil, err
}
_ = imagegen.ApplyModelOptions(opts)
return &imageModel{p: p, id: id}, nil
@@ -27,60 +28,82 @@ type imageModel struct {
id string
}
// imageRequest is the OpenAI /v1/images/generations request shape. We always
// request b64_json so the bytes come back inline (no second fetch).
type imageRequest struct {
Model string `json:"model"`
Prompt string `json:"prompt"`
N int `json:"n,omitempty"`
Size string `json:"size,omitempty"`
ResponseFormat string `json:"response_format"`
// txt2imgRequest is the stable-diffusion.cpp sd-server A1111 request shape
// (POST /sdapi/v1/txt2img). We use this endpoint rather than the OpenAI
// /v1/images/generations one because that endpoint IGNORES `seed` on this
// sd-server build — every render of a given prompt comes back byte-identical,
// so a batch of N collapses to one image. /sdapi/v1/txt2img honours `seed`,
// giving real variety. llama-swap still routes by the `model` field in the
// body. Optional fields are pointers/omitempty so an unset value falls back to
// the model's baked default (the per-model --steps/--cfg-scale/etc. flags).
type txt2imgRequest struct {
Model string `json:"model"`
Prompt string `json:"prompt"`
NegativePrompt string `json:"negative_prompt,omitempty"`
Seed *int64 `json:"seed,omitempty"`
Steps *int `json:"steps,omitempty"`
CFGScale *float64 `json:"cfg_scale,omitempty"`
Width *int `json:"width,omitempty"`
Height *int `json:"height,omitempty"`
SampleMethod string `json:"sample_method,omitempty"`
BatchCount int `json:"batch_count,omitempty"`
}
type imageResponse struct {
Created int64 `json:"created"`
Data []struct {
B64JSON string `json:"b64_json"`
URL string `json:"url"`
} `json:"data"`
type txt2imgResponse struct {
Images []string `json:"images"`
}
// Generate implements imagegen.Model via POST {base}/v1/images/generations.
// sdWire validates the generation knobs shared by Generate and Edit and
// builds the common txt2img wire fields. verb labels validation errors.
func (m *imageModel) sdWire(verb, prompt, negativePrompt, sampler, size string, seed *int64, steps *int, cfgScale *float64, n int) (txt2imgRequest, error) {
if strings.TrimSpace(prompt) == "" {
return txt2imgRequest{}, fmt.Errorf("%w: image %s requires a prompt", llm.ErrUnsupported, verb)
}
if n < 0 {
return txt2imgRequest{}, fmt.Errorf("%w: image count N must be >= 0, got %d", llm.ErrUnsupported, n)
}
width, height, err := parseSize(size)
if err != nil {
return txt2imgRequest{}, fmt.Errorf("%w: %v", llm.ErrUnsupported, err)
}
return txt2imgRequest{
Model: m.id,
Prompt: prompt,
NegativePrompt: negativePrompt,
Seed: seed,
Steps: steps,
CFGScale: cfgScale,
Width: width,
Height: height,
SampleMethod: sampler,
BatchCount: n,
}, nil
}
// Generate implements imagegen.Model via POST {base}/sdapi/v1/txt2img.
func (m *imageModel) Generate(ctx context.Context, req imagegen.Request, opts ...imagegen.Option) (*imagegen.Result, error) {
req = req.Apply(opts...)
if strings.TrimSpace(req.Prompt) == "" {
return nil, fmt.Errorf("%w: image generation requires a prompt", llm.ErrUnsupported)
}
if req.N < 0 {
return nil, fmt.Errorf("%w: image count N must be >= 0, got %d", llm.ErrUnsupported, req.N)
}
wire := imageRequest{
Model: m.id,
Prompt: req.Prompt,
N: req.N,
Size: req.Size,
ResponseFormat: "b64_json",
}
var resp imageResponse
if err := m.p.doJSON(ctx, http.MethodPost, "/v1/images/generations", m.id, &wire, &resp); err != nil {
wire, err := m.sdWire("generation", req.Prompt, req.NegativePrompt, req.Sampler, req.Size, req.Seed, req.Steps, req.CFGScale, req.N)
if err != nil {
return nil, err
}
out := &imagegen.Result{Raw: &resp}
for i, d := range resp.Data {
if d.B64JSON == "" {
// Why error rather than skip: a url-only entry means the backend
// ignored response_format; we don't fetch remote content (mirrors
// llm.ImagePart's bytes-only contract), so surface it.
return nil, &llm.APIError{
Provider: m.p.name,
Model: m.id,
Message: fmt.Sprintf("image %d returned no inline b64_json data", i),
}
var resp txt2imgResponse
if err := m.p.doJSON(ctx, http.MethodPost, "/sdapi/v1/txt2img", m.id, &wire, &resp); err != nil {
return nil, err
}
return decodeImages(m.p.name, m.id, &resp)
}
// decodeImages converts an SDAPI response's base64 images into an
// imagegen.Result, erroring when nothing decodable came back.
func decodeImages(provider, model string, resp *txt2imgResponse) (*imagegen.Result, error) {
out := &imagegen.Result{Raw: resp}
for i, b64 := range resp.Images {
if b64 == "" {
continue
}
raw, err := base64.StdEncoding.DecodeString(d.B64JSON)
raw, err := base64.StdEncoding.DecodeString(b64)
if err != nil {
return nil, fmt.Errorf("llama-swap: decode image %d: %w", i, err)
}
@@ -88,14 +111,127 @@ func (m *imageModel) Generate(ctx context.Context, req imagegen.Request, opts ..
}
if len(out.Images) == 0 {
return nil, &llm.APIError{
Provider: m.p.name,
Model: m.id,
Provider: provider,
Model: model,
Message: "image response contained no images",
}
}
return out, nil
}
// img2imgRequest is the stable-diffusion.cpp sd-server A1111 request shape
// (POST /sdapi/v1/img2img): txt2img's fields plus the init image(s) and
// denoising strength. Same endpoint-family choice as txt2img — the OpenAI
// /v1/images/edits route is multipart and drops `seed` on this sd-server
// build, while the SDAPI shape reuses doJSON and keeps seed parity.
type img2imgRequest struct {
txt2imgRequest
InitImages []string `json:"init_images"`
DenoisingStrength *float64 `json:"denoising_strength,omitempty"`
// Mask enables inpainting: base64 image, white = repaint, black = keep
// (sd-server also accepts a data URL; plain base64 keeps symmetry with
// init_images). sd-server has no mask_blur/inpaint_full_res — callers
// wanting soft edges pre-feather the mask.
Mask string `json:"mask,omitempty"`
}
// refEditRequest is the wire shape for an INSTRUCTION-EDIT model. sd-server
// exposes reference images as `extra_images` on the shared img-gen request
// builder (routes_sdapi.cpp lands them in gen_params.ref_images — the same
// place the CLI's -r/--ref-image goes), and that field is read on BOTH
// /txt2img and /img2img.
//
// It posts to /txt2img because there is no init latent to denoise: the
// reference IS the conditioning, so an init image plus a denoising strength
// would only add noise to a pipeline that does not want any. Output
// resolution follows the reference image.
type refEditRequest struct {
txt2imgRequest
ExtraImages []string `json:"extra_images"`
}
// Edit implements imagegen.Editor. Two different pipelines live behind it,
// selected by the request: RefImages routes to an instruction-edit model via
// /sdapi/v1/txt2img + extra_images, everything else is img2img. See
// imagegen.EditRequest.RefImages for why they are not interchangeable.
func (m *imageModel) Edit(ctx context.Context, req imagegen.EditRequest, opts ...imagegen.EditOption) (*imagegen.Result, error) {
req = req.Apply(opts...)
if len(req.RefImages) > 0 {
return m.editByReference(ctx, req)
}
if len(req.Init.Data) == 0 {
return nil, fmt.Errorf("%w: image edit requires an init image", llm.ErrUnsupported)
}
if req.Strength != nil && (*req.Strength < 0 || *req.Strength > 1) {
return nil, fmt.Errorf("%w: edit strength must be in [0,1], got %g", llm.ErrUnsupported, *req.Strength)
}
base, err := m.sdWire("edit", req.Prompt, req.NegativePrompt, req.Sampler, req.Size, req.Seed, req.Steps, req.CFGScale, req.N)
if err != nil {
return nil, err
}
wire := img2imgRequest{
txt2imgRequest: base,
InitImages: []string{base64.StdEncoding.EncodeToString(req.Init.Data)},
DenoisingStrength: req.Strength,
}
if len(req.Mask.Data) > 0 {
wire.Mask = base64.StdEncoding.EncodeToString(req.Mask.Data)
}
var resp txt2imgResponse
if err := m.p.doJSON(ctx, http.MethodPost, "/sdapi/v1/img2img", m.id, &wire, &resp); err != nil {
return nil, err
}
return decodeImages(m.p.name, m.id, &resp)
}
// editByReference runs the instruction-edit path. Mask and Strength are
// deliberately NOT rejected when set: a caller that hands the same
// EditRequest to whichever model is configured should get the better result
// on a Kontext-class model, not an error, and both fields describe a
// pipeline that simply does not exist here.
func (m *imageModel) editByReference(ctx context.Context, req imagegen.EditRequest) (*imagegen.Result, error) {
base, err := m.sdWire("reference edit", req.Prompt, req.NegativePrompt, req.Sampler, req.Size, req.Seed, req.Steps, req.CFGScale, req.N)
if err != nil {
return nil, err
}
wire := refEditRequest{txt2imgRequest: base}
for _, ref := range req.RefImages {
if len(ref.Data) == 0 {
continue
}
wire.ExtraImages = append(wire.ExtraImages, base64.StdEncoding.EncodeToString(ref.Data))
}
if len(wire.ExtraImages) == 0 {
return nil, fmt.Errorf("%w: reference edit requires at least one non-empty reference image", llm.ErrUnsupported)
}
var resp txt2imgResponse
if err := m.p.doJSON(ctx, http.MethodPost, "/sdapi/v1/txt2img", m.id, &wire, &resp); err != nil {
return nil, err
}
return decodeImages(m.p.name, m.id, &resp)
}
// parseSize splits a "WxH" string into width/height pointers. "" yields
// (nil, nil) so the model's own default resolution applies.
func parseSize(size string) (*int, *int, error) {
size = strings.TrimSpace(size)
if size == "" {
return nil, nil, nil
}
parts := strings.SplitN(strings.ToLower(size), "x", 2)
if len(parts) != 2 {
return nil, nil, fmt.Errorf("invalid size %q (want WxH)", size)
}
w, err1 := strconv.Atoi(strings.TrimSpace(parts[0]))
h, err2 := strconv.Atoi(strings.TrimSpace(parts[1]))
if err1 != nil || err2 != nil || w <= 0 || h <= 0 {
return nil, nil, fmt.Errorf("invalid size %q (want WxH)", size)
}
return &w, &h, nil
}
// sniffImageMIME identifies the image format from its leading bytes, defaulting
// to image/png (stable-diffusion.cpp emits PNG) when detection is inconclusive.
func sniffImageMIME(data []byte) string {
+97
View File
@@ -0,0 +1,97 @@
// lipsync.go implements videogen.LipsyncProvider against a SadTalker shim
// reached through llama-swap's /upstream passthrough (ADR-0025):
//
// POST /upstream/<id>/v1/talking_head multipart image,audio[,still,enhance,preprocess]
//
// The response body IS the encoded clip (same contract as /v1/videos/sync),
// hence the video-sized response cap and the same MIME resolution rules.
// Generation is sync and slow (minutes) — bound calls with a context
// deadline (Hunyuan precedent).
package llamaswap
import (
"bytes"
"context"
"fmt"
"mime/multipart"
"net/http"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/majordomo/videogen"
)
// LipsyncModel implements videogen.LipsyncProvider. The id selects which
// upstream llama-swap loads.
func (p *Provider) LipsyncModel(id string, opts ...videogen.LipsyncModelOption) (videogen.Lipsyncer, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
_ = videogen.ApplyLipsyncModelOptions(opts)
return &lipsyncModel{p: p, id: id}, nil
}
type lipsyncModel struct {
p *Provider
id string
}
// Lipsync implements videogen.Lipsyncer.
func (m *lipsyncModel) Lipsync(ctx context.Context, req videogen.LipsyncRequest, opts ...videogen.LipsyncOption) (*videogen.Result, error) {
req = req.Apply(opts...)
if len(req.Image.Data) == 0 {
return nil, fmt.Errorf("%w: lipsync requires a portrait image", llm.ErrUnsupported)
}
if len(req.Audio) == 0 {
return nil, fmt.Errorf("%w: lipsync requires audio bytes", llm.ErrUnsupported)
}
if req.Preprocess != "" && req.Preprocess != "crop" && req.Preprocess != "full" {
return nil, fmt.Errorf("%w: lipsync preprocess must be \"crop\" or \"full\", got %q", llm.ErrUnsupported, req.Preprocess)
}
path, err := upstreamPath(m.id, "/v1/talking_head")
if err != nil {
return nil, err
}
// Two file parts — buildMultipart handles exactly one, so assemble by
// hand (mirrors videoModel.Generate).
var buf bytes.Buffer
w := multipart.NewWriter(&buf)
fw, err := w.CreateFormFile("image", imageFilename(req.Image.MIME, "frame"))
if err != nil {
return nil, fmt.Errorf("llama-swap: build lipsync form: %w", err)
}
if _, err := fw.Write(req.Image.Data); err != nil {
return nil, fmt.Errorf("llama-swap: build lipsync form: %w", err)
}
fw, err = w.CreateFormFile("audio", transcriptionFilename(req.AudioFilename, req.AudioMIME))
if err != nil {
return nil, fmt.Errorf("llama-swap: build lipsync form: %w", err)
}
if _, err := fw.Write(req.Audio); err != nil {
return nil, fmt.Errorf("llama-swap: build lipsync form: %w", err)
}
still := ""
if req.Still {
still = "true"
}
enhance := ""
if req.Enhance {
enhance = "true"
}
if err := writeFormFields(w, "build lipsync form", []formField{
{"still", still, false},
{"enhance", enhance, false},
{"preprocess", req.Preprocess, false},
}); err != nil {
return nil, err
}
if err := w.Close(); err != nil {
return nil, fmt.Errorf("llama-swap: build lipsync form: %w", err)
}
raw, respType, err := m.p.doRaw(ctx, http.MethodPost, path, m.id, w.FormDataContentType(), &buf, maxVideoResponseBytes)
if err != nil {
return nil, err
}
return singleVideoResult(m.p.name, m.id, "lipsync", raw, respType)
}
@@ -0,0 +1,295 @@
package llamaswap
import (
"context"
"errors"
"io"
"net/http"
"net/http/httptest"
"testing"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/majordomo/videogen"
)
// webmFixture is a minimal EBML header so http.DetectContentType sniffs
// video/webm.
func webmFixture() []byte {
return append([]byte{0x1A, 0x45, 0xDF, 0xA3}, make([]byte, 20)...)
}
func TestLipsync(t *testing.T) {
png := pngFixture(t)
var gotPath, gotStill, gotEnhance, gotPreprocess string
var gotImage, gotAudio []byte
var gotImageName, gotAudioName string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Fatalf("parse multipart: %v", err)
}
gotStill = r.FormValue("still")
gotEnhance = r.FormValue("enhance")
gotPreprocess = r.FormValue("preprocess")
if f, hdr, err := r.FormFile("image"); err == nil {
gotImage, _ = io.ReadAll(f)
gotImageName = hdr.Filename
f.Close()
} else {
t.Errorf("image part: %v", err)
}
if f, hdr, err := r.FormFile("audio"); err == nil {
gotAudio, _ = io.ReadAll(f)
gotAudioName = hdr.Filename
f.Close()
} else {
t.Errorf("audio part: %v", err)
}
w.Header().Set("Content-Type", "video/mp4")
_, _ = w.Write(mp4Fixture())
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
ls, err := p.LipsyncModel("lipsync-sadtalker")
if err != nil {
t.Fatalf("LipsyncModel: %v", err)
}
res, err := ls.Lipsync(context.Background(),
videogen.LipsyncRequest{
Image: videogen.Image{MIME: "image/png", Data: png},
Audio: []byte("SPEECH"),
AudioMIME: "audio/wav",
},
videogen.WithLipsyncStill(), videogen.WithLipsyncEnhance(), videogen.WithLipsyncPreprocess("full"))
if err != nil {
t.Fatalf("Lipsync: %v", err)
}
if gotPath != "/upstream/lipsync-sadtalker/v1/talking_head" {
t.Errorf("path = %q", gotPath)
}
if gotStill != "true" || gotEnhance != "true" || gotPreprocess != "full" {
t.Errorf("still/enhance/preprocess = %q/%q/%q", gotStill, gotEnhance, gotPreprocess)
}
if string(gotImage) != string(png) || gotImageName != "frame.png" {
t.Errorf("image bytes/name = %d bytes/%q", len(gotImage), gotImageName)
}
if string(gotAudio) != "SPEECH" || gotAudioName != "audio.wav" {
t.Errorf("audio = %q name = %q", gotAudio, gotAudioName)
}
if res.Video.MIME != "video/mp4" || len(res.Video.Data) == 0 {
t.Fatalf("video = %q/%d bytes", res.Video.MIME, len(res.Video.Data))
}
}
func TestLipsyncOmitsUnsetFlags(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Fatalf("parse multipart: %v", err)
}
for _, k := range []string{"still", "enhance", "preprocess"} {
if v, ok := r.MultipartForm.Value[k]; ok {
t.Errorf("unset request sent %q = %v, want omitted", k, v)
}
}
w.Header().Set("Content-Type", "video/mp4")
_, _ = w.Write(mp4Fixture())
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
ls, _ := p.LipsyncModel("lipsync-sadtalker")
if _, err := ls.Lipsync(context.Background(),
videogen.LipsyncRequest{Image: videogen.Image{Data: pngFixture(t)}, Audio: []byte("A")}); err != nil {
t.Fatalf("Lipsync: %v", err)
}
}
func TestLipsyncRejectsBadArgs(t *testing.T) {
p := New(WithBaseURL("http://unused"))
ls, _ := p.LipsyncModel("lipsync-sadtalker")
if _, err := ls.Lipsync(context.Background(),
videogen.LipsyncRequest{Audio: []byte("A")}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("no image: err = %v, want ErrUnsupported", err)
}
if _, err := ls.Lipsync(context.Background(),
videogen.LipsyncRequest{Image: videogen.Image{Data: []byte{1}}}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("no audio: err = %v, want ErrUnsupported", err)
}
if _, err := ls.Lipsync(context.Background(),
videogen.LipsyncRequest{Image: videogen.Image{Data: []byte{1}}, Audio: []byte{1}, Preprocess: "zoom"}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("preprocess zoom: err = %v, want ErrUnsupported", err)
}
}
func TestLipsyncRejectsNonVideoResponse(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/html")
_, _ = w.Write([]byte("<html>proxy error page</html>"))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
ls, _ := p.LipsyncModel("lipsync-sadtalker")
_, err := ls.Lipsync(context.Background(),
videogen.LipsyncRequest{Image: videogen.Image{Data: pngFixture(t)}, Audio: []byte("A")})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want APIError for non-video body", err)
}
}
func TestRemoveVideoBackground(t *testing.T) {
var gotPath, gotOutput, gotFilename string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Fatalf("parse multipart: %v", err)
}
gotOutput = r.FormValue("output")
if _, hdr, err := r.FormFile("file"); err == nil {
gotFilename = hdr.Filename
} else {
t.Errorf("file part: %v", err)
}
w.Header().Set("Content-Type", "video/webm")
_, _ = w.Write(webmFixture())
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
vb, err := p.VideoBackgroundRemoverModel("mediautils")
if err != nil {
t.Fatalf("VideoBackgroundRemoverModel: %v", err)
}
res, err := vb.RemoveVideoBackground(context.Background(),
videogen.VideoBackgroundRemovalRequest{Video: mp4Fixture(), MIME: "video/mp4"},
videogen.WithVideoBackgroundOutput("alpha_webm"))
if err != nil {
t.Fatalf("RemoveVideoBackground: %v", err)
}
if gotPath != "/upstream/mediautils/v1/video/matte" {
t.Errorf("path = %q", gotPath)
}
if gotOutput != "alpha_webm" || gotFilename != "video.mp4" {
t.Errorf("output/filename = %q/%q", gotOutput, gotFilename)
}
if res.Video.MIME != "video/webm" || len(res.Video.Data) == 0 {
t.Fatalf("video = %q/%d bytes", res.Video.MIME, len(res.Video.Data))
}
}
func TestRemoveVideoBackgroundOmitsDefaultOutput(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Fatalf("parse multipart: %v", err)
}
if v, ok := r.MultipartForm.Value["output"]; ok {
t.Errorf("output field sent for default: %v; want omitted", v)
}
w.Header().Set("Content-Type", "video/mp4")
_, _ = w.Write(mp4Fixture())
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
vb, _ := p.VideoBackgroundRemoverModel("mediautils")
if _, err := vb.RemoveVideoBackground(context.Background(),
videogen.VideoBackgroundRemovalRequest{Video: mp4Fixture()}); err != nil {
t.Fatalf("RemoveVideoBackground: %v", err)
}
}
func TestRemoveVideoBackgroundRejectsBadArgs(t *testing.T) {
p := New(WithBaseURL("http://unused"))
vb, _ := p.VideoBackgroundRemoverModel("mediautils")
if _, err := vb.RemoveVideoBackground(context.Background(),
videogen.VideoBackgroundRemovalRequest{}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("no video: err = %v, want ErrUnsupported", err)
}
if _, err := vb.RemoveVideoBackground(context.Background(),
videogen.VideoBackgroundRemovalRequest{Video: []byte{1}, Output: "gif"}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("output gif: err = %v, want ErrUnsupported", err)
}
}
func TestUpscaleVideo(t *testing.T) {
var gotPath, gotScale string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Fatalf("parse multipart: %v", err)
}
gotScale = r.FormValue("scale")
w.Header().Set("Content-Type", "video/mp4")
_, _ = w.Write(mp4Fixture())
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
vu, err := p.VideoUpscalerModel("mediautils")
if err != nil {
t.Fatalf("VideoUpscalerModel: %v", err)
}
res, err := vu.UpscaleVideo(context.Background(),
videogen.VideoUpscaleRequest{Video: mp4Fixture(), MIME: "video/mp4"},
videogen.WithVideoUpscaleScale(2))
if err != nil {
t.Fatalf("UpscaleVideo: %v", err)
}
if gotPath != "/upstream/mediautils/v1/video/upscale" {
t.Errorf("path = %q", gotPath)
}
if gotScale != "2" {
t.Errorf("scale = %q", gotScale)
}
if res.Video.MIME != "video/mp4" || len(res.Video.Data) == 0 {
t.Fatalf("video = %q/%d bytes", res.Video.MIME, len(res.Video.Data))
}
}
func TestUpscaleVideoRejectsBadArgs(t *testing.T) {
p := New(WithBaseURL("http://unused"))
vu, _ := p.VideoUpscalerModel("mediautils")
if _, err := vu.UpscaleVideo(context.Background(), videogen.VideoUpscaleRequest{}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("no video: err = %v, want ErrUnsupported", err)
}
if _, err := vu.UpscaleVideo(context.Background(),
videogen.VideoUpscaleRequest{Video: []byte{1}, Scale: 3}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("scale 3: err = %v, want ErrUnsupported", err)
}
}
func TestUpscaleVideoRejectsNonVideoResponse(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header()["Content-Type"] = nil // NO Content-Type at all
_, _ = w.Write([]byte("502 bad gateway"))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
vu, _ := p.VideoUpscalerModel("mediautils")
_, err := vu.UpscaleVideo(context.Background(), videogen.VideoUpscaleRequest{Video: mp4Fixture()})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want APIError for headerless non-video body", err)
}
}
func TestVideoInputFilename(t *testing.T) {
cases := []struct {
filename, mime, want string
}{
{"clip.mp4", "", "clip.mp4"},
{"evil\r\nclip.mp4", "", "evilclip.mp4"},
{"", "video/mp4", "video.mp4"},
{"", "video/webm", "video.webm"},
{"", "video/quicktime", "video.mov"},
{"", "", "video"},
}
for _, tc := range cases {
if got := videoInputFilename(tc.filename, tc.mime); got != tc.want {
t.Errorf("videoInputFilename(%q, %q) = %q, want %q", tc.filename, tc.mime, got, tc.want)
}
}
}
+82 -16
View File
@@ -9,9 +9,11 @@
// package adds beyond a bare OpenAI-compat endpoint is the "tailored" surface:
//
// - llama-swap management endpoints exposed as concrete methods — ListModels
// (GET /v1/models), Running (GET /running), Unload (POST /api/models/unload)
// — which have no place on the canonical llm.Provider interface;
// - image generation via the imagegen interface (see image.go); and
// (GET /v1/models), Running (GET /running), Unload (POST /api/models/unload),
// ListVoices (GET /v1/audio/voices), Health (GET /health) — which have no
// place on the canonical llm.Provider interface;
// - image generation + editing via the imagegen interfaces (see image.go);
// - speech synthesis + transcription via the audio interfaces (see audio.go); and
// - swap-aware defaults: the HTTP client carries NO timeout, because the
// first request to an unloaded model blocks while llama-swap spawns the
// upstream (its healthCheckTimeout is at least 15s). Bound a call with a
@@ -28,6 +30,8 @@ import (
"encoding/json"
"fmt"
"io"
"mime"
"mime/multipart"
"net/http"
"strings"
@@ -43,6 +47,18 @@ const DefaultName = "llama-swap"
// can't make a decode allocate without limit.
const maxResponseBytes = 64 << 20
// maxVideoResponseBytes caps the /v1/videos/sync body — the response IS an
// encoded clip, which legitimately dwarfs any JSON/audio payload (a long
// high-bitrate generation can pass 64MB). Still bounded so a buggy upstream
// can't allocate without limit.
const maxVideoResponseBytes = 512 << 20
// maxAudioResponseBytes caps bodies that ARE a single encoded audio clip
// (voice clone, speech enhancement, sfx): a long uncompressed WAV
// legitimately passes the 64MB JSON cap. Still bounded so a buggy upstream
// can't allocate without limit.
const maxAudioResponseBytes = 256 << 20
// Provider is a llama-swap client. It satisfies llm.Provider (chat, delegated
// to provider/openai) and imagegen.Provider (image generation), and exposes
// llama-swap's management endpoints as concrete methods.
@@ -100,8 +116,8 @@ func (p *Provider) BaseURL() string { return p.baseURL }
// endpoint, delegating to provider/openai. The id is passed through verbatim
// and selects which upstream llama-swap loads.
func (p *Provider) Model(id string, opts ...llm.ModelOption) (llm.Model, error) {
if p.baseURL == "" {
return nil, fmt.Errorf("llama-swap provider %q: no base URL configured (set one via WithBaseURL or an LLM_* env DSN)", p.name)
if err := p.requireBaseURL(); err != nil {
return nil, err
}
return p.chatProvider().Model(id, opts...)
}
@@ -178,7 +194,32 @@ func (p *Provider) Unload(ctx context.Context, model string) error {
return p.doJSON(ctx, http.MethodPost, path, "", nil, nil)
}
// --- shared HTTP helper for management + image endpoints ---
// --- shared HTTP helpers for management + image + audio endpoints ---
// requireBaseURL is the shared guard for every entry point: construction
// never fails (see New), so a missing base URL surfaces here, at use time.
func (p *Provider) requireBaseURL() error {
if p.baseURL == "" {
return fmt.Errorf("llama-swap provider %q: no base URL configured (set one via WithBaseURL or an LLM_* env DSN)", p.name)
}
return nil
}
// newRequest builds an authenticated request relative to baseURL.
// contentType is applied only when a body is present.
func (p *Provider) newRequest(ctx context.Context, method, path, contentType string, body io.Reader) (*http.Request, error) {
req, err := http.NewRequestWithContext(ctx, method, p.baseURL+path, body)
if err != nil {
return nil, fmt.Errorf("llama-swap: build request: %w", err)
}
if body != nil && contentType != "" {
req.Header.Set("Content-Type", contentType)
}
if p.token != "" {
req.Header.Set("Authorization", "Bearer "+p.token)
}
return req, nil
}
// doJSON performs a request to a llama-swap endpoint relative to baseURL,
// optionally encoding body and decoding into out (either may be nil). model
@@ -186,8 +227,8 @@ func (p *Provider) Unload(ctx context.Context, model string) error {
// model-specific). Transport failures are wrapped raw so llm.Classify still
// sees the underlying net error; non-2xx responses become *llm.APIError.
func (p *Provider) doJSON(ctx context.Context, method, path, model string, body, out any) error {
if p.baseURL == "" {
return fmt.Errorf("llama-swap provider %q: no base URL configured (set one via WithBaseURL or an LLM_* env DSN)", p.name)
if err := p.requireBaseURL(); err != nil {
return err
}
var rdr io.Reader
if body != nil {
@@ -197,15 +238,9 @@ func (p *Provider) doJSON(ctx context.Context, method, path, model string, body,
}
rdr = bytes.NewReader(b)
}
req, err := http.NewRequestWithContext(ctx, method, p.baseURL+path, rdr)
req, err := p.newRequest(ctx, method, path, "application/json", rdr)
if err != nil {
return fmt.Errorf("llama-swap: build request: %w", err)
}
if body != nil {
req.Header.Set("Content-Type", "application/json")
}
if p.token != "" {
req.Header.Set("Authorization", "Bearer "+p.token)
return err
}
resp, err := p.client.Do(req)
if err != nil {
@@ -259,3 +294,34 @@ func (p *Provider) apiError(resp *http.Response, model string) error {
e.Message = strings.TrimSpace(string(body))
return e
}
// formField is one multipart form entry for the media endpoints. Required
// fields always go on the wire (an empty model id should fail loudly
// upstream, not silently vanish); optional ones only when set.
type formField struct {
key, value string
required bool
}
// writeFormFields appends fields to a multipart writer, skipping unset
// optional entries. wrap labels errors ("build transcription form", ...).
func writeFormFields(w *multipart.Writer, wrap string, fields []formField) error {
for _, f := range fields {
if !f.required && f.value == "" {
continue
}
if err := w.WriteField(f.key, f.value); err != nil {
return fmt.Errorf("llama-swap: %s: %w", wrap, err)
}
}
return nil
}
// mimeFromContentType returns the parsed media type when it matches prefix
// ("audio/", "video/"), else "".
func mimeFromContentType(contentType, prefix string) string {
if mt, _, err := mime.ParseMediaType(contentType); err == nil && strings.HasPrefix(mt, prefix) {
return mt
}
return ""
}
+46 -7
View File
@@ -166,11 +166,11 @@ func TestRunningRaw(t *testing.T) {
func TestImageGenerate(t *testing.T) {
var gotBody map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v1/images/generations" {
if r.URL.Path != "/sdapi/v1/txt2img" {
t.Errorf("path = %q", r.URL.Path)
}
_ = json.NewDecoder(r.Body).Decode(&gotBody)
_, _ = w.Write([]byte(`{"created":1,"data":[{"b64_json":"` + onePixelPNG + `"}]}`))
_, _ = w.Write([]byte(`{"images":["` + onePixelPNG + `"]}`))
}))
defer srv.Close()
@@ -192,12 +192,51 @@ func TestImageGenerate(t *testing.T) {
if len(res.Images[0].Data) == 0 {
t.Error("decoded image has no bytes")
}
// response_format must be forced to b64_json, and options applied.
if gotBody["response_format"] != "b64_json" {
t.Errorf("response_format = %v, want b64_json", gotBody["response_format"])
// Size is split into width/height ints for the A1111 endpoint.
if gotBody["width"] != float64(512) || gotBody["height"] != float64(512) {
t.Errorf("width/height = %v/%v, want 512/512", gotBody["width"], gotBody["height"])
}
if gotBody["size"] != "512x512" {
t.Errorf("size = %v, want 512x512", gotBody["size"])
}
func TestImageGenerateSettings(t *testing.T) {
var gotBody map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_ = json.NewDecoder(r.Body).Decode(&gotBody)
_, _ = w.Write([]byte(`{"images":["` + onePixelPNG + `"]}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
im, _ := p.ImageModel("sd")
// Unset overrides must be omitted entirely so sd-server keeps its own
// per-model defaults.
if _, err := im.Generate(context.Background(), imagegen.Request{Prompt: "x"}); err != nil {
t.Fatalf("Generate: %v", err)
}
for _, k := range []string{"steps", "cfg_scale", "negative_prompt", "sample_method", "seed"} {
if v, ok := gotBody[k]; ok {
t.Errorf("unset request sent %q = %v, want omitted", k, v)
}
}
// Set overrides are forwarded with the sd-server-friendly field names.
gotBody = nil
_, err := im.Generate(context.Background(), imagegen.Request{Prompt: "x"},
imagegen.WithSteps(8),
imagegen.WithCFGScale(3.5),
imagegen.WithNegativePrompt("blurry"),
imagegen.WithSampler("euler"),
imagegen.WithSeed(42),
)
if err != nil {
t.Fatalf("Generate: %v", err)
}
want := map[string]any{"steps": float64(8), "cfg_scale": 3.5, "negative_prompt": "blurry", "sample_method": "euler", "seed": float64(42)}
for k, w := range want {
if gotBody[k] != w {
t.Errorf("%s = %v, want %v", k, gotBody[k], w)
}
}
}
+211
View File
@@ -0,0 +1,211 @@
// mediautil.go implements the imagegen.UpscaleProvider,
// imagegen.BackgroundRemovalProvider, and videogen.InterpolationProvider
// surfaces against upstreams reached through llama-swap's /upstream
// passthrough (ADR-0020):
//
// upscale POST /upstream/<id>/v1/upscale (mediautils shim)
// background POST /upstream/<id>/api/remove (rembg server)
// interpolate POST /upstream/<id>/v1/interpolate (mediautils shim)
//
// All three are one-file multipart in, raw bytes out.
package llamaswap
import (
"context"
"fmt"
"net/http"
"strconv"
"strings"
"gitea.stevedudenhoeffer.com/steve/majordomo/imagegen"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/majordomo/videogen"
)
// maxImageResponseBytes caps the raw-image bodies from the upscale and
// background-removal endpoints. The 64MB maxResponseBytes is a JSON cap;
// a 4x-upscaled PNG legitimately exceeds it. Bounded (not video-sized)
// because a single still image past this is an upstream bug, not data.
const maxImageResponseBytes = 256 << 20
// --- upscale ---
// UpscaleModel implements imagegen.UpscaleProvider against the mediautils
// shim's POST /v1/upscale. The id selects which upstream llama-swap loads.
func (p *Provider) UpscaleModel(id string, opts ...imagegen.UpscaleModelOption) (imagegen.Upscaler, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
_ = imagegen.ApplyUpscaleModelOptions(opts)
return &upscaleModel{p: p, id: id}, nil
}
type upscaleModel struct {
p *Provider
id string
}
// Upscale implements imagegen.Upscaler.
func (m *upscaleModel) Upscale(ctx context.Context, req imagegen.UpscaleRequest, opts ...imagegen.UpscaleOption) (*imagegen.Result, error) {
req = req.Apply(opts...)
if len(req.Image.Data) == 0 {
return nil, fmt.Errorf("%w: upscale requires an image", llm.ErrUnsupported)
}
if req.Scale != 0 && req.Scale != 2 && req.Scale != 4 {
return nil, fmt.Errorf("%w: upscale scale must be 2 or 4, got %d", llm.ErrUnsupported, req.Scale)
}
path, err := upstreamPath(m.id, "/v1/upscale")
if err != nil {
return nil, err
}
scale := ""
if req.Scale != 0 {
scale = strconv.Itoa(req.Scale)
}
body, contentType, err := buildMultipart("build upscale form",
filePart{field: "file", filename: "image.png", data: req.Image.Data},
[]formField{{"scale", scale, false}})
if err != nil {
return nil, err
}
raw, respType, err := m.p.doRaw(ctx, http.MethodPost, path, m.id, contentType, body, maxImageResponseBytes)
if err != nil {
return nil, err
}
return singleImageResult(m.p.name, m.id, "upscale", raw, respType)
}
// --- background removal ---
// BackgroundRemovalModel implements imagegen.BackgroundRemovalProvider
// against a rembg server's POST /api/remove.
func (p *Provider) BackgroundRemovalModel(id string, opts ...imagegen.BackgroundRemovalModelOption) (imagegen.BackgroundRemover, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
_ = imagegen.ApplyBackgroundRemovalModelOptions(opts)
return &backgroundModel{p: p, id: id}, nil
}
type backgroundModel struct {
p *Provider
id string
}
// RemoveBackground implements imagegen.BackgroundRemover. rembg's `model`
// form field is the request's Net (its internal network); the llama-swap
// model id only picks the upstream. `om=true` returns the black/white
// foreground mask instead of the RGBA cutout.
func (m *backgroundModel) RemoveBackground(ctx context.Context, req imagegen.BackgroundRemovalRequest, opts ...imagegen.BackgroundRemovalOption) (*imagegen.Result, error) {
req = req.Apply(opts...)
if len(req.Image.Data) == 0 {
return nil, fmt.Errorf("%w: background removal requires an image", llm.ErrUnsupported)
}
path, err := upstreamPath(m.id, "/api/remove")
if err != nil {
return nil, err
}
om := ""
if req.OnlyMask {
om = "true"
}
body, contentType, err := buildMultipart("build background-removal form",
filePart{field: "file", filename: "image.png", data: req.Image.Data},
[]formField{
{"model", req.Net, false},
{"om", om, false},
})
if err != nil {
return nil, err
}
raw, respType, err := m.p.doRaw(ctx, http.MethodPost, path, m.id, contentType, body, maxImageResponseBytes)
if err != nil {
return nil, err
}
return singleImageResult(m.p.name, m.id, "background removal", raw, respType)
}
// singleImageResult wraps one raw image body into an imagegen.Result.
// Acceptance requires positive evidence of image-ness — sniffed magic
// bytes or a declared image/* Content-Type — so a proxy error page with
// an empty Content-Type can never become "the image" (sniffImageMIME's
// PNG default is a labelling fallback, not a validator).
func singleImageResult(provider, model, verb string, raw []byte, contentType string) (*imagegen.Result, error) {
if len(raw) == 0 {
return nil, &llm.APIError{Provider: provider, Model: model, Message: verb + " response contained no image"}
}
sniffed := http.DetectContentType(raw)
declared := strings.TrimSpace(contentType)
if !strings.HasPrefix(sniffed, "image/") && !strings.HasPrefix(declared, "image/") {
return nil, &llm.APIError{Provider: provider, Model: model,
Message: fmt.Sprintf("%s response is not an image (Content-Type %q, sniffed %q)", verb, declared, sniffed)}
}
mimeType := sniffed
if !strings.HasPrefix(mimeType, "image/") {
mimeType = declared
}
return &imagegen.Result{Images: []llm.ImagePart{{MIME: mimeType, Data: raw}}}, nil
}
// --- frame interpolation ---
// InterpolatorModel implements videogen.InterpolationProvider against the
// mediautils shim's POST /v1/interpolate.
func (p *Provider) InterpolatorModel(id string, opts ...videogen.InterpolatorModelOption) (videogen.Interpolator, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
_ = videogen.ApplyInterpolatorModelOptions(opts)
return &interpolatorModel{p: p, id: id}, nil
}
type interpolatorModel struct {
p *Provider
id string
}
// Interpolate implements videogen.Interpolator. The response body IS the
// encoded clip (same contract as /v1/videos/sync), hence the video-sized
// response cap and the same MIME resolution rules.
func (m *interpolatorModel) Interpolate(ctx context.Context, req videogen.InterpolateRequest, opts ...videogen.InterpolateOption) (*videogen.Result, error) {
req = req.Apply(opts...)
if len(req.Video.Data) == 0 {
return nil, fmt.Errorf("%w: interpolation requires a video", llm.ErrUnsupported)
}
if req.Multi != 0 && req.Multi != 2 && req.Multi != 4 {
return nil, fmt.Errorf("%w: interpolation multi must be 2 or 4, got %d", llm.ErrUnsupported, req.Multi)
}
if req.TargetFPS < 0 {
return nil, fmt.Errorf("%w: target fps must be >= 0, got %d", llm.ErrUnsupported, req.TargetFPS)
}
path, err := upstreamPath(m.id, "/v1/interpolate")
if err != nil {
return nil, err
}
multi := ""
if req.Multi != 0 {
multi = strconv.Itoa(req.Multi)
}
mode := ""
targetFPS := ""
if req.SlowMo {
mode = "slowmo"
} else if req.TargetFPS > 0 {
targetFPS = strconv.Itoa(req.TargetFPS)
}
body, contentType, err := buildMultipart("build interpolate form",
filePart{field: "file", filename: "video.mp4", data: req.Video.Data},
[]formField{
{"multi", multi, false},
{"mode", mode, false},
{"target_fps", targetFPS, false},
})
if err != nil {
return nil, err
}
raw, respType, err := m.p.doRaw(ctx, http.MethodPost, path, m.id, contentType, body, maxVideoResponseBytes)
if err != nil {
return nil, err
}
return singleVideoResult(m.p.name, m.id, "interpolate", raw, respType)
}
+282
View File
@@ -0,0 +1,282 @@
package llamaswap
import (
"context"
"encoding/base64"
"errors"
"net/http"
"net/http/httptest"
"strings"
"testing"
"gitea.stevedudenhoeffer.com/steve/majordomo/imagegen"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/majordomo/videogen"
)
func pngFixture(t *testing.T) []byte {
t.Helper()
raw, err := base64.StdEncoding.DecodeString(onePixelPNG)
if err != nil {
t.Fatalf("decode fixture: %v", err)
}
return raw
}
func TestUpscale(t *testing.T) {
png := pngFixture(t)
var gotPath, gotScale, gotFilename string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Errorf("parse multipart: %v", err)
}
gotScale = r.FormValue("scale")
if f, hdr, err := r.FormFile("file"); err == nil {
gotFilename = hdr.Filename
f.Close()
} else {
t.Errorf("file part: %v", err)
}
w.Header().Set("Content-Type", "image/png")
_, _ = w.Write(png)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
up, err := p.UpscaleModel("mediautils")
if err != nil {
t.Fatalf("UpscaleModel: %v", err)
}
res, err := up.Upscale(context.Background(),
imagegen.UpscaleRequest{Image: imagegen.Image{MIME: "image/png", Data: png}},
imagegen.WithUpscaleScale(2))
if err != nil {
t.Fatalf("Upscale: %v", err)
}
if gotPath != "/upstream/mediautils/v1/upscale" {
t.Errorf("path = %q", gotPath)
}
if gotScale != "2" || gotFilename != "image.png" {
t.Errorf("scale/filename = %q/%q", gotScale, gotFilename)
}
if len(res.Images) != 1 || res.Images[0].MIME != "image/png" {
t.Fatalf("images = %+v", res.Images)
}
}
func TestUpscaleRejectsBadArgs(t *testing.T) {
p := New(WithBaseURL("http://unused"))
up, _ := p.UpscaleModel("mediautils")
if _, err := up.Upscale(context.Background(), imagegen.UpscaleRequest{}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("no image: err = %v, want ErrUnsupported", err)
}
if _, err := up.Upscale(context.Background(),
imagegen.UpscaleRequest{Image: imagegen.Image{Data: []byte{1}}, Scale: 3}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("scale 3: err = %v, want ErrUnsupported", err)
}
}
func TestUpscaleRejectsNonImageResponse(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/html")
_, _ = w.Write([]byte("<html>proxy error page</html>"))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
up, _ := p.UpscaleModel("mediautils")
_, err := up.Upscale(context.Background(),
imagegen.UpscaleRequest{Image: imagegen.Image{Data: pngFixture(t)}})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want APIError for non-image body", err)
}
}
func TestRemoveBackground(t *testing.T) {
png := pngFixture(t)
var gotPath, gotNet, gotOM string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Errorf("parse multipart: %v", err)
}
gotNet = r.FormValue("model")
gotOM = r.FormValue("om")
w.Header().Set("Content-Type", "image/png")
_, _ = w.Write(png)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
br, err := p.BackgroundRemovalModel("rembg")
if err != nil {
t.Fatalf("BackgroundRemovalModel: %v", err)
}
res, err := br.RemoveBackground(context.Background(),
imagegen.BackgroundRemovalRequest{Image: imagegen.Image{Data: png}},
imagegen.WithBackgroundNet("birefnet-general"), imagegen.WithOnlyMask())
if err != nil {
t.Fatalf("RemoveBackground: %v", err)
}
if gotPath != "/upstream/rembg/api/remove" {
t.Errorf("path = %q", gotPath)
}
if gotNet != "birefnet-general" || gotOM != "true" {
t.Errorf("net/om = %q/%q", gotNet, gotOM)
}
if len(res.Images) != 1 {
t.Fatalf("images = %+v", res.Images)
}
}
func TestRemoveBackgroundOmitsUnsetFields(t *testing.T) {
png := pngFixture(t)
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Errorf("parse multipart: %v", err)
}
if _, ok := r.MultipartForm.Value["model"]; ok {
t.Error("model field sent for default net; want omitted")
}
if _, ok := r.MultipartForm.Value["om"]; ok {
t.Error("om field sent for cutout mode; want omitted")
}
w.Header().Set("Content-Type", "image/png")
_, _ = w.Write(png)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
br, _ := p.BackgroundRemovalModel("rembg")
if _, err := br.RemoveBackground(context.Background(),
imagegen.BackgroundRemovalRequest{Image: imagegen.Image{Data: png}}); err != nil {
t.Fatalf("RemoveBackground: %v", err)
}
}
// mp4Fixture is a minimal ftyp box so http.DetectContentType sniffs video/mp4.
func mp4Fixture() []byte {
return []byte{0, 0, 0, 24, 'f', 't', 'y', 'p', 'i', 's', 'o', 'm',
0, 0, 2, 0, 'i', 's', 'o', 'm', 'i', 's', 'o', '2'}
}
func TestInterpolate(t *testing.T) {
var gotPath, gotMulti, gotMode, gotTargetFPS string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Errorf("parse multipart: %v", err)
}
gotMulti = r.FormValue("multi")
gotMode = r.FormValue("mode")
gotTargetFPS = r.FormValue("target_fps")
w.Header().Set("Content-Type", "video/mp4")
_, _ = w.Write(mp4Fixture())
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
ip, err := p.InterpolatorModel("mediautils")
if err != nil {
t.Fatalf("InterpolatorModel: %v", err)
}
res, err := ip.Interpolate(context.Background(),
videogen.InterpolateRequest{Video: videogen.Video{Data: mp4Fixture(), MIME: "video/mp4"}},
videogen.WithInterpolateMulti(2), videogen.WithTargetFPS(60))
if err != nil {
t.Fatalf("Interpolate: %v", err)
}
if gotPath != "/upstream/mediautils/v1/interpolate" {
t.Errorf("path = %q", gotPath)
}
if gotMulti != "2" || gotMode != "" || gotTargetFPS != "60" {
t.Errorf("multi/mode/target_fps = %q/%q/%q", gotMulti, gotMode, gotTargetFPS)
}
if res.Video.MIME != "video/mp4" || len(res.Video.Data) == 0 {
t.Fatalf("video = %q/%d bytes", res.Video.MIME, len(res.Video.Data))
}
}
func TestInterpolateSlowMoSetsModeAndDropsTargetFPS(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Errorf("parse multipart: %v", err)
}
if got := r.FormValue("mode"); got != "slowmo" {
t.Errorf("mode = %q, want slowmo", got)
}
if _, ok := r.MultipartForm.Value["target_fps"]; ok {
t.Error("target_fps sent in slowmo mode; want omitted")
}
w.Header().Set("Content-Type", "video/mp4")
_, _ = w.Write(mp4Fixture())
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
ip, _ := p.InterpolatorModel("mediautils")
_, err := ip.Interpolate(context.Background(),
videogen.InterpolateRequest{Video: videogen.Video{Data: mp4Fixture()}, TargetFPS: 60},
videogen.WithSlowMo())
if err != nil {
t.Fatalf("Interpolate: %v", err)
}
}
func TestInterpolateRejectsBadArgs(t *testing.T) {
p := New(WithBaseURL("http://unused"))
ip, _ := p.InterpolatorModel("mediautils")
if _, err := ip.Interpolate(context.Background(), videogen.InterpolateRequest{}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("no video: err = %v, want ErrUnsupported", err)
}
if _, err := ip.Interpolate(context.Background(),
videogen.InterpolateRequest{Video: videogen.Video{Data: []byte{1}}, Multi: 3}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("multi 3: err = %v, want ErrUnsupported", err)
}
}
func TestUpstreamPathRejectsSeparators(t *testing.T) {
for _, bad := range []string{"", "a/b", "a?b", "a#b", "a%2Fb", "%2e%2e", "a%b"} {
if _, err := upstreamPath(bad, "/x"); err == nil {
t.Errorf("upstreamPath(%q) succeeded; want error", bad)
}
}
got, err := upstreamPath("rembg", "api/remove")
if err != nil || got != "/upstream/rembg/api/remove" {
t.Errorf("upstreamPath = %q, %v", got, err)
}
if !strings.HasPrefix(got, "/upstream/") {
t.Errorf("path prefix wrong: %q", got)
}
}
func TestUpscaleRejectsEmptyContentTypeNonImage(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header()["Content-Type"] = nil // suppress auto-detection: NO Content-Type at all
_, _ = w.Write([]byte("502 bad gateway"))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
up, _ := p.UpscaleModel("mediautils")
_, err := up.Upscale(context.Background(),
imagegen.UpscaleRequest{Image: imagegen.Image{Data: pngFixture(t)}})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want APIError for headerless non-image body", err)
}
}
func TestUpstreamPathRejectsDotDot(t *testing.T) {
if _, err := upstreamPath("..", "/x"); err == nil {
t.Error("model '..' accepted")
}
if _, err := upstreamPath("m", "/v1/audio?path=../../api/models/unload"); err == nil {
t.Error("dot-dot rest accepted")
}
if _, err := upstreamPath("m", "/v1/audio?path=https://evil.example/x"); err == nil {
t.Error("absolute-URL rest accepted")
}
}
+157
View File
@@ -0,0 +1,157 @@
// mesh.go implements meshgen.Provider against a Hunyuan3D-2.1-style
// api_server reached through llama-swap's /upstream passthrough (ADR-0020):
//
// POST /upstream/<id>/generate JSON {image: <b64>, type: "glb"|"stl"|...}
//
// The response body IS the encoded mesh (binary, Content-Type
// application/octet-stream), so one request yields exactly one asset.
package llamaswap
import (
"bytes"
"context"
"encoding/base64"
"encoding/json"
"fmt"
"net/http"
"strings"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/majordomo/meshgen"
)
// maxMeshResponseBytes caps the /generate body — a high-resolution textured
// mesh legitimately passes the 64MB JSON cap, but stays far under video
// scale.
const maxMeshResponseBytes = 256 << 20
// MeshModel implements meshgen.Provider. The id selects which upstream
// llama-swap loads.
func (p *Provider) MeshModel(id string, opts ...meshgen.ModelOption) (meshgen.Model, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
_ = meshgen.ApplyModelOptions(opts)
return &meshModel{p: p, id: id}, nil
}
type meshModel struct {
p *Provider
id string
}
// hunyuanGenerateRequest is the Hunyuan3D api_server /generate shape.
// Optional fields are pointers/omitempty so unset values fall back to the
// server's defaults (mirrors the sd-server wire structs). Type is sent for
// forward-compat but the LIVE server's GenerationRequest has no such field
// and always returns GLB (verified 2026-07-14) — the response format is
// therefore SNIFFED, and non-GLB output is the caller's conversion problem
// (meshgen.Converter / the mediautils shim).
type hunyuanGenerateRequest struct {
Image string `json:"image"`
Type string `json:"type,omitempty"`
Texture bool `json:"texture"`
RemoveBackground *bool `json:"remove_background,omitempty"`
OctreeResolution int `json:"octree_resolution,omitempty"`
NumInferenceSteps *int `json:"num_inference_steps,omitempty"`
GuidanceScale *float64 `json:"guidance_scale,omitempty"`
Seed *int64 `json:"seed,omitempty"`
FaceCount int `json:"face_count,omitempty"`
}
// meshFormats maps the supported output containers to MIME types.
var meshFormats = map[string]string{
"glb": "model/gltf-binary",
"stl": "model/stl",
"obj": "model/obj",
}
// Generate implements meshgen.Model.
func (m *meshModel) Generate(ctx context.Context, req meshgen.Request, opts ...meshgen.Option) (*meshgen.Result, error) {
req = req.Apply(opts...)
if len(req.Image.Data) == 0 {
return nil, fmt.Errorf("%w: mesh generation requires an image", llm.ErrUnsupported)
}
format := strings.ToLower(strings.TrimSpace(req.Format))
if format == "" {
format = "glb"
}
mimeType, ok := meshFormats[format]
if !ok {
return nil, fmt.Errorf("%w: unsupported mesh format %q (want glb, stl, or obj)", llm.ErrUnsupported, req.Format)
}
if req.OctreeResolution < 0 || req.FaceCount < 0 {
return nil, fmt.Errorf("%w: octree resolution and face count must be >= 0", llm.ErrUnsupported)
}
path, err := upstreamPath(m.id, "/generate")
if err != nil {
return nil, err
}
wire := hunyuanGenerateRequest{
Image: base64.StdEncoding.EncodeToString(req.Image.Data),
Type: format,
Texture: req.Texture,
RemoveBackground: req.RemoveBackground,
OctreeResolution: req.OctreeResolution,
NumInferenceSteps: req.Steps,
GuidanceScale: req.GuidanceScale,
Seed: req.Seed,
FaceCount: req.FaceCount,
}
// doRaw + manual JSON encode (not doJSON): the SUCCESS body is binary
// mesh bytes, not JSON.
encoded, err := json.Marshal(wire)
if err != nil {
return nil, fmt.Errorf("llama-swap: encode mesh request: %w", err)
}
raw, respType, err := m.p.doRaw(ctx, http.MethodPost, path, m.id, "application/json", bytes.NewReader(encoded), maxMeshResponseBytes)
if err != nil {
return nil, err
}
if len(raw) == 0 {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id, Message: "mesh response contained no data"}
}
// A JSON body on the success path is the server reporting a soft error
// (or an API drift) — never hand it back as "the mesh". Two signals:
// the declared Content-Type, and a whitespace-tolerant peek at the
// leading bytes (512 covers any indented error envelope; a binary STL
// header theoretically CAN start with '{', but a real one also won't
// be all-whitespace-then-brace).
if strings.Contains(strings.ToLower(respType), "application/json") {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: "mesh response is JSON, not mesh bytes: " + truncateForError(raw)}
}
if first := strings.TrimLeft(string(raw[:min(len(raw), 512)]), " \t\r\n"); strings.HasPrefix(first, "{") || strings.HasPrefix(first, "[") {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: "mesh response is JSON, not mesh bytes: " + truncateForError(raw)}
}
// Label the result by what the bytes ARE, not what was requested: the
// live server ignores the format field entirely.
actualFormat, actualMIME := sniffMeshFormat(raw, format, mimeType)
return &meshgen.Result{Mesh: meshgen.Mesh{Data: raw, Format: actualFormat, MIME: actualMIME}}, nil
}
// sniffMeshFormat identifies the mesh container from magic bytes, falling
// back to the requested format only when the bytes are ambiguous (binary
// STL has no magic).
func sniffMeshFormat(raw []byte, requested, requestedMIME string) (string, string) {
switch {
case len(raw) >= 4 && string(raw[:4]) == "glTF":
return "glb", meshFormats["glb"]
case len(raw) >= 6 && strings.EqualFold(string(raw[:6]), "solid "):
return "stl", meshFormats["stl"]
default:
return requested, requestedMIME
}
}
// truncateForError bounds a payload quoted into an error message.
func truncateForError(b []byte) string {
const maxErrLen = 500
if len(b) > maxErrLen {
return string(b[:maxErrLen]) + "..."
}
return string(b)
}
+64
View File
@@ -0,0 +1,64 @@
// mesh_convert.go implements meshgen.ConverterProvider against the
// mediautils shim's POST /v1/convert_mesh (multipart file + target),
// reached through the /upstream passthrough (ADR-0020). Exists because
// Hunyuan3D's api_server always returns GLB — STL for the printer
// pipeline is produced by this conversion hop.
package llamaswap
import (
"context"
"fmt"
"net/http"
"strings"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/majordomo/meshgen"
)
// MeshConverter implements meshgen.ConverterProvider.
func (p *Provider) MeshConverter(id string) (meshgen.Converter, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
return &meshConverter{p: p, id: id}, nil
}
type meshConverter struct {
p *Provider
id string
}
// Convert implements meshgen.Converter.
func (m *meshConverter) Convert(ctx context.Context, mesh meshgen.Mesh, format string) (*meshgen.Result, error) {
if len(mesh.Data) == 0 {
return nil, fmt.Errorf("%w: mesh conversion requires mesh bytes", llm.ErrUnsupported)
}
format = strings.ToLower(strings.TrimSpace(format))
mimeType, ok := meshFormats[format]
if !ok {
return nil, fmt.Errorf("%w: unsupported mesh format %q (want glb, stl, or obj)", llm.ErrUnsupported, format)
}
path, err := upstreamPath(m.id, "/v1/convert_mesh")
if err != nil {
return nil, err
}
filename := "mesh." + mesh.Format
if mesh.Format == "" {
filename = "mesh"
}
body, contentType, err := buildMultipart("build mesh-convert form",
filePart{field: "file", filename: filename, data: mesh.Data},
[]formField{{"target", format, true}})
if err != nil {
return nil, err
}
raw, _, err := m.p.doRaw(ctx, http.MethodPost, path, m.id, contentType, body, maxMeshResponseBytes)
if err != nil {
return nil, err
}
if len(raw) == 0 {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id, Message: "mesh conversion returned no data"}
}
actualFormat, actualMIME := sniffMeshFormat(raw, format, mimeType)
return &meshgen.Result{Mesh: meshgen.Mesh{Data: raw, Format: actualFormat, MIME: actualMIME}}, nil
}
+170
View File
@@ -0,0 +1,170 @@
package llamaswap
import (
"context"
"encoding/base64"
"encoding/json"
"errors"
"net/http"
"net/http/httptest"
"testing"
"gitea.stevedudenhoeffer.com/steve/majordomo/audio"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/majordomo/meshgen"
)
func TestMeshGenerate(t *testing.T) {
png := pngFixture(t)
stl := []byte("solid mort\nendsolid mort\n")
var gotPath string
var gotBody map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
_ = json.NewDecoder(r.Body).Decode(&gotBody)
w.Header().Set("Content-Type", "application/octet-stream")
_, _ = w.Write(stl)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
mm, err := p.MeshModel("image3d-hunyuan21")
if err != nil {
t.Fatalf("MeshModel: %v", err)
}
res, err := mm.Generate(context.Background(),
meshgen.Request{Image: meshgen.Image{MIME: "image/png", Data: png}},
meshgen.WithFormat("stl"), meshgen.WithSeed(7))
if err != nil {
t.Fatalf("Generate: %v", err)
}
if gotPath != "/upstream/image3d-hunyuan21/generate" {
t.Errorf("path = %q", gotPath)
}
if gotBody["type"] != "stl" || gotBody["image"] != base64.StdEncoding.EncodeToString(png) {
t.Errorf("type/image = %v/(b64 mismatch)", gotBody["type"])
}
if gotBody["seed"] != float64(7) {
t.Errorf("seed = %v", gotBody["seed"])
}
if _, ok := gotBody["octree_resolution"]; ok {
t.Error("octree_resolution sent when unset; want omitted")
}
if res.Mesh.Format != "stl" || res.Mesh.MIME != "model/stl" || string(res.Mesh.Data) != string(stl) {
t.Fatalf("mesh = %+v", res.Mesh)
}
}
func TestMeshGenerateRejectsJSONBody(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/octet-stream")
_, _ = w.Write([]byte(` {"detail": "queue full"}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
mm, _ := p.MeshModel("image3d-hunyuan21")
_, err := mm.Generate(context.Background(),
meshgen.Request{Image: meshgen.Image{Data: pngFixture(t)}})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want APIError for JSON success body", err)
}
}
func TestMeshGenerateRejectsBadArgs(t *testing.T) {
p := New(WithBaseURL("http://unused"))
mm, _ := p.MeshModel("image3d-hunyuan21")
if _, err := mm.Generate(context.Background(), meshgen.Request{}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("no image: err = %v, want ErrUnsupported", err)
}
if _, err := mm.Generate(context.Background(),
meshgen.Request{Image: meshgen.Image{Data: []byte{1}}, Format: "fbx"}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("format fbx: err = %v, want ErrUnsupported", err)
}
}
func TestDiarize(t *testing.T) {
var gotPath string
var gotQuery map[string][]string
var gotField string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
gotQuery = r.URL.Query()
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Errorf("parse multipart: %v", err)
}
if f, _, err := r.FormFile("audio_file"); err == nil {
gotField = "audio_file"
f.Close()
}
_, _ = w.Write([]byte(`{
"text": "hello there. general kenobi.",
"language": "en",
"segments": [
{"start": 0.0, "end": 1.2, "text": "hello there.", "speaker": "SPEAKER_00"},
{"start": 1.4, "end": 3.0, "text": "general kenobi.", "speaker": "SPEAKER_01"}
]
}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
dm, err := p.DiarizationModel("whisperx-diarize")
if err != nil {
t.Fatalf("DiarizationModel: %v", err)
}
res, err := dm.Diarize(context.Background(),
audio.DiarizationRequest{Audio: []byte("RIFFfake"), MIME: "audio/wav"},
audio.WithSpeakerBounds(2, 4), audio.WithDiarizationLanguage("en"))
if err != nil {
t.Fatalf("Diarize: %v", err)
}
if gotPath != "/upstream/whisperx-diarize/asr" {
t.Errorf("path = %q", gotPath)
}
for k, want := range map[string]string{
"output": "json", "diarize": "true",
"min_speakers": "2", "max_speakers": "4", "language": "en",
} {
if got := gotQuery[k]; len(got) != 1 || got[0] != want {
t.Errorf("query %s = %v, want %q", k, got, want)
}
}
if gotField != "audio_file" {
t.Error("audio_file part missing")
}
if len(res.Segments) != 2 || res.Segments[1].Speaker != "SPEAKER_01" {
t.Fatalf("segments = %+v", res.Segments)
}
if res.Language != "en" || res.Text == "" {
t.Errorf("language/text = %q/%q", res.Language, res.Text)
}
}
func TestDiarizeRejectsBadArgs(t *testing.T) {
p := New(WithBaseURL("http://unused"))
dm, _ := p.DiarizationModel("whisperx-diarize")
if _, err := dm.Diarize(context.Background(), audio.DiarizationRequest{}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("no audio: err = %v, want ErrUnsupported", err)
}
if _, err := dm.Diarize(context.Background(),
audio.DiarizationRequest{Audio: []byte{1}, MinSpeakers: 5, MaxSpeakers: 2}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("bounds 5>2: err = %v, want ErrUnsupported", err)
}
}
func TestDiarizeEmptyTranscriptIsError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{"text": "", "segments": []}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
dm, _ := p.DiarizationModel("whisperx-diarize")
_, err := dm.Diarize(context.Background(), audio.DiarizationRequest{Audio: []byte{1}})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want APIError for empty transcript", err)
}
}
+293
View File
@@ -0,0 +1,293 @@
// music.go implements musicgen.Provider against an ACE-Step-1.5-style API
// server reached through llama-swap's /upstream passthrough (ADR-0021):
//
// POST /upstream/<id>/release_task {prompt, lyrics, ...} -> {task_id}
// POST /upstream/<id>/query_result {task_id_list: [...]} -> status+result
// GET /upstream/<id>/<result file URL> -> audio bytes
//
// The backend is an async job queue; Generate wraps it into the blocking
// one-call contract by polling, so a context deadline is the caller's
// budget for the whole job (mort's tool timeout sits well under its
// agent-runtime ceiling for exactly this reason).
package llamaswap
import (
"context"
"encoding/json"
"fmt"
"net/http"
"strings"
"time"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/majordomo/musicgen"
)
// musicPollInterval is the delay between query_result polls. Long enough to
// be polite to the queue, short enough that a ~10s xl-turbo song isn't
// dominated by poll latency. A var so tests can shrink it.
var musicPollInterval = 2 * time.Second
// musicPollMaxConsecutiveFailures bounds how many consecutive BAD polls
// (transport error, unparseable payload, task momentarily absent) are
// tolerated before aborting. A multi-minute GPU job must not die to one
// blip; a genuinely broken upstream still fails within ~5 intervals.
const musicPollMaxConsecutiveFailures = 5
// MusicModel implements musicgen.Provider. The id selects which upstream
// llama-swap loads.
func (p *Provider) MusicModel(id string, opts ...musicgen.ModelOption) (musicgen.Model, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
_ = musicgen.ApplyModelOptions(opts)
return &musicModel{p: p, id: id}, nil
}
type musicModel struct {
p *Provider
id string
}
// releaseTaskRequest is the ACE-Step POST /release_task shape. audio_duration
// is the v1 param name; verify against the upstream ACE-Step-1.5 repo's
// docs/en/API.md at smoke time — an
// unknown field is ignored upstream, degrading to the default clip length,
// never an error.
type releaseTaskRequest struct {
Prompt string `json:"prompt"`
Lyrics string `json:"lyrics,omitempty"`
AudioFormat string `json:"audio_format,omitempty"`
TaskType string `json:"task_type"`
AudioDuration int `json:"audio_duration,omitempty"`
InferenceSteps *int `json:"inference_steps,omitempty"`
Seed *int64 `json:"seed,omitempty"`
}
// queryItem is one task's poll state. `result` arrives as a JSON-encoded
// STRING (the ACE-Step API double-encodes it).
type queryItem struct {
TaskID string `json:"task_id"`
Status int `json:"status"` // 0 queued/running, 1 succeeded, 2 failed
Result string `json:"result"`
}
// musicResult is the useful subset of ACE-Step's double-encoded result
// blob.
type musicResult struct {
File string `json:"file"`
}
// parseMusicResult decodes the `result` string, tolerating two live-API
// quirks (observed 2026-07-14): the payload is an ARRAY of result objects
// (not a bare object), and string values can contain RAW control
// characters (a literal newline in timing fields) that strict JSON
// rejects. Control chars can only legally sit inside string values in the
// double-encoded blob, so replacing them with spaces preserves structure.
func parseMusicResult(blob string) (musicResult, bool) {
sanitized := strings.Map(func(r rune) rune {
if r < 0x20 {
return ' '
}
return r
}, blob)
var arr []musicResult
if err := json.Unmarshal([]byte(sanitized), &arr); err == nil && len(arr) > 0 {
return arr[0], true
}
var one musicResult
if err := json.Unmarshal([]byte(sanitized), &one); err == nil {
return one, true
}
return musicResult{}, false
}
// musicFormatMIME resolves the clip MIME from the response Content-Type
// or the requested format, reusing speechMIME's table (one format->MIME
// map for the whole provider). wav32 is ACE-Step-specific: normalize it
// to wav before the shared lookup.
func musicFormatMIME(contentType, format string) string {
format = strings.ToLower(strings.TrimSpace(format))
if format == "wav32" {
format = "wav"
}
return speechMIME(contentType, format)
}
// Generate implements musicgen.Model.
func (m *musicModel) Generate(ctx context.Context, req musicgen.Request, opts ...musicgen.Option) (*musicgen.Result, error) {
req = req.Apply(opts...)
if strings.TrimSpace(req.Prompt) == "" {
return nil, fmt.Errorf("%w: music generation requires a prompt", llm.ErrUnsupported)
}
if req.DurationSeconds < 0 {
return nil, fmt.Errorf("%w: duration must be >= 0, got %d", llm.ErrUnsupported, req.DurationSeconds)
}
if req.Steps != nil && *req.Steps <= 0 {
return nil, fmt.Errorf("%w: inference steps must be > 0, got %d", llm.ErrUnsupported, *req.Steps)
}
taskID, err := m.releaseTask(ctx, req)
if err != nil {
return nil, err
}
item, err := m.pollResult(ctx, taskID)
if err != nil {
return nil, err
}
return m.fetchResult(ctx, req.Format, item)
}
// releaseTask submits the job and returns its task id.
func (m *musicModel) releaseTask(ctx context.Context, req musicgen.Request) (string, error) {
path, err := upstreamPath(m.id, "/release_task")
if err != nil {
return "", err
}
wire := releaseTaskRequest{
Prompt: req.Prompt,
Lyrics: req.Lyrics,
AudioFormat: req.Format,
TaskType: "text2music",
AudioDuration: req.DurationSeconds,
InferenceSteps: req.Steps,
Seed: req.Seed,
}
// Tolerant envelope: {"data": {"task_id": ...}} per the docs, with a
// top-level fallback in case the wrapper changes.
var resp struct {
Data struct {
TaskID string `json:"task_id"`
} `json:"data"`
TaskID string `json:"task_id"`
}
if err := m.p.doJSON(ctx, http.MethodPost, path, m.id, &wire, &resp); err != nil {
return "", err
}
taskID := resp.Data.TaskID
if taskID == "" {
taskID = resp.TaskID
}
if taskID == "" {
return "", &llm.APIError{Provider: m.p.name, Model: m.id, Message: "release_task returned no task_id"}
}
return taskID, nil
}
// pollResult polls query_result until the task succeeds, fails, or ctx
// expires. Transient trouble — a transport blip, a momentarily
// unparseable payload, the task briefly absent from the response — is
// tolerated up to musicPollMaxConsecutiveFailures in a row: a
// multi-minute exclusive-GPU job must not die to one flaky poll. Only an
// explicit status=2, a run of consecutive failures, or the ctx deadline
// aborts.
func (m *musicModel) pollResult(ctx context.Context, taskID string) (*queryItem, error) {
path, err := upstreamPath(m.id, "/query_result")
if err != nil {
return nil, err
}
body := map[string]any{"task_id_list": []string{taskID}}
ticker := time.NewTicker(musicPollInterval)
defer ticker.Stop()
failures := 0
var lastErr error
for {
item, pollErr := m.pollOnce(ctx, path, body, taskID)
switch {
case pollErr != nil:
// Ctx expiry is never transient — bail with the deadline error.
if ctx.Err() != nil {
return nil, fmt.Errorf("llama-swap: music generation: %w", ctx.Err())
}
failures++
lastErr = pollErr
if failures >= musicPollMaxConsecutiveFailures {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: fmt.Sprintf("music poll failed %d times in a row: %v", failures, lastErr)}
}
case item.Status == 1:
return item, nil
case item.Status == 2:
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: "music generation failed upstream: " + truncateForError([]byte(item.Result))}
default:
failures = 0 // healthy queued/running poll
}
select {
case <-ctx.Done():
return nil, fmt.Errorf("llama-swap: music generation: %w", ctx.Err())
case <-ticker.C:
}
}
}
// pollOnce performs one query_result round trip and locates the task.
func (m *musicModel) pollOnce(ctx context.Context, path string, body any, taskID string) (*queryItem, error) {
// Tolerant envelope: items under "data" or a bare array.
var raw json.RawMessage
if err := m.p.doJSON(ctx, http.MethodPost, path, m.id, body, &raw); err != nil {
return nil, err
}
return findQueryItem(raw, taskID)
}
// findQueryItem digs the task's entry out of the query_result payload,
// tolerating {"data": [...]}, {"data": {...}}, and bare-array envelopes.
func findQueryItem(raw json.RawMessage, taskID string) (*queryItem, error) {
var env struct {
Data json.RawMessage `json:"data"`
}
candidates := raw
if json.Unmarshal(raw, &env) == nil && len(env.Data) > 0 {
candidates = env.Data
}
var items []queryItem
if err := json.Unmarshal(candidates, &items); err != nil {
var one queryItem
if err := json.Unmarshal(candidates, &one); err != nil {
return nil, fmt.Errorf("unrecognized query_result payload shape")
}
items = []queryItem{one}
}
for i := range items {
// Single-item responses without a task_id echo are assumed to be
// ours — we only ever poll one task; a mismatch surfaces as a
// transient miss and is retried by the caller.
if items[i].TaskID == taskID || (items[i].TaskID == "" && len(items) == 1) {
return &items[i], nil
}
}
return nil, fmt.Errorf("query_result did not include task %s", taskID)
}
// fetchResult downloads the finished clip named by the job's result blob.
func (m *musicModel) fetchResult(ctx context.Context, format string, item *queryItem) (*musicgen.Result, error) {
result, ok := parseMusicResult(item.Result)
if !ok || result.File == "" {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: "music result blob missing file URL: " + truncateForError([]byte(item.Result))}
}
// The file URL is server-relative (e.g. "/v1/audio?path=..."); route it
// back through the same upstream. upstreamPath additionally refuses
// dot-dot/scheme smuggling in this SERVER-SUPPLIED value.
if !strings.HasPrefix(result.File, "/") {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: "music result file URL is not server-relative: " + truncateForError([]byte(result.File))}
}
path, err := upstreamPath(m.id, result.File)
if err != nil {
return nil, err
}
raw, contentType, err := m.p.doRaw(ctx, http.MethodGet, path, m.id, "", nil, maxResponseBytes)
if err != nil {
return nil, err
}
if len(raw) == 0 {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id, Message: "music response contained no audio"}
}
mimeType := musicFormatMIME(contentType, format)
return &musicgen.Result{
Audio: musicgen.Audio{Data: raw, MIME: mimeType},
Raw: json.RawMessage(item.Result),
}, nil
}
+368
View File
@@ -0,0 +1,368 @@
package llamaswap
import (
"context"
"encoding/json"
"errors"
"net/http"
"net/http/httptest"
"sync/atomic"
"testing"
"time"
"gitea.stevedudenhoeffer.com/steve/majordomo/embeddings"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/majordomo/meshgen"
"gitea.stevedudenhoeffer.com/steve/majordomo/musicgen"
)
// aceStepStub emulates the ACE-Step job API: one queued poll, then success.
func aceStepStub(t *testing.T, mp3 []byte) (*httptest.Server, *atomic.Int32) {
t.Helper()
var polls atomic.Int32
var gotRelease map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/upstream/musicgen-acestep/release_task":
_ = json.NewDecoder(r.Body).Decode(&gotRelease)
if gotRelease["task_type"] != "text2music" {
t.Errorf("task_type = %v", gotRelease["task_type"])
}
_, _ = w.Write([]byte(`{"data": {"task_id": "t-1", "status": "queued"}}`))
case "/upstream/musicgen-acestep/query_result":
n := polls.Add(1)
if n == 1 {
_, _ = w.Write([]byte(`{"data": [{"task_id": "t-1", "status": 0, "result": ""}]}`))
return
}
_, _ = w.Write([]byte(`{"data": [{"task_id": "t-1", "status": 1,
"result": "{\"file\": \"/v1/audio?path=out.mp3\", \"metas\": {\"bpm\": 120}}"}]}`))
case "/upstream/musicgen-acestep/v1/audio":
if got := r.URL.Query().Get("path"); got != "out.mp3" {
t.Errorf("audio path = %q", got)
}
w.Header().Set("Content-Type", "audio/mpeg")
_, _ = w.Write(mp3)
default:
t.Errorf("unexpected path %q", r.URL.Path)
w.WriteHeader(404)
}
}))
return srv, &polls
}
func TestMusicGenerate(t *testing.T) {
old := musicPollInterval
musicPollInterval = 5 * time.Millisecond
defer func() { musicPollInterval = old }()
mp3 := []byte("ID3fakeaudio")
srv, polls := aceStepStub(t, mp3)
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
mm, err := p.MusicModel("musicgen-acestep")
if err != nil {
t.Fatalf("MusicModel: %v", err)
}
// Shrink the poll interval indirectly by bounding the whole call: the
// stub succeeds on poll #2, so a generous deadline still finishes fast.
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
res, err := mm.Generate(ctx,
musicgen.Request{Prompt: "chiptune anthem about mortbux"},
musicgen.WithLyrics("mort mort mort"), musicgen.WithDuration(30))
if err != nil {
t.Fatalf("Generate: %v", err)
}
if polls.Load() < 2 {
t.Errorf("polls = %d, want >= 2 (queued then done)", polls.Load())
}
if res.Audio.MIME != "audio/mpeg" || string(res.Audio.Data) != string(mp3) {
t.Fatalf("audio = %q/%d bytes", res.Audio.MIME, len(res.Audio.Data))
}
}
func TestMusicGenerateUpstreamFailure(t *testing.T) {
old := musicPollInterval
musicPollInterval = 5 * time.Millisecond
defer func() { musicPollInterval = old }()
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/upstream/musicgen-acestep/release_task":
_, _ = w.Write([]byte(`{"data": {"task_id": "t-2"}}`))
case "/upstream/musicgen-acestep/query_result":
_, _ = w.Write([]byte(`{"data": [{"task_id": "t-2", "status": 2, "result": "{\"error\": \"OOM\"}"}]}`))
}
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
mm, _ := p.MusicModel("musicgen-acestep")
_, err := mm.Generate(context.Background(), musicgen.Request{Prompt: "p"})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want APIError for failed job", err)
}
}
func TestMusicGenerateRejectsBadArgs(t *testing.T) {
p := New(WithBaseURL("http://unused"))
mm, _ := p.MusicModel("musicgen-acestep")
if _, err := mm.Generate(context.Background(), musicgen.Request{}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("no prompt: err = %v, want ErrUnsupported", err)
}
}
func TestEmbed(t *testing.T) {
var gotBody map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v1/embeddings" {
t.Errorf("path = %q", r.URL.Path)
}
_ = json.NewDecoder(r.Body).Decode(&gotBody)
// Deliberately out of order: the client must sort by index.
_, _ = w.Write([]byte(`{"object":"list","data":[
{"object":"embedding","index":1,"embedding":[0.3,0.4]},
{"object":"embedding","index":0,"embedding":[0.1,0.2]}
]}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
em, err := p.EmbedModel("embed-qwen3-0.6b")
if err != nil {
t.Fatalf("EmbedModel: %v", err)
}
res, err := em.Embed(context.Background(), embeddings.EmbedRequest{Inputs: []string{"a", "b"}})
if err != nil {
t.Fatalf("Embed: %v", err)
}
if gotBody["model"] != "embed-qwen3-0.6b" {
t.Errorf("model = %v", gotBody["model"])
}
if len(res.Vectors) != 2 || res.Vectors[0][0] != 0.1 || res.Vectors[1][0] != 0.3 {
t.Fatalf("vectors = %+v (index ordering broken?)", res.Vectors)
}
}
func TestEmbedCountMismatchIsError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{"data":[{"index":0,"embedding":[0.1]}]}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
em, _ := p.EmbedModel("embed-qwen3-0.6b")
_, err := em.Embed(context.Background(), embeddings.EmbedRequest{Inputs: []string{"a", "b"}})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want APIError for count mismatch", err)
}
}
func TestEmbedRejectsEmptyInputs(t *testing.T) {
p := New(WithBaseURL("http://unused"))
em, _ := p.EmbedModel("embed-qwen3-0.6b")
if _, err := em.Embed(context.Background(), embeddings.EmbedRequest{}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("no inputs: err = %v, want ErrUnsupported", err)
}
if _, err := em.Embed(context.Background(), embeddings.EmbedRequest{Inputs: []string{"a", " "}}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("blank input: err = %v, want ErrUnsupported", err)
}
}
func TestRerank(t *testing.T) {
var gotBody map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v1/rerank" {
t.Errorf("path = %q", r.URL.Path)
}
_ = json.NewDecoder(r.Body).Decode(&gotBody)
// Out of score order: the client must sort descending.
_, _ = w.Write([]byte(`{"results":[
{"index":0,"relevance_score":0.11},
{"index":2,"relevance_score":0.93},
{"index":1,"relevance_score":0.42}
]}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
rm, err := p.RerankModel("rerank-bge-v2-m3")
if err != nil {
t.Fatalf("RerankModel: %v", err)
}
res, err := rm.Rerank(context.Background(),
embeddings.RerankRequest{Query: "what is a panda?", Documents: []string{"a", "b", "c"}},
embeddings.WithTopN(3))
if err != nil {
t.Fatalf("Rerank: %v", err)
}
if gotBody["top_n"] != float64(3) || gotBody["query"] != "what is a panda?" {
t.Errorf("top_n/query = %v/%v", gotBody["top_n"], gotBody["query"])
}
if len(res.Results) != 3 || res.Results[0].Index != 2 || res.Results[2].Index != 0 {
t.Fatalf("results = %+v (descending sort broken?)", res.Results)
}
}
func TestRerankRejectsOutOfRangeIndex(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{"results":[{"index":7,"relevance_score":0.9}]}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
rm, _ := p.RerankModel("rerank-bge-v2-m3")
_, err := rm.Rerank(context.Background(),
embeddings.RerankRequest{Query: "q", Documents: []string{"a"}})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want APIError for out-of-range index", err)
}
}
func TestInstructedQuery(t *testing.T) {
got := embeddings.InstructedQuery("", "how tall is everest")
want := "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: how tall is everest"
if got != want {
t.Errorf("InstructedQuery = %q", got)
}
}
func TestMusicGenerateSurvivesTransientPollFailures(t *testing.T) {
old := musicPollInterval
musicPollInterval = 5 * time.Millisecond
defer func() { musicPollInterval = old }()
mp3 := []byte("ID3fakeaudio")
var polls atomic.Int32
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/upstream/musicgen-acestep/release_task":
_, _ = w.Write([]byte(`{"data": {"task_id": "t-3"}}`))
case "/upstream/musicgen-acestep/query_result":
switch polls.Add(1) {
case 1:
w.WriteHeader(http.StatusBadGateway) // transient transport blip
case 2:
_, _ = w.Write([]byte(`{"data": []}`)) // task momentarily absent
default:
_, _ = w.Write([]byte(`{"data": [{"task_id": "t-3", "status": 1,
"result": "{\"file\": \"/v1/audio?path=out.mp3\"}"}]}`))
}
case "/upstream/musicgen-acestep/v1/audio":
w.Header().Set("Content-Type", "audio/mpeg")
_, _ = w.Write(mp3)
}
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
mm, _ := p.MusicModel("musicgen-acestep")
res, err := mm.Generate(context.Background(), musicgen.Request{Prompt: "p"})
if err != nil {
t.Fatalf("Generate should survive 2 transient failures: %v", err)
}
if len(res.Audio.Data) == 0 {
t.Fatal("no audio")
}
}
func TestMusicGenerateRejectsHostileFileURL(t *testing.T) {
old := musicPollInterval
musicPollInterval = 5 * time.Millisecond
defer func() { musicPollInterval = old }()
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/upstream/musicgen-acestep/release_task":
_, _ = w.Write([]byte(`{"data": {"task_id": "t-4"}}`))
case "/upstream/musicgen-acestep/query_result":
_, _ = w.Write([]byte(`{"data": [{"task_id": "t-4", "status": 1,
"result": "{\"file\": \"/v1/audio?path=../../api/models/unload\"}"}]}`))
}
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
mm, _ := p.MusicModel("musicgen-acestep")
_, err := mm.Generate(context.Background(), musicgen.Request{Prompt: "p"})
if err == nil {
t.Fatal("dot-dot result file URL accepted")
}
}
func TestParseMusicResultLiveShapes(t *testing.T) {
// Live ACE-Step (2026-07-14): result is an ARRAY and carries raw
// control characters inside string values.
arrayWithCtrl := "[{\"file\": \"/v1/audio?path=x.mp3\", \"prompt\": \"line1\nline2\"}]"
res, ok := parseMusicResult(arrayWithCtrl)
if !ok || res.File != "/v1/audio?path=x.mp3" {
t.Fatalf("array+ctrl: ok=%v res=%+v", ok, res)
}
// Docs shape (bare object) still parses.
res, ok = parseMusicResult(`{"file": "/v1/audio?path=y.mp3"}`)
if !ok || res.File != "/v1/audio?path=y.mp3" {
t.Fatalf("object: ok=%v res=%+v", ok, res)
}
if _, ok := parseMusicResult("not json"); ok {
t.Fatal("garbage parsed")
}
}
func TestMeshResultSniffsActualFormat(t *testing.T) {
// Live Hunyuan3D always returns GLB regardless of the requested
// format — the result must be labelled by its magic bytes.
glb := append([]byte("glTF"), []byte("\x02\x00\x00\x00rest")...)
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/octet-stream")
_, _ = w.Write(glb)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
mm, _ := p.MeshModel("image3d-hunyuan21")
res, err := mm.Generate(context.Background(),
meshgen.Request{Image: meshgen.Image{Data: []byte{1}}, Format: "stl"})
if err != nil {
t.Fatalf("Generate: %v", err)
}
if res.Mesh.Format != "glb" || res.Mesh.MIME != "model/gltf-binary" {
t.Fatalf("mesh labelled %s/%s, want glb (sniffed)", res.Mesh.Format, res.Mesh.MIME)
}
}
func TestMeshConverter(t *testing.T) {
stl := []byte("solid m\nendsolid m\n")
var gotTarget, gotPath string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Errorf("multipart: %v", err)
}
gotTarget = r.FormValue("target")
w.Header().Set("Content-Type", "model/stl")
_, _ = w.Write(stl)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
mc, err := p.MeshConverter("mediautils")
if err != nil {
t.Fatalf("MeshConverter: %v", err)
}
res, err := mc.Convert(context.Background(),
meshgen.Mesh{Data: []byte("glTFxxxx"), Format: "glb"}, "stl")
if err != nil {
t.Fatalf("Convert: %v", err)
}
if gotPath != "/upstream/mediautils/v1/convert_mesh" || gotTarget != "stl" {
t.Errorf("path/target = %q/%q", gotPath, gotTarget)
}
if res.Mesh.Format != "stl" {
t.Errorf("format = %q", res.Mesh.Format)
}
}
+140
View File
@@ -0,0 +1,140 @@
// ocr.go implements ocr.Provider against a Surya-style shim reached through
// llama-swap's /upstream passthrough (ADR-0023):
//
// POST /upstream/<id>/v1/ocr multipart file[,langs,max_pages]
//
// The document may be an image or a PDF (the shim rasterizes PDFs itself).
// The response is per-page JSON: {pages:[{number,text,lines,layout}]}. The
// decode is tolerant — when a page carries no aggregate `text`, its line
// texts are joined instead — and the full payload survives in Result.Raw for
// callers that want bboxes/confidence/layout.
package llamaswap
import (
"context"
"encoding/json"
"fmt"
"mime"
"net/http"
"strconv"
"strings"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/majordomo/ocr"
)
// OCRModel implements ocr.Provider. The id selects which upstream llama-swap
// loads.
func (p *Provider) OCRModel(id string, opts ...ocr.ModelOption) (ocr.Model, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
_ = ocr.ApplyModelOptions(opts)
return &ocrModel{p: p, id: id}, nil
}
type ocrModel struct {
p *Provider
id string
}
// ocrResponse is the Surya shim's /v1/ocr shape (the subset this package
// relies on; per-line bbox/confidence and layout stay in Raw).
type ocrResponse struct {
Pages []struct {
Number int `json:"number"`
Text string `json:"text"`
Lines []struct {
Text string `json:"text"`
} `json:"lines"`
} `json:"pages"`
}
// Recognize implements ocr.Model.
func (m *ocrModel) Recognize(ctx context.Context, req ocr.Request, opts ...ocr.Option) (*ocr.Result, error) {
req = req.Apply(opts...)
if len(req.Document) == 0 {
return nil, fmt.Errorf("%w: ocr requires document bytes", llm.ErrUnsupported)
}
if req.MaxPages < 0 {
return nil, fmt.Errorf("%w: ocr max pages must be >= 0, got %d", llm.ErrUnsupported, req.MaxPages)
}
path, err := upstreamPath(m.id, "/v1/ocr")
if err != nil {
return nil, err
}
maxPages := ""
if req.MaxPages != 0 {
maxPages = strconv.Itoa(req.MaxPages)
}
body, contentType, err := buildMultipart("build ocr form",
filePart{field: "file", filename: documentFilename(req.Filename, req.MIME), data: req.Document},
[]formField{
{"langs", strings.Join(req.Languages, ","), false},
{"max_pages", maxPages, false},
})
if err != nil {
return nil, err
}
raw, _, err := m.p.doRaw(ctx, http.MethodPost, path, m.id, contentType, body, maxResponseBytes)
if err != nil {
return nil, err
}
var out ocrResponse
if err := json.Unmarshal(raw, &out); err != nil {
return nil, fmt.Errorf("llama-swap: decode ocr response: %w", err)
}
if len(out.Pages) == 0 {
// A blank page still comes back as a page with empty text; zero pages
// is API drift or a soft error, never "the document was empty".
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: "ocr response contained no pages: " + truncateForError(raw)}
}
res := &ocr.Result{Raw: json.RawMessage(raw)}
texts := make([]string, 0, len(out.Pages))
for i, pg := range out.Pages {
text := pg.Text
if text == "" && len(pg.Lines) > 0 {
lines := make([]string, 0, len(pg.Lines))
for _, ln := range pg.Lines {
lines = append(lines, ln.Text)
}
text = strings.Join(lines, "\n")
}
number := pg.Number
if number == 0 {
number = i + 1
}
res.Pages = append(res.Pages, ocr.Page{Number: number, Text: text})
texts = append(texts, text)
}
res.Text = strings.Join(texts, "\n\n")
return res, nil
}
// documentFilename picks the multipart filename hint for an OCR document: the
// caller's (sanitized), else one derived from the MIME subtype
// ("document.pdf"), else "document". MIME parameters are stripped before
// matching, mirroring transcriptionFilename.
func documentFilename(filename, mimeType string) string {
if name := sanitizeFilename(filename); name != "" {
return name
}
mt := strings.ToLower(strings.TrimSpace(mimeType))
if parsed, _, err := mime.ParseMediaType(mt); err == nil {
mt = parsed
}
switch mt {
case "application/pdf":
return "document.pdf"
case "image/png":
return "document.png"
case "image/jpeg", "image/jpg":
return "document.jpg"
case "image/webp":
return "document.webp"
default:
return "document"
}
}
+200
View File
@@ -0,0 +1,200 @@
package llamaswap
import (
"context"
"errors"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/majordomo/ocr"
)
func TestRecognize(t *testing.T) {
var gotPath, gotLangs, gotMaxPages, gotFilename string
var gotFile []byte
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Fatalf("parse multipart: %v", err)
}
gotLangs = r.FormValue("langs")
gotMaxPages = r.FormValue("max_pages")
f, hdr, err := r.FormFile("file")
if err != nil {
t.Fatalf("form file: %v", err)
}
defer f.Close()
gotFile, _ = io.ReadAll(f)
gotFilename = hdr.Filename
_, _ = w.Write([]byte(`{"pages":[
{"number":1,"text":"page one","lines":[{"text":"page","bbox":[0,0,1,1],"confidence":0.9},{"text":"one"}],"layout":{}},
{"number":2,"text":"page two"}
]}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
om, err := p.OCRModel("ocr-surya")
if err != nil {
t.Fatalf("OCRModel: %v", err)
}
res, err := om.Recognize(context.Background(),
ocr.Request{Document: []byte("%PDF"), MIME: "application/pdf"},
ocr.WithLanguages("en", "de"), ocr.WithMaxPages(5))
if err != nil {
t.Fatalf("Recognize: %v", err)
}
if gotPath != "/upstream/ocr-surya/v1/ocr" {
t.Errorf("path = %q", gotPath)
}
if gotLangs != "en,de" || gotMaxPages != "5" {
t.Errorf("langs/max_pages = %q/%q", gotLangs, gotMaxPages)
}
if string(gotFile) != "%PDF" || gotFilename != "document.pdf" {
t.Errorf("file = %q name = %q", gotFile, gotFilename)
}
if len(res.Pages) != 2 || res.Pages[0].Number != 1 || res.Pages[0].Text != "page one" ||
res.Pages[1].Number != 2 || res.Pages[1].Text != "page two" {
t.Errorf("pages = %+v", res.Pages)
}
if res.Text != "page one\n\npage two" {
t.Errorf("text = %q", res.Text)
}
if res.Raw == nil {
t.Error("Raw = nil, want raw payload")
}
}
func TestRecognizeJoinsLinesWhenPageTextAbsent(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{"pages":[{"lines":[{"text":"first line"},{"text":"second line"}]}]}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
om, _ := p.OCRModel("ocr-surya")
res, err := om.Recognize(context.Background(), ocr.Request{Document: []byte("img")})
if err != nil {
t.Fatalf("Recognize: %v", err)
}
if len(res.Pages) != 1 || res.Pages[0].Text != "first line\nsecond line" {
t.Errorf("pages = %+v", res.Pages)
}
// Missing page number defaults to position.
if res.Pages[0].Number != 1 {
t.Errorf("number = %d, want 1", res.Pages[0].Number)
}
if res.Text != "first line\nsecond line" {
t.Errorf("text = %q", res.Text)
}
}
func TestRecognizeOmitsUnsetFields(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Fatalf("parse multipart: %v", err)
}
for _, k := range []string{"langs", "max_pages"} {
if v, ok := r.MultipartForm.Value[k]; ok {
t.Errorf("unset request sent %q = %v, want omitted", k, v)
}
}
if _, hdr, err := r.FormFile("file"); err == nil {
if hdr.Filename != "document" {
t.Errorf("filename = %q, want document fallback", hdr.Filename)
}
}
_, _ = w.Write([]byte(`{"pages":[{"number":1,"text":"x"}]}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
om, _ := p.OCRModel("ocr-surya")
if _, err := om.Recognize(context.Background(), ocr.Request{Document: []byte("img")}); err != nil {
t.Fatalf("Recognize: %v", err)
}
}
func TestRecognizeRejectsBadArgs(t *testing.T) {
p := New(WithBaseURL("http://unused"))
om, _ := p.OCRModel("ocr-surya")
if _, err := om.Recognize(context.Background(), ocr.Request{}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("no document: err = %v, want ErrUnsupported", err)
}
if _, err := om.Recognize(context.Background(),
ocr.Request{Document: []byte{1}, MaxPages: -1}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("negative max pages: err = %v, want ErrUnsupported", err)
}
}
func TestRecognizeRejectsZeroPages(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{"pages":[]}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
om, _ := p.OCRModel("ocr-surya")
_, err := om.Recognize(context.Background(), ocr.Request{Document: []byte("img")})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want APIError for zero pages", err)
}
}
func TestRecognizeRejectsNonJSONResponse(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/html")
_, _ = w.Write([]byte("<html>proxy error page</html>"))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
om, _ := p.OCRModel("ocr-surya")
if _, err := om.Recognize(context.Background(), ocr.Request{Document: []byte("img")}); err == nil ||
!strings.Contains(err.Error(), "decode ocr response") {
t.Fatalf("err = %v, want decode error for non-JSON body", err)
}
}
func TestRecognizeSurfacesAPIError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusBadRequest)
_, _ = w.Write([]byte(`{"error":{"message":"unsupported file type"}}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
om, _ := p.OCRModel("ocr-surya")
_, err := om.Recognize(context.Background(), ocr.Request{Document: []byte("bad")})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %T %v, want *llm.APIError", err, err)
}
if apiErr.Status != http.StatusBadRequest || apiErr.Message != "unsupported file type" || apiErr.Model != "ocr-surya" {
t.Errorf("apiErr = %+v", apiErr)
}
}
func TestDocumentFilename(t *testing.T) {
cases := []struct {
filename, mime, want string
}{
{"scan.pdf", "", "scan.pdf"},
{"evil\r\nname.pdf", "", "evilname.pdf"},
{"", "application/pdf", "document.pdf"},
{"", "image/png", "document.png"},
{"", "image/jpeg", "document.jpg"},
{"", "image/webp; charset=binary", "document.webp"},
{"", "", "document"},
}
for _, tc := range cases {
if got := documentFilename(tc.filename, tc.mime); got != tc.want {
t.Errorf("documentFilename(%q, %q) = %q, want %q", tc.filename, tc.mime, got, tc.want)
}
}
}
+106
View File
@@ -0,0 +1,106 @@
// restore.go implements the imagegen.ColorizeProvider and
// imagegen.FaceRestoreProvider surfaces against the mediautils shim reached
// through llama-swap's /upstream passthrough (ADR-0023):
//
// colorize POST /upstream/<id>/v1/colorize (DDColor)
// restore_faces POST /upstream/<id>/v1/restore_faces (GFPGAN)
//
// Both are one-file multipart in, one PNG out, mirroring mediautil.go.
package llamaswap
import (
"context"
"fmt"
"net/http"
"strconv"
"gitea.stevedudenhoeffer.com/steve/majordomo/imagegen"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
// --- colorize ---
// ColorizeModel implements imagegen.ColorizeProvider against the mediautils
// shim's POST /v1/colorize. The id selects which upstream llama-swap loads.
func (p *Provider) ColorizeModel(id string, opts ...imagegen.ColorizeModelOption) (imagegen.Colorizer, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
_ = imagegen.ApplyColorizeModelOptions(opts)
return &colorizeModel{p: p, id: id}, nil
}
type colorizeModel struct {
p *Provider
id string
}
// Colorize implements imagegen.Colorizer.
func (m *colorizeModel) Colorize(ctx context.Context, req imagegen.ColorizeRequest, opts ...imagegen.ColorizeOption) (*imagegen.Result, error) {
req = req.Apply(opts...)
if len(req.Image.Data) == 0 {
return nil, fmt.Errorf("%w: colorization requires an image", llm.ErrUnsupported)
}
path, err := upstreamPath(m.id, "/v1/colorize")
if err != nil {
return nil, err
}
body, contentType, err := buildMultipart("build colorize form",
filePart{field: "file", filename: "image.png", data: req.Image.Data},
nil)
if err != nil {
return nil, err
}
raw, respType, err := m.p.doRaw(ctx, http.MethodPost, path, m.id, contentType, body, maxImageResponseBytes)
if err != nil {
return nil, err
}
return singleImageResult(m.p.name, m.id, "colorize", raw, respType)
}
// --- face restoration ---
// FaceRestoreModel implements imagegen.FaceRestoreProvider against the
// mediautils shim's POST /v1/restore_faces.
func (p *Provider) FaceRestoreModel(id string, opts ...imagegen.FaceRestoreModelOption) (imagegen.FaceRestorer, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
_ = imagegen.ApplyFaceRestoreModelOptions(opts)
return &faceRestoreModel{p: p, id: id}, nil
}
type faceRestoreModel struct {
p *Provider
id string
}
// RestoreFaces implements imagegen.FaceRestorer.
func (m *faceRestoreModel) RestoreFaces(ctx context.Context, req imagegen.FaceRestoreRequest, opts ...imagegen.FaceRestoreOption) (*imagegen.Result, error) {
req = req.Apply(opts...)
if len(req.Image.Data) == 0 {
return nil, fmt.Errorf("%w: face restoration requires an image", llm.ErrUnsupported)
}
if req.Upscale != 0 && req.Upscale != 1 && req.Upscale != 2 {
return nil, fmt.Errorf("%w: face-restore upscale must be 1 or 2, got %d", llm.ErrUnsupported, req.Upscale)
}
path, err := upstreamPath(m.id, "/v1/restore_faces")
if err != nil {
return nil, err
}
upscale := ""
if req.Upscale != 0 {
upscale = strconv.Itoa(req.Upscale)
}
body, contentType, err := buildMultipart("build restore-faces form",
filePart{field: "file", filename: "image.png", data: req.Image.Data},
[]formField{{"upscale", upscale, false}})
if err != nil {
return nil, err
}
raw, respType, err := m.p.doRaw(ctx, http.MethodPost, path, m.id, contentType, body, maxImageResponseBytes)
if err != nil {
return nil, err
}
return singleImageResult(m.p.name, m.id, "face restoration", raw, respType)
}
+79
View File
@@ -0,0 +1,79 @@
// segment.go implements imagegen.SegmentationProvider against a
// GroundingDINO+SAM shim (segment-langsam) reached through llama-swap's
// /upstream passthrough (ADR-0023):
//
// POST /upstream/<id>/v1/segment multipart file,prompt[,threshold],output=mask
//
// The response is a single grayscale mask PNG where WHITE marks the prompted
// region — directly usable as imagegen.EditRequest.Mask (white = repaint).
// The shim also offers output=cutout|boxes; this client always requests the
// mask, because a cutout is derivable client-side from mask+original with no
// second GPU call.
package llamaswap
import (
"context"
"fmt"
"math"
"net/http"
"strconv"
"gitea.stevedudenhoeffer.com/steve/majordomo/imagegen"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
// SegmentationModel implements imagegen.SegmentationProvider. The id selects
// which upstream llama-swap loads.
func (p *Provider) SegmentationModel(id string, opts ...imagegen.SegmentationModelOption) (imagegen.Segmenter, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
_ = imagegen.ApplySegmentationModelOptions(opts)
return &segmentationModel{p: p, id: id}, nil
}
type segmentationModel struct {
p *Provider
id string
}
// Segment implements imagegen.Segmenter.
func (m *segmentationModel) Segment(ctx context.Context, req imagegen.SegmentationRequest, opts ...imagegen.SegmentationOption) (*imagegen.Result, error) {
req = req.Apply(opts...)
if len(req.Image.Data) == 0 {
return nil, fmt.Errorf("%w: segmentation requires an image", llm.ErrUnsupported)
}
if req.Prompt == "" {
return nil, fmt.Errorf("%w: segmentation requires a prompt", llm.ErrUnsupported)
}
// NaN fails every comparison, so it would sail through a bare range
// check and reach the upstream as the literal string "NaN".
if math.IsNaN(req.Threshold) || req.Threshold < 0 || req.Threshold > 1 {
return nil, fmt.Errorf("%w: segmentation threshold must be in [0,1], got %g", llm.ErrUnsupported, req.Threshold)
}
path, err := upstreamPath(m.id, "/v1/segment")
if err != nil {
return nil, err
}
threshold := ""
if req.Threshold != 0 {
threshold = strconv.FormatFloat(req.Threshold, 'g', -1, 64)
}
body, contentType, err := buildMultipart("build segment form",
filePart{field: "file", filename: "image.png", data: req.Image.Data},
[]formField{
{"prompt", req.Prompt, true},
{"threshold", threshold, false},
// Always the mask: white = prompted region, EditRequest.Mask
// polarity. Cutouts are derived client-side.
{"output", "mask", true},
})
if err != nil {
return nil, err
}
raw, respType, err := m.p.doRaw(ctx, http.MethodPost, path, m.id, contentType, body, maxImageResponseBytes)
if err != nil {
return nil, err
}
return singleImageResult(m.p.name, m.id, "segmentation", raw, respType)
}
+281
View File
@@ -0,0 +1,281 @@
package llamaswap
import (
"context"
"errors"
"math"
"net/http"
"net/http/httptest"
"testing"
"gitea.stevedudenhoeffer.com/steve/majordomo/imagegen"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
func TestSegment(t *testing.T) {
png := pngFixture(t)
var gotPath, gotPrompt, gotThreshold, gotOutput, gotFilename string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Errorf("parse multipart: %v", err)
}
gotPrompt = r.FormValue("prompt")
gotThreshold = r.FormValue("threshold")
gotOutput = r.FormValue("output")
if f, hdr, err := r.FormFile("file"); err == nil {
gotFilename = hdr.Filename
f.Close()
} else {
t.Errorf("file part: %v", err)
}
w.Header().Set("Content-Type", "image/png")
_, _ = w.Write(png)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
sg, err := p.SegmentationModel("segment-langsam")
if err != nil {
t.Fatalf("SegmentationModel: %v", err)
}
res, err := sg.Segment(context.Background(),
imagegen.SegmentationRequest{Image: imagegen.Image{MIME: "image/png", Data: png}, Prompt: "the red car"},
imagegen.WithSegmentationThreshold(0.35))
if err != nil {
t.Fatalf("Segment: %v", err)
}
if gotPath != "/upstream/segment-langsam/v1/segment" {
t.Errorf("path = %q", gotPath)
}
if gotPrompt != "the red car" || gotThreshold != "0.35" || gotOutput != "mask" || gotFilename != "image.png" {
t.Errorf("prompt/threshold/output/filename = %q/%q/%q/%q", gotPrompt, gotThreshold, gotOutput, gotFilename)
}
if len(res.Images) != 1 || res.Images[0].MIME != "image/png" {
t.Fatalf("images = %+v", res.Images)
}
}
func TestSegmentOmitsDefaultThreshold(t *testing.T) {
png := pngFixture(t)
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Errorf("parse multipart: %v", err)
}
if _, ok := r.MultipartForm.Value["threshold"]; ok {
t.Error("threshold field sent for default; want omitted")
}
if got := r.FormValue("output"); got != "mask" {
t.Errorf("output = %q, want mask (always sent)", got)
}
w.Header().Set("Content-Type", "image/png")
_, _ = w.Write(png)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
sg, _ := p.SegmentationModel("segment-langsam")
if _, err := sg.Segment(context.Background(),
imagegen.SegmentationRequest{Image: imagegen.Image{Data: png}, Prompt: "dog"}); err != nil {
t.Fatalf("Segment: %v", err)
}
}
func TestSegmentRejectsBadArgs(t *testing.T) {
p := New(WithBaseURL("http://unused"))
sg, _ := p.SegmentationModel("segment-langsam")
if _, err := sg.Segment(context.Background(),
imagegen.SegmentationRequest{Prompt: "dog"}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("no image: err = %v, want ErrUnsupported", err)
}
if _, err := sg.Segment(context.Background(),
imagegen.SegmentationRequest{Image: imagegen.Image{Data: []byte{1}}}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("no prompt: err = %v, want ErrUnsupported", err)
}
if _, err := sg.Segment(context.Background(),
imagegen.SegmentationRequest{Image: imagegen.Image{Data: []byte{1}}, Prompt: "dog", Threshold: 1.5}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("threshold 1.5: err = %v, want ErrUnsupported", err)
}
for name, bad := range map[string]float64{
"NaN": math.NaN(),
"+Inf": math.Inf(1),
"-Inf": math.Inf(-1),
} {
if _, err := sg.Segment(context.Background(),
imagegen.SegmentationRequest{Image: imagegen.Image{Data: []byte{1}}, Prompt: "dog", Threshold: bad}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("threshold %s: err = %v, want ErrUnsupported", name, err)
}
}
}
func TestSegmentRejectsNonImageResponse(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/html")
_, _ = w.Write([]byte("<html>proxy error page</html>"))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
sg, _ := p.SegmentationModel("segment-langsam")
_, err := sg.Segment(context.Background(),
imagegen.SegmentationRequest{Image: imagegen.Image{Data: pngFixture(t)}, Prompt: "dog"})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want APIError for non-image body", err)
}
}
func TestColorize(t *testing.T) {
png := pngFixture(t)
var gotPath, gotFilename string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Errorf("parse multipart: %v", err)
}
if f, hdr, err := r.FormFile("file"); err == nil {
gotFilename = hdr.Filename
f.Close()
} else {
t.Errorf("file part: %v", err)
}
w.Header().Set("Content-Type", "image/png")
_, _ = w.Write(png)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
cl, err := p.ColorizeModel("mediautils")
if err != nil {
t.Fatalf("ColorizeModel: %v", err)
}
res, err := cl.Colorize(context.Background(),
imagegen.ColorizeRequest{Image: imagegen.Image{Data: png}})
if err != nil {
t.Fatalf("Colorize: %v", err)
}
if gotPath != "/upstream/mediautils/v1/colorize" {
t.Errorf("path = %q", gotPath)
}
if gotFilename != "image.png" {
t.Errorf("filename = %q", gotFilename)
}
if len(res.Images) != 1 || res.Images[0].MIME != "image/png" {
t.Fatalf("images = %+v", res.Images)
}
}
func TestColorizeRejectsEmptyImage(t *testing.T) {
p := New(WithBaseURL("http://unused"))
cl, _ := p.ColorizeModel("mediautils")
if _, err := cl.Colorize(context.Background(), imagegen.ColorizeRequest{}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("err = %v, want ErrUnsupported", err)
}
}
func TestColorizeSurfacesAPIError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusServiceUnavailable)
_, _ = w.Write([]byte(`{"error":{"message":"model is loading"}}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
cl, _ := p.ColorizeModel("mediautils")
_, err := cl.Colorize(context.Background(),
imagegen.ColorizeRequest{Image: imagegen.Image{Data: pngFixture(t)}})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %T %v, want *llm.APIError", err, err)
}
if apiErr.Status != http.StatusServiceUnavailable || apiErr.Message != "model is loading" || apiErr.Model != "mediautils" {
t.Errorf("apiErr = %+v", apiErr)
}
}
func TestRestoreFaces(t *testing.T) {
png := pngFixture(t)
var gotPath, gotUpscale string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Errorf("parse multipart: %v", err)
}
gotUpscale = r.FormValue("upscale")
w.Header().Set("Content-Type", "image/png")
_, _ = w.Write(png)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
fr, err := p.FaceRestoreModel("mediautils")
if err != nil {
t.Fatalf("FaceRestoreModel: %v", err)
}
res, err := fr.RestoreFaces(context.Background(),
imagegen.FaceRestoreRequest{Image: imagegen.Image{Data: png}},
imagegen.WithFaceRestoreUpscale(2))
if err != nil {
t.Fatalf("RestoreFaces: %v", err)
}
if gotPath != "/upstream/mediautils/v1/restore_faces" {
t.Errorf("path = %q", gotPath)
}
if gotUpscale != "2" {
t.Errorf("upscale = %q", gotUpscale)
}
if len(res.Images) != 1 {
t.Fatalf("images = %+v", res.Images)
}
}
func TestRestoreFacesOmitsDefaultUpscale(t *testing.T) {
png := pngFixture(t)
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Errorf("parse multipart: %v", err)
}
if _, ok := r.MultipartForm.Value["upscale"]; ok {
t.Error("upscale field sent for default; want omitted")
}
w.Header().Set("Content-Type", "image/png")
_, _ = w.Write(png)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
fr, _ := p.FaceRestoreModel("mediautils")
if _, err := fr.RestoreFaces(context.Background(),
imagegen.FaceRestoreRequest{Image: imagegen.Image{Data: png}}); err != nil {
t.Fatalf("RestoreFaces: %v", err)
}
}
func TestRestoreFacesRejectsBadArgs(t *testing.T) {
p := New(WithBaseURL("http://unused"))
fr, _ := p.FaceRestoreModel("mediautils")
if _, err := fr.RestoreFaces(context.Background(), imagegen.FaceRestoreRequest{}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("no image: err = %v, want ErrUnsupported", err)
}
if _, err := fr.RestoreFaces(context.Background(),
imagegen.FaceRestoreRequest{Image: imagegen.Image{Data: []byte{1}}, Upscale: 3}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("upscale 3: err = %v, want ErrUnsupported", err)
}
}
func TestRestoreFacesRejectsNonImageResponse(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header()["Content-Type"] = nil // NO Content-Type at all
_, _ = w.Write([]byte("502 bad gateway"))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
fr, _ := p.FaceRestoreModel("mediautils")
_, err := fr.RestoreFaces(context.Background(),
imagegen.FaceRestoreRequest{Image: imagegen.Image{Data: pngFixture(t)}})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want APIError for headerless non-image body", err)
}
}
+100
View File
@@ -0,0 +1,100 @@
// sfx.go implements a second musicgen.Model factory against a Stable Audio
// Open shim (sfxgen) reached through llama-swap's /upstream passthrough
// (ADR-0024):
//
// POST /upstream/<id>/v1/sfx JSON {prompt, seconds?, steps?, cfg_scale?, seed?}
//
// Unlike the ACE-Step music path (music.go), /v1/sfx is SYNCHRONOUS: the
// response body is the finished WAV clip — no job queue, no polling. The
// surface reuses the musicgen types (a sound effect is a short audio clip
// from a text prompt); only the provider method differs.
package llamaswap
import (
"bytes"
"context"
"encoding/json"
"fmt"
"net/http"
"strings"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/majordomo/musicgen"
)
// SFXModel returns a musicgen.Model bound to a sync sound-effect backend.
// The id selects which upstream llama-swap loads.
func (p *Provider) SFXModel(id string, opts ...musicgen.ModelOption) (musicgen.Model, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
_ = musicgen.ApplyModelOptions(opts)
return &sfxModel{p: p, id: id}, nil
}
type sfxModel struct {
p *Provider
id string
}
// sfxRequest is the sfxgen shim's /v1/sfx shape. Optional fields stay off
// the wire so the model's own defaults apply.
type sfxRequest struct {
Prompt string `json:"prompt"`
Seconds int `json:"seconds,omitempty"`
Steps *int `json:"steps,omitempty"`
CFGScale *float64 `json:"cfg_scale,omitempty"`
Seed *int64 `json:"seed,omitempty"`
}
// Generate implements musicgen.Model. The clip-length ceiling (~11s for
// Stable Audio Open Small) is the backend's to enforce, not this client's.
func (m *sfxModel) Generate(ctx context.Context, req musicgen.Request, opts ...musicgen.Option) (*musicgen.Result, error) {
req = req.Apply(opts...)
if strings.TrimSpace(req.Prompt) == "" {
return nil, fmt.Errorf("%w: sfx generation requires a prompt", llm.ErrUnsupported)
}
if req.Lyrics != "" {
return nil, fmt.Errorf("%w: sfx generation does not support lyrics", llm.ErrUnsupported)
}
if req.DurationSeconds < 0 {
return nil, fmt.Errorf("%w: duration must be >= 0, got %d", llm.ErrUnsupported, req.DurationSeconds)
}
if req.Steps != nil && *req.Steps <= 0 {
return nil, fmt.Errorf("%w: inference steps must be > 0, got %d", llm.ErrUnsupported, *req.Steps)
}
// The endpoint emits WAV only; a caller asking for another container
// would silently get mislabelled bytes — reject instead.
if f := strings.ToLower(strings.TrimSpace(req.Format)); f != "" && f != "wav" {
return nil, fmt.Errorf("%w: sfx output is wav only, got format %q", llm.ErrUnsupported, req.Format)
}
path, err := upstreamPath(m.id, "/v1/sfx")
if err != nil {
return nil, err
}
wire := sfxRequest{
Prompt: req.Prompt,
Seconds: req.DurationSeconds,
Steps: req.Steps,
CFGScale: req.CFGScale,
Seed: req.Seed,
}
encoded, err := json.Marshal(wire)
if err != nil {
return nil, fmt.Errorf("llama-swap: encode sfx request: %w", err)
}
raw, contentType, err := m.p.doRaw(ctx, http.MethodPost, path, m.id, "application/json", bytes.NewReader(encoded), maxAudioResponseBytes)
if err != nil {
return nil, err
}
if len(raw) == 0 {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id, Message: "sfx response contained no audio"}
}
mimeType := audioResultMIME(contentType, raw)
if mimeType == "" {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: fmt.Sprintf("sfx response is not audio (Content-Type %q): %s", contentType, truncateForError(raw))}
}
return &musicgen.Result{Audio: musicgen.Audio{Data: raw, MIME: mimeType}}, nil
}
+165
View File
@@ -0,0 +1,165 @@
// stems.go implements audio.StemSeparationProvider against a Demucs shim
// (audioutils) reached through llama-swap's /upstream passthrough (ADR-0024):
//
// POST /upstream/<id>/v1/stems multipart file[,model,two_stems,format]
//
// The response is a ZIP of the separated stems — one entry per stem, entry
// name = stem name, extension = container. Zip transport keeps a 4-stem WAV
// result (hundreds of MB decoded) off the JSON-of-base64 path entirely.
package llamaswap
import (
"archive/zip"
"bytes"
"context"
"fmt"
"io"
"net/http"
"path"
"strings"
"gitea.stevedudenhoeffer.com/steve/majordomo/audio"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
// maxStemsResponseBytes caps the /v1/stems zip body: four WAV stems of a
// long song legitimately pass the 64MB JSON cap. Bounded so a buggy
// upstream can't allocate without limit.
const maxStemsResponseBytes = 512 << 20
// maxStemEntryBytes caps ONE decompressed zip entry — the zip-bomb guard.
// A single WAV stem of even a very long song stays far under this.
const maxStemEntryBytes = 256 << 20
// maxStemEntries caps how many stem entries are unpacked: Demucs emits at
// most six, so anything past a small multiple of that is a hostile or
// broken archive, not a result.
const maxStemEntries = 16
// maxStemsTotalBytes caps the AGGREGATE decompressed size across entries —
// the per-entry bound alone would still let a many-entry bomb multiply up.
const maxStemsTotalBytes = 1 << 30
// StemSeparatorModel implements audio.StemSeparationProvider. The id selects
// which upstream llama-swap loads.
func (p *Provider) StemSeparatorModel(id string, opts ...audio.StemSeparatorModelOption) (audio.StemSeparator, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
_ = audio.ApplyStemSeparatorModelOptions(opts)
return &stemSeparatorModel{p: p, id: id}, nil
}
type stemSeparatorModel struct {
p *Provider
id string
}
// SeparateStems implements audio.StemSeparator.
func (m *stemSeparatorModel) SeparateStems(ctx context.Context, req audio.StemSeparationRequest, opts ...audio.StemSeparationOption) (*audio.StemSeparationResult, error) {
req = req.Apply(opts...)
if len(req.Audio) == 0 {
return nil, fmt.Errorf("%w: stem separation requires audio bytes", llm.ErrUnsupported)
}
mode := strings.ToLower(strings.TrimSpace(req.Mode))
if mode != "" && mode != "two" && mode != "four" {
return nil, fmt.Errorf("%w: stem mode must be \"two\" or \"four\", got %q", llm.ErrUnsupported, req.Mode)
}
format := strings.ToLower(strings.TrimSpace(req.Format))
if format != "" && format != "mp3" && format != "wav" {
return nil, fmt.Errorf("%w: stem format must be \"mp3\" or \"wav\", got %q", llm.ErrUnsupported, req.Format)
}
upPath, err := upstreamPath(m.id, "/v1/stems")
if err != nil {
return nil, err
}
// Demucs' two-stem mode is "isolate one source vs the rest"; vocals is
// the split this surface promises ("two" = vocals + accompaniment).
twoStems := ""
if mode == "two" {
twoStems = "vocals"
}
body, contentType, err := buildMultipart("build stems form",
filePart{field: "file", filename: transcriptionFilename(req.Filename, req.MIME), data: req.Audio},
[]formField{
{"model", req.Model, false},
{"two_stems", twoStems, false},
{"format", format, false},
})
if err != nil {
return nil, err
}
raw, respType, err := m.p.doRaw(ctx, http.MethodPost, upPath, m.id, contentType, body, maxStemsResponseBytes)
if err != nil {
return nil, err
}
if len(raw) == 0 {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id, Message: "stems response contained no data"}
}
zr, err := zip.NewReader(bytes.NewReader(raw), int64(len(raw)))
if err != nil {
// A non-zip 2xx body is a misconfigured upstream (an HTML error page
// behind a proxy, a JSON soft error) — fail loud, quoting a slice.
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: fmt.Sprintf("stems response is not a zip (Content-Type %q): %s", respType, truncateForError(raw))}
}
res := &audio.StemSeparationResult{}
var totalBytes int64
for _, f := range zr.File {
if f.FileInfo().IsDir() {
continue
}
if len(res.Stems) >= maxStemEntries {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: fmt.Sprintf("stems zip holds more than %d entries", maxStemEntries)}
}
data, err := readZipEntry(f)
if err != nil {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: fmt.Sprintf("stems zip entry %q: %v", f.Name, err)}
}
totalBytes += int64(len(data))
if totalBytes > maxStemsTotalBytes {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: fmt.Sprintf("stems zip decompresses past %d bytes", int64(maxStemsTotalBytes))}
}
// Entry name → stem name; extension → MIME. Entries may sit in a
// per-model directory ("htdemucs/vocals.mp3"), so use the base name.
base := path.Base(f.Name)
ext := path.Ext(base)
res.Stems = append(res.Stems, audio.Stem{
Name: strings.TrimSuffix(base, ext),
Audio: data,
MIME: stemMIME(ext),
})
}
if len(res.Stems) == 0 {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id, Message: "stems zip contained no stems"}
}
return res, nil
}
// readZipEntry decompresses one entry, bounded by maxStemEntryBytes so a
// zip bomb can't allocate without limit.
func readZipEntry(f *zip.File) ([]byte, error) {
rc, err := f.Open()
if err != nil {
return nil, err
}
defer rc.Close()
data, err := io.ReadAll(io.LimitReader(rc, maxStemEntryBytes+1))
if err != nil {
return nil, err
}
if int64(len(data)) > maxStemEntryBytes {
return nil, fmt.Errorf("decompressed entry exceeds %d bytes", int64(maxStemEntryBytes))
}
return data, nil
}
// stemMIME maps a stem file extension to its MIME type, reusing speechMIME's
// format table ("" and unknown extensions land on the mp3 default — Demucs'
// own default container).
func stemMIME(ext string) string {
return speechMIME("", strings.TrimPrefix(strings.ToLower(ext), "."))
}
@@ -0,0 +1,355 @@
package llamaswap
import (
"archive/zip"
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"net/http/httptest"
"strings"
"testing"
"gitea.stevedudenhoeffer.com/steve/majordomo/audio"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/majordomo/musicgen"
)
// stemsZipFixture builds a Demucs-style stems zip: entries under a per-model
// directory, entry name = stem name, extension = container.
func stemsZipFixture(t *testing.T, entries map[string]string) []byte {
t.Helper()
var buf bytes.Buffer
zw := zip.NewWriter(&buf)
for name, data := range entries {
w, err := zw.Create(name)
if err != nil {
t.Fatalf("zip create: %v", err)
}
if _, err := w.Write([]byte(data)); err != nil {
t.Fatalf("zip write: %v", err)
}
}
if err := zw.Close(); err != nil {
t.Fatalf("zip close: %v", err)
}
return buf.Bytes()
}
func TestSeparateStems(t *testing.T) {
zipBody := stemsZipFixture(t, map[string]string{
"htdemucs/vocals.mp3": "VOX",
"htdemucs/no_vocals.mp3": "ACC",
})
var gotPath, gotTwoStems, gotModel, gotFormat, gotFilename string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Fatalf("parse multipart: %v", err)
}
gotTwoStems = r.FormValue("two_stems")
gotModel = r.FormValue("model")
gotFormat = r.FormValue("format")
if _, hdr, err := r.FormFile("file"); err == nil {
gotFilename = hdr.Filename
} else {
t.Errorf("file part: %v", err)
}
w.Header().Set("Content-Type", "application/zip")
_, _ = w.Write(zipBody)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
ss, err := p.StemSeparatorModel("audioutils")
if err != nil {
t.Fatalf("StemSeparatorModel: %v", err)
}
res, err := ss.SeparateStems(context.Background(),
audio.StemSeparationRequest{Audio: []byte("SONG"), MIME: "audio/mpeg"},
audio.WithStemMode("two"), audio.WithStemModel("htdemucs_ft"), audio.WithStemFormat("mp3"))
if err != nil {
t.Fatalf("SeparateStems: %v", err)
}
if gotPath != "/upstream/audioutils/v1/stems" {
t.Errorf("path = %q", gotPath)
}
if gotTwoStems != "vocals" || gotModel != "htdemucs_ft" || gotFormat != "mp3" || gotFilename != "audio.mp3" {
t.Errorf("two_stems/model/format/filename = %q/%q/%q/%q", gotTwoStems, gotModel, gotFormat, gotFilename)
}
if len(res.Stems) != 2 {
t.Fatalf("stems = %+v", res.Stems)
}
byName := map[string]audio.Stem{}
for _, s := range res.Stems {
byName[s.Name] = s
}
if v := byName["vocals"]; string(v.Audio) != "VOX" || v.MIME != "audio/mpeg" {
t.Errorf("vocals = %+v", v)
}
if a := byName["no_vocals"]; string(a.Audio) != "ACC" || a.MIME != "audio/mpeg" {
t.Errorf("no_vocals = %+v", a)
}
}
func TestSeparateStemsOmitsUnsetFields(t *testing.T) {
zipBody := stemsZipFixture(t, map[string]string{"vocals.wav": "V"})
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Fatalf("parse multipart: %v", err)
}
for _, k := range []string{"two_stems", "model", "format"} {
if v, ok := r.MultipartForm.Value[k]; ok {
t.Errorf("unset request sent %q = %v, want omitted", k, v)
}
}
_, _ = w.Write(zipBody)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
ss, _ := p.StemSeparatorModel("audioutils")
res, err := ss.SeparateStems(context.Background(),
audio.StemSeparationRequest{Audio: []byte("SONG")})
if err != nil {
t.Fatalf("SeparateStems: %v", err)
}
// Top-level entry, wav extension.
if len(res.Stems) != 1 || res.Stems[0].Name != "vocals" || res.Stems[0].MIME != "audio/wav" {
t.Errorf("stems = %+v", res.Stems)
}
}
func TestSeparateStemsRejectsBadArgs(t *testing.T) {
p := New(WithBaseURL("http://unused"))
ss, _ := p.StemSeparatorModel("audioutils")
if _, err := ss.SeparateStems(context.Background(), audio.StemSeparationRequest{}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("no audio: err = %v, want ErrUnsupported", err)
}
if _, err := ss.SeparateStems(context.Background(),
audio.StemSeparationRequest{Audio: []byte{1}, Mode: "three"}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("mode three: err = %v, want ErrUnsupported", err)
}
if _, err := ss.SeparateStems(context.Background(),
audio.StemSeparationRequest{Audio: []byte{1}, Format: "flac"}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("format flac: err = %v, want ErrUnsupported", err)
}
}
func TestSeparateStemsRejectsNonZip(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/html")
_, _ = w.Write([]byte("<html>proxy error page</html>"))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
ss, _ := p.StemSeparatorModel("audioutils")
_, err := ss.SeparateStems(context.Background(), audio.StemSeparationRequest{Audio: []byte("SONG")})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want APIError for non-zip body", err)
}
}
func TestSeparateStemsRejectsEmptyZip(t *testing.T) {
zipBody := stemsZipFixture(t, map[string]string{})
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/zip")
_, _ = w.Write(zipBody)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
ss, _ := p.StemSeparatorModel("audioutils")
_, err := ss.SeparateStems(context.Background(), audio.StemSeparationRequest{Audio: []byte("SONG")})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want APIError for stem-less zip", err)
}
}
func TestSeparateStemsRejectsTooManyEntries(t *testing.T) {
entries := map[string]string{}
for i := 0; i <= maxStemEntries; i++ {
entries[fmt.Sprintf("stem%02d.wav", i)] = "X"
}
zipBody := stemsZipFixture(t, entries)
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/zip")
_, _ = w.Write(zipBody)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
ss, _ := p.StemSeparatorModel("audioutils")
_, err := ss.SeparateStems(context.Background(), audio.StemSeparationRequest{Audio: []byte("SONG")})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) || !strings.Contains(apiErr.Message, "entries") {
t.Fatalf("err = %v, want APIError for over-long stems zip", err)
}
}
// wavFixture is a minimal RIFF/WAVE header so http.DetectContentType sniffs
// audio/wave.
func wavFixture() []byte {
return []byte("RIFF\x24\x00\x00\x00WAVEfmt ")
}
func TestSFXGenerate(t *testing.T) {
var gotPath string
var gotBody map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
_ = json.NewDecoder(r.Body).Decode(&gotBody)
w.Header().Set("Content-Type", "audio/wav")
_, _ = w.Write(wavFixture())
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
sm, err := p.SFXModel("sfxgen-stableaudio")
if err != nil {
t.Fatalf("SFXModel: %v", err)
}
res, err := sm.Generate(context.Background(),
musicgen.Request{Prompt: "glass shattering", DurationSeconds: 8},
musicgen.WithSteps(50), musicgen.WithCFGScale(7), musicgen.WithSeed(42))
if err != nil {
t.Fatalf("Generate: %v", err)
}
if gotPath != "/upstream/sfxgen-stableaudio/v1/sfx" {
t.Errorf("path = %q", gotPath)
}
want := map[string]any{"prompt": "glass shattering", "seconds": 8.0, "steps": 50.0, "cfg_scale": 7.0, "seed": 42.0}
for k, w := range want {
if gotBody[k] != w {
t.Errorf("%s = %v, want %v", k, gotBody[k], w)
}
}
if res.Audio.MIME != "audio/wav" || len(res.Audio.Data) == 0 {
t.Errorf("audio = %q/%d bytes", res.Audio.MIME, len(res.Audio.Data))
}
}
func TestSFXOmitsDefaults(t *testing.T) {
var gotBody map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_ = json.NewDecoder(r.Body).Decode(&gotBody)
// No Content-Type: the RIFF sniff must still label the clip.
w.Header()["Content-Type"] = nil
_, _ = w.Write(wavFixture())
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
sm, _ := p.SFXModel("sfxgen-stableaudio")
res, err := sm.Generate(context.Background(), musicgen.Request{Prompt: "boom"})
if err != nil {
t.Fatalf("Generate: %v", err)
}
for _, k := range []string{"seconds", "steps", "cfg_scale", "seed"} {
if v, ok := gotBody[k]; ok {
t.Errorf("unset request sent %q = %v, want omitted", k, v)
}
}
if res.Audio.MIME != "audio/wav" {
t.Errorf("MIME = %q, want sniffed audio/wav", res.Audio.MIME)
}
}
func TestSFXRejectsBadArgs(t *testing.T) {
p := New(WithBaseURL("http://unused"))
sm, _ := p.SFXModel("sfxgen-stableaudio")
if _, err := sm.Generate(context.Background(), musicgen.Request{Prompt: " "}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("empty prompt: err = %v, want ErrUnsupported", err)
}
if _, err := sm.Generate(context.Background(),
musicgen.Request{Prompt: "boom", Lyrics: "la la"}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("lyrics: err = %v, want ErrUnsupported", err)
}
if _, err := sm.Generate(context.Background(),
musicgen.Request{Prompt: "boom", Format: "mp3"}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("format mp3: err = %v, want ErrUnsupported", err)
}
}
func TestSFXRejectsNonAudioResponse(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"detail":"queue full"}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
sm, _ := p.SFXModel("sfxgen-stableaudio")
_, err := sm.Generate(context.Background(), musicgen.Request{Prompt: "boom"})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want APIError for non-audio body", err)
}
}
func TestEnhance(t *testing.T) {
var gotPath, gotFilename string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
if err := r.ParseMultipartForm(1 << 20); err != nil {
t.Fatalf("parse multipart: %v", err)
}
if _, hdr, err := r.FormFile("file"); err == nil {
gotFilename = hdr.Filename
} else {
t.Errorf("file part: %v", err)
}
w.Header().Set("Content-Type", "audio/wav")
_, _ = w.Write(wavFixture())
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
en, err := p.SpeechEnhancerModel("audioutils")
if err != nil {
t.Fatalf("SpeechEnhancerModel: %v", err)
}
res, err := en.Enhance(context.Background(),
audio.EnhancementRequest{Audio: []byte("NOISY"), MIME: "audio/ogg"})
if err != nil {
t.Fatalf("Enhance: %v", err)
}
if gotPath != "/upstream/audioutils/v1/enhance" {
t.Errorf("path = %q", gotPath)
}
if gotFilename != "audio.ogg" {
t.Errorf("filename = %q", gotFilename)
}
if res.MIME != "audio/wav" || len(res.Audio) == 0 {
t.Errorf("result = %q/%d bytes", res.MIME, len(res.Audio))
}
}
func TestEnhanceRejectsEmptyAudio(t *testing.T) {
p := New(WithBaseURL("http://unused"))
en, _ := p.SpeechEnhancerModel("audioutils")
if _, err := en.Enhance(context.Background(), audio.EnhancementRequest{}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("err = %v, want ErrUnsupported", err)
}
}
func TestEnhanceRejectsNonAudioResponse(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/html")
_, _ = w.Write([]byte("<html>oops</html>"))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
en, _ := p.SpeechEnhancerModel("audioutils")
_, err := en.Enhance(context.Background(), audio.EnhancementRequest{Audio: []byte("NOISY")})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want APIError for non-audio body", err)
}
}
+83
View File
@@ -0,0 +1,83 @@
package llamaswap
import (
"bytes"
"fmt"
"mime/multipart"
"strings"
)
// upstreamPath builds a path through llama-swap's generic /upstream/<model>/
// passthrough, which pins the model (triggering the normal load/swap queue)
// and forwards the remaining path to the upstream verbatim. This is how the
// provider reaches upstreams whose native APIs carry no routable `model`
// field (rembg, mediautils, WhisperX, ACE-Step, ...) without needing a
// llama-swap route per endpoint (ADR-0020).
//
// Why reject rather than escape: same rationale as Unload — model ids
// legitimately contain ":" but never path-structure characters, and escaping
// would mask a config error instead of surfacing it. '%' is rejected too:
// ids never legitimately carry percent-escapes, and %2F/%2E%2E would decode
// back into path structure on the server side.
func upstreamPath(model, rest string) (string, error) {
if strings.TrimSpace(model) == "" {
return "", fmt.Errorf("llama-swap: upstream call requires a model id")
}
if strings.ContainsAny(model, "/?#%") || strings.Contains(model, "..") {
return "", fmt.Errorf("llama-swap: invalid model id %q for upstream call (contains a path separator)", model)
}
if !strings.HasPrefix(rest, "/") {
rest = "/" + rest
}
// rest may embed SERVER-SUPPLIED components (e.g. ACE-Step's result
// file URL) — refuse dot-dot segments and absolute-URL smuggling so a
// hostile/buggy upstream cannot redirect the follow-up request at
// another proxy endpoint (/api/models/unload, ...).
if strings.Contains(rest, "..") || strings.Contains(rest, "://") {
return "", fmt.Errorf("llama-swap: invalid upstream path %q (dot-dot or scheme)", rest)
}
return "/upstream/" + model + rest, nil
}
// filePart is the single file entry of a media multipart form.
type filePart struct {
field string // form field name ("file", "audio_file", ...)
filename string // already sanitized
data []byte
}
// buildMultipart assembles a one-file multipart body: the file part first,
// then the given fields (optional fields skipped when empty, matching
// writeFormFields). wrap labels errors. Returns the body and its content
// type.
func buildMultipart(wrap string, file filePart, fields []formField) (*bytes.Buffer, string, error) {
return buildMultipartFiles(wrap, []filePart{file}, fields)
}
// buildMultipartFiles is buildMultipart for endpoints taking SEVERAL files
// (face swap sends a target and a source). Files are written in the given
// order, then the fields. One writer loop serves both so the two cannot drift
// in how they escape names or terminate the body.
func buildMultipartFiles(wrap string, files []filePart, fields []formField) (*bytes.Buffer, string, error) {
if len(files) == 0 {
return nil, "", fmt.Errorf("llama-swap: %s: no file parts", wrap)
}
var buf bytes.Buffer
w := multipart.NewWriter(&buf)
for _, file := range files {
fw, err := w.CreateFormFile(file.field, file.filename)
if err != nil {
return nil, "", fmt.Errorf("llama-swap: %s: %w", wrap, err)
}
if _, err := fw.Write(file.data); err != nil {
return nil, "", fmt.Errorf("llama-swap: %s: %w", wrap, err)
}
}
if err := writeFormFields(w, wrap, fields); err != nil {
return nil, "", err
}
if err := w.Close(); err != nil {
return nil, "", fmt.Errorf("llama-swap: %s: %w", wrap, err)
}
return &buf, w.FormDataContentType(), nil
}
+204
View File
@@ -0,0 +1,204 @@
package llamaswap
import (
"bytes"
"context"
"fmt"
"mime/multipart"
"net/http"
"strconv"
"strings"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/majordomo/videogen"
)
// VideoModel implements videogen.Provider, binding a video-generation model
// served by llama-swap (routed to a vLLM-Omni-style upstream, or any shim
// exposing the same /v1/videos/sync shape). The id is passed through verbatim
// and selects which upstream llama-swap loads.
func (p *Provider) VideoModel(id string, opts ...videogen.ModelOption) (videogen.Model, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
_ = videogen.ApplyModelOptions(opts)
return &videoModel{p: p, id: id}, nil
}
type videoModel struct {
p *Provider
id string
}
// Generate implements videogen.Model via POST {base}/v1/videos/sync
// (multipart/form-data — llama-swap routes by the `model` form field). The
// blocking sync endpoint answers with the encoded video itself, so the
// response body is the result; there is no job id to poll. Generation runs
// for minutes — the provider client carries no timeout by design, and callers
// bound the call with a context deadline.
//
// Parameter names follow vLLM-Omni's videos API (num_frames, fps,
// num_inference_steps, guidance_scale); the leading conditioning frame is sent
// as an `input_reference` file part, following OpenAI's videos API, and a
// trailing keyframe (Request.LastImage) as `input_reference_last`. Upstreams
// ignore fields they don't understand, and optional fields stay off the wire
// entirely so the model's own defaults apply — which is also why a backend
// without first-last-frame support returns an ordinary clip here rather than
// an error.
func (m *videoModel) Generate(ctx context.Context, req videogen.Request, opts ...videogen.Option) (*videogen.Result, error) {
req = req.Apply(opts...)
if strings.TrimSpace(req.Prompt) == "" {
return nil, fmt.Errorf("%w: video generation requires a prompt", llm.ErrUnsupported)
}
if req.NumFrames < 0 {
return nil, fmt.Errorf("%w: video frame count must be >= 0, got %d", llm.ErrUnsupported, req.NumFrames)
}
if req.FPS < 0 {
return nil, fmt.Errorf("%w: video fps must be >= 0, got %d", llm.ErrUnsupported, req.FPS)
}
if req.InitImage != nil && len(req.InitImage.Data) == 0 {
return nil, fmt.Errorf("%w: video init image has no bytes", llm.ErrUnsupported)
}
if req.LastImage != nil && len(req.LastImage.Data) == 0 {
return nil, fmt.Errorf("%w: video last image has no bytes", llm.ErrUnsupported)
}
width, height, err := parseSize(req.Size)
if err != nil {
return nil, fmt.Errorf("%w: %v", llm.ErrUnsupported, err)
}
var buf bytes.Buffer
w := multipart.NewWriter(&buf)
if err := writeFormFields(w, "build video form", []formField{
{"model", m.id, true},
{"prompt", req.Prompt, true},
{"negative_prompt", req.NegativePrompt, false},
// Resolution rides the wire twice: width/height (vLLM-Omni's
// names) AND the equivalent OpenAI-style size string, since
// upstreams silently ignore fields they don't understand and the
// values can never disagree.
{"width", formatInt(width), false},
{"height", formatInt(height), false},
{"size", strings.TrimSpace(req.Size), false},
{"num_frames", formatNonZero(req.NumFrames), false},
{"fps", formatNonZero(req.FPS), false},
{"num_inference_steps", formatInt(req.Steps), false},
{"guidance_scale", formatFloat(req.GuidanceScale), false},
{"seed", formatInt64(req.Seed), false},
}); err != nil {
return nil, err
}
if req.InitImage != nil {
if err := writeImagePart(w, "input_reference", "frame", req.InitImage); err != nil {
return nil, err
}
}
// The trailing keyframe rides a SEPARATE part rather than a second
// `input_reference`: multipart permits repeated names, but the receiving
// end would then have to rely on part ORDER to tell first from last, and
// an ordering contract that is invisible in the field name is one nobody
// can see they have broken. A backend that does not know the name ignores
// the part, which is the same degradation as any other unknown field.
if req.LastImage != nil {
if err := writeImagePart(w, "input_reference_last", "frame_last", req.LastImage); err != nil {
return nil, err
}
}
if err := w.Close(); err != nil {
return nil, fmt.Errorf("llama-swap: build video form: %w", err)
}
videoBytes, contentType, err := m.p.doRaw(ctx, http.MethodPost, "/v1/videos/sync", m.id, w.FormDataContentType(), &buf, maxVideoResponseBytes)
if err != nil {
return nil, err
}
return singleVideoResult(m.p.name, m.id, "video", videoBytes, contentType)
}
// videoMIME resolves the result MIME type: the response Content-Type when it
// is a concrete video type, else content sniffing (mp4/webm magic bytes),
// else "" — the caller treats undetectable as an upstream error, unlike the
// audio path where the request's format param implies the container.
func videoMIME(contentType string, data []byte) string {
if mt := mimeFromContentType(contentType, "video/"); mt != "" {
return mt
}
if mt := http.DetectContentType(data); strings.HasPrefix(mt, "video/") {
return mt
}
return ""
}
// singleVideoResult wraps one raw video body into a videogen.Result,
// requiring positive evidence of video-ness (declared video/* Content-Type
// or sniffed mp4/webm magic) so a 2xx body that is anything else — a JSON
// job envelope, an HTML error page behind a proxy — fails loud instead of
// coming back as "the clip". The video sibling of singleImageResult, shared
// by every surface whose response body IS the encoded clip.
func singleVideoResult(provider, model, verb string, raw []byte, contentType string) (*videogen.Result, error) {
if len(raw) == 0 {
return nil, &llm.APIError{Provider: provider, Model: model, Message: verb + " response contained no video"}
}
mimeType := videoMIME(contentType, raw)
if mimeType == "" {
return nil, &llm.APIError{Provider: provider, Model: model,
Message: fmt.Sprintf("%s response is not a video (Content-Type %q)", verb, contentType)}
}
return &videogen.Result{Video: videogen.Video{Data: raw, MIME: mimeType}}, nil
}
// writeImagePart attaches one conditioning frame under the given field name,
// with a filename derived from nameStem. Shared by the first- and last-frame
// parts so the two cannot drift in how they encode, which is the usual way a
// second copy of a block goes wrong.
//
// The two frames MUST carry DISTINCT filenames, not merely distinct field
// names. Backends commonly stage an uploaded frame under a name derived from
// the filename — our own ComfyUI shim posts to /upload/image with
// overwrite=true — so two parts sharing "frame.png" would have the second
// clobber the first, and BOTH keyframe inputs would then resolve to the same
// stored image. The clip would render clean, pinned at both ends to the same
// frame, with nothing anywhere reporting a problem.
func writeImagePart(w *multipart.Writer, field, nameStem string, img *videogen.Image) error {
fw, err := w.CreateFormFile(field, imageFilename(img.MIME, nameStem))
if err != nil {
return fmt.Errorf("llama-swap: build video form: %w", err)
}
if _, err := fw.Write(img.Data); err != nil {
return fmt.Errorf("llama-swap: build video form: %w", err)
}
return nil
}
// formatInt renders an optional int pointer for a form field; nil = "" (omit).
func formatInt(v *int) string {
if v == nil {
return ""
}
return strconv.Itoa(*v)
}
// formatInt64 renders an optional int64 pointer for a form field; nil = "" (omit).
func formatInt64(v *int64) string {
if v == nil {
return ""
}
return strconv.FormatInt(*v, 10)
}
// formatFloat renders an optional float pointer for a form field; nil = "" (omit).
func formatFloat(v *float64) string {
if v == nil {
return ""
}
return strconv.FormatFloat(*v, 'g', -1, 64)
}
// formatNonZero renders a non-negative int for a form field; 0 = "" (omit,
// backend default).
func formatNonZero(v int) string {
if v == 0 {
return ""
}
return strconv.Itoa(v)
}
+356
View File
@@ -0,0 +1,356 @@
package llamaswap
import (
"context"
"encoding/base64"
"errors"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/majordomo/videogen"
)
func TestVideoGenerate(t *testing.T) {
var gotPath, gotContentType string
var gotForm map[string]string
var gotFrame []byte
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
gotContentType = r.Header.Get("Content-Type")
if err := r.ParseMultipartForm(32 << 20); err != nil {
t.Errorf("parse form: %v", err)
return
}
gotForm = map[string]string{}
for k, v := range r.MultipartForm.Value {
gotForm[k] = v[0]
}
if f, _, err := r.FormFile("input_reference"); err == nil {
gotFrame, _ = io.ReadAll(f)
f.Close()
}
w.Header().Set("Content-Type", "video/mp4")
_, _ = w.Write([]byte("fake-mp4-bytes"))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
vm, err := p.VideoModel("videogen-wan")
if err != nil {
t.Fatalf("VideoModel: %v", err)
}
frame, _ := base64.StdEncoding.DecodeString(onePixelPNG)
res, err := vm.Generate(context.Background(),
videogen.Request{Prompt: "a cat surfing", InitImage: &videogen.Image{MIME: "image/png", Data: frame}},
videogen.WithSize("1280x704"),
videogen.WithNumFrames(81),
videogen.WithFPS(16),
videogen.WithSteps(4),
videogen.WithGuidanceScale(1.0),
videogen.WithNegativePrompt("blurry"),
videogen.WithSeed(42),
)
if err != nil {
t.Fatalf("Generate: %v", err)
}
if string(res.Video.Data) != "fake-mp4-bytes" || res.Video.MIME != "video/mp4" {
t.Fatalf("video = %d bytes, MIME %q", len(res.Video.Data), res.Video.MIME)
}
if gotPath != "/v1/videos/sync" {
t.Errorf("path = %q", gotPath)
}
if !strings.HasPrefix(gotContentType, "multipart/form-data") {
t.Errorf("content-type = %q", gotContentType)
}
want := map[string]string{
"model": "videogen-wan",
"prompt": "a cat surfing",
"negative_prompt": "blurry",
"width": "1280",
"height": "704",
"num_frames": "81",
"fps": "16",
"num_inference_steps": "4",
"guidance_scale": "1",
"seed": "42",
}
for k, v := range want {
if gotForm[k] != v {
t.Errorf("form[%q] = %q, want %q", k, gotForm[k], v)
}
}
if string(gotFrame) != string(frame) {
t.Errorf("input_reference = %d bytes, want %d", len(gotFrame), len(frame))
}
}
func TestVideoGenerateOmitsUnsetOverrides(t *testing.T) {
var gotForm map[string][]string
var hadFrame bool
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_ = r.ParseMultipartForm(32 << 20)
gotForm = r.MultipartForm.Value
_, _, err := r.FormFile("input_reference")
hadFrame = err == nil
w.Header().Set("Content-Type", "video/mp4")
_, _ = w.Write([]byte("v"))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
vm, _ := p.VideoModel("videogen-wan")
if _, err := vm.Generate(context.Background(), videogen.Request{Prompt: "x"}); err != nil {
t.Fatalf("Generate: %v", err)
}
for _, k := range []string{"negative_prompt", "width", "height", "num_frames", "fps", "num_inference_steps", "guidance_scale", "seed"} {
if _, ok := gotForm[k]; ok {
t.Errorf("form field %q sent, want omitted", k)
}
}
if hadFrame {
t.Error("input_reference sent, want omitted")
}
}
func TestVideoGenerateValidation(t *testing.T) {
p := New(WithBaseURL("http://unused.invalid"))
vm, _ := p.VideoModel("videogen-wan")
cases := []struct {
name string
req videogen.Request
}{
{"empty prompt", videogen.Request{}},
{"negative frames", videogen.Request{Prompt: "x", NumFrames: -1}},
{"negative fps", videogen.Request{Prompt: "x", FPS: -1}},
{"empty init image", videogen.Request{Prompt: "x", InitImage: &videogen.Image{}}},
{"bad size", videogen.Request{Prompt: "x", Size: "banana"}},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
_, err := vm.Generate(context.Background(), tc.req)
if !errors.Is(err, llm.ErrUnsupported) {
t.Fatalf("err = %v, want ErrUnsupported", err)
}
})
}
}
func TestVideoGenerateUpstreamError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.Error(w, `{"error":{"message":"boom"}}`, http.StatusInternalServerError)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
vm, _ := p.VideoModel("videogen-wan")
_, err := vm.Generate(context.Background(), videogen.Request{Prompt: "x"})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want *llm.APIError", err)
}
}
func TestVideoGenerateEmptyResponse(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "video/mp4")
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
vm, _ := p.VideoModel("videogen-wan")
_, err := vm.Generate(context.Background(), videogen.Request{Prompt: "x"})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want *llm.APIError for empty body", err)
}
}
func TestVideoModelRequiresBaseURL(t *testing.T) {
p := New()
if _, err := p.VideoModel("videogen-wan"); err == nil {
t.Fatal("VideoModel with no base URL should error")
}
}
func TestVideoMIME(t *testing.T) {
// A minimal ISO-BMFF prefix http.DetectContentType sniffs as video/mp4
// (the "mp4" brand prefix must appear inside the declared ftyp box).
mp4Magic := append([]byte{0, 0, 0, 20}, []byte("ftypmp42\x00\x00\x00\x00mp42")...)
cases := []struct {
contentType string
data []byte
want string
}{
{"video/webm", []byte("x"), "video/webm"},
{"video/mp4; charset=binary", []byte("x"), "video/mp4"},
{"application/octet-stream", mp4Magic, "video/mp4"},
// Neither declared nor sniffable as video → "" (Generate errors).
{"application/octet-stream", []byte(`{"id":"job-1"}`), ""},
{"", []byte("x"), ""},
}
for _, tc := range cases {
if got := videoMIME(tc.contentType, tc.data); got != tc.want {
t.Errorf("videoMIME(%q, %q) = %q, want %q", tc.contentType, tc.data, got, tc.want)
}
}
}
func TestVideoGenerateNonVideoBodyErrors(t *testing.T) {
// A stock async /v1/videos handler mounted at the sync path (or an HTML
// error page behind a proxy) answers 200 with a non-video body — that
// must be an error, never a "successful" garbage clip.
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"id":"job-1","status":"queued"}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
vm, _ := p.VideoModel("videogen-wan")
_, err := vm.Generate(context.Background(), videogen.Request{Prompt: "x"})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want *llm.APIError for non-video 2xx body", err)
}
if !strings.Contains(apiErr.Message, "not a video") {
t.Errorf("message = %q, want mention of non-video body", apiErr.Message)
}
}
// Both keyframes reach the wire, under DISTINCT field names.
//
// The distinct-name property is the actual contract with the backend shim: the
// two frames could have shared one repeated `input_reference` name, and then
// which is first and which is last would depend on multipart part ORDER — an
// ordering contract invisible in the payload, that nothing would notice
// breaking. Asserting the names is what pins it.
func TestVideoGenerateSendsBothKeyframes(t *testing.T) {
var gotFirst, gotLast []byte
var firstName, lastName string
var sawLastPart bool
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if err := r.ParseMultipartForm(32 << 20); err != nil {
t.Errorf("parse form: %v", err)
return
}
if f, hdr, err := r.FormFile("input_reference"); err == nil {
gotFirst, _ = io.ReadAll(f)
firstName = hdr.Filename
f.Close()
}
if f, hdr, err := r.FormFile("input_reference_last"); err == nil {
sawLastPart = true
gotLast, _ = io.ReadAll(f)
lastName = hdr.Filename
f.Close()
}
w.Header().Set("Content-Type", "video/mp4")
_, _ = w.Write([]byte("fake-mp4-bytes"))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
vm, err := p.VideoModel("videogen-minimax-h3")
if err != nil {
t.Fatalf("VideoModel: %v", err)
}
first, _ := base64.StdEncoding.DecodeString(onePixelPNG)
last := append(append([]byte{}, first...), 0x00) // distinguishable from first
if _, err := vm.Generate(context.Background(), videogen.Request{
Prompt: "a cat surfing",
InitImage: &videogen.Image{MIME: "image/png", Data: first},
LastImage: &videogen.Image{MIME: "image/png", Data: last},
}); err != nil {
t.Fatalf("Generate: %v", err)
}
if !sawLastPart {
t.Fatal("input_reference_last was not sent — a pinned end frame would be silently dropped")
}
if string(gotFirst) != string(first) {
t.Errorf("input_reference = %d bytes, want %d", len(gotFirst), len(first))
}
if string(gotLast) != string(last) {
t.Errorf("input_reference_last = %d bytes, want %d", len(gotLast), len(last))
}
// The two must not be the same bytes, or a swap/aliasing bug reads as a pass.
if string(gotFirst) == string(gotLast) {
t.Error("both parts carry identical bytes — the frames are being aliased")
}
// DISTINCT FILENAMES, not just distinct field names. Backends stage an
// uploaded frame under a name derived from the filename (our ComfyUI shim
// posts to /upload/image with overwrite=true), so two parts sharing
// "frame.png" would have the second clobber the first and BOTH keyframes
// would resolve to the same stored image — a clip pinned at both ends to
// the same frame, rendering cleanly with nothing reporting a fault.
if firstName == "" || lastName == "" {
t.Fatalf("filenames = %q / %q, want both set", firstName, lastName)
}
if firstName == lastName {
t.Errorf("both parts use filename %q — the second upload would clobber the first", firstName)
}
}
// LastImage alone (no InitImage) is a legitimate request: pin the destination
// and let the model invent the approach. It must not require a first frame.
func TestVideoGenerateLastImageAloneIsAllowed(t *testing.T) {
var sawFirst, sawLast bool
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if err := r.ParseMultipartForm(32 << 20); err != nil {
t.Errorf("parse form: %v", err)
return
}
if f, _, err := r.FormFile("input_reference"); err == nil {
sawFirst = true
f.Close()
}
if f, _, err := r.FormFile("input_reference_last"); err == nil {
sawLast = true
f.Close()
}
w.Header().Set("Content-Type", "video/mp4")
_, _ = w.Write([]byte("fake-mp4-bytes"))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
vm, _ := p.VideoModel("videogen-minimax-h3")
frame, _ := base64.StdEncoding.DecodeString(onePixelPNG)
if _, err := vm.Generate(context.Background(),
videogen.Request{Prompt: "arrive here"},
videogen.WithLastImage(videogen.Image{MIME: "image/png", Data: frame}),
); err != nil {
t.Fatalf("Generate: %v", err)
}
if sawFirst {
t.Error("input_reference sent, want omitted")
}
if !sawLast {
t.Error("input_reference_last omitted, want sent")
}
}
// An empty LastImage is rejected before the request is built, matching
// InitImage's existing contract — a zero-byte frame reaching the backend is a
// confusing upstream error instead of a clear local one.
func TestVideoGenerateRejectsEmptyLastImage(t *testing.T) {
p := New(WithBaseURL("http://unused"))
vm, _ := p.VideoModel("videogen-minimax-h3")
_, err := vm.Generate(context.Background(), videogen.Request{
Prompt: "x",
LastImage: &videogen.Image{MIME: "image/png"},
})
if !errors.Is(err, llm.ErrUnsupported) {
t.Fatalf("err = %v, want llm.ErrUnsupported", err)
}
}
+217
View File
@@ -0,0 +1,217 @@
// videochain.go implements videogen.ChainerProvider against the videoutils
// chain orchestrator reached through llama-swap's /upstream passthrough
// (ADR-0025):
//
// POST /upstream/<id>/v1/video/chain JSON submit -> {job_id}
// GET /upstream/<id>/v1/jobs/{id} -> {status,segment,total,segments}
// GET /upstream/<id>/v1/jobs/{id}/result -> encoded clip
// GET /upstream/<id>/v1/jobs/{id}/segments/{n} -> encoded clip
//
// Unlike the ACE-Step music path this client does NOT hide the job queue
// behind a blocking call: a chain runs through multiple GPU swaps for many
// minutes, and the caller (mort's long-video tool) owns the poll loop so it
// can deliver PARTIAL results — completed segments survive a mid-chain
// failure and stay fetchable via ChainSegmentResult.
package llamaswap
import (
"context"
"encoding/base64"
"encoding/json"
"fmt"
"math"
"net/http"
"strconv"
"strings"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/majordomo/videogen"
)
// ChainerModel implements videogen.ChainerProvider. The id selects which
// upstream llama-swap loads (the videoutils orchestrator, which in turn
// drives the generation model through llama-swap itself).
func (p *Provider) ChainerModel(id string, opts ...videogen.ChainerModelOption) (videogen.Chainer, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
_ = videogen.ApplyChainerModelOptions(opts)
return &chainerModel{p: p, id: id}, nil
}
type chainerModel struct {
p *Provider
id string
}
// chainSubmitRequest is the videoutils POST /v1/video/chain shape. The init
// image rides as base64 in the JSON body (`init_image_b64`) — the submit is
// JSON, not multipart, per the pinned host contract.
type chainSubmitRequest struct {
Segments []chainSegmentWire `json:"segments"`
InitImageB64 string `json:"init_image_b64,omitempty"`
SmoothJoins bool `json:"smooth_joins,omitempty"`
Size string `json:"size,omitempty"`
}
type chainSegmentWire struct {
Prompt string `json:"prompt"`
Seconds float64 `json:"seconds,omitempty"`
}
// SubmitChain implements videogen.Chainer.
func (m *chainerModel) SubmitChain(ctx context.Context, req videogen.ChainRequest) (string, error) {
if len(req.Segments) == 0 {
return "", fmt.Errorf("%w: video chain requires at least one segment", llm.ErrUnsupported)
}
wire := chainSubmitRequest{
SmoothJoins: req.SmoothJoins,
Size: strings.TrimSpace(req.Size),
}
for i, seg := range req.Segments {
if strings.TrimSpace(seg.Prompt) == "" {
return "", fmt.Errorf("%w: video chain segment %d requires a prompt", llm.ErrUnsupported, i)
}
// NaN/±Inf would otherwise surface as an obscure json.Marshal error
// (NaN fails every comparison; +Inf passes the >= 0 check).
if seg.Seconds < 0 || math.IsNaN(seg.Seconds) || math.IsInf(seg.Seconds, 0) {
return "", fmt.Errorf("%w: video chain segment %d seconds must be a finite value >= 0, got %g", llm.ErrUnsupported, i, seg.Seconds)
}
wire.Segments = append(wire.Segments, chainSegmentWire{Prompt: seg.Prompt, Seconds: seg.Seconds})
}
if len(req.InitImage) > 0 {
wire.InitImageB64 = base64.StdEncoding.EncodeToString(req.InitImage)
}
path, err := upstreamPath(m.id, "/v1/video/chain")
if err != nil {
return "", err
}
// Tolerant envelope: {"job_id": ...} per the contract, with data-wrapped
// and bare-id fallbacks (musicgen release_task precedent).
var resp struct {
JobID string `json:"job_id"`
ID string `json:"id"`
Data struct {
JobID string `json:"job_id"`
ID string `json:"id"`
} `json:"data"`
}
if err := m.p.doJSON(ctx, http.MethodPost, path, m.id, &wire, &resp); err != nil {
return "", err
}
for _, id := range []string{resp.JobID, resp.Data.JobID, resp.ID, resp.Data.ID} {
if id != "" {
return id, nil
}
}
return "", &llm.APIError{Provider: m.p.name, Model: m.id, Message: "video chain submit returned no job_id"}
}
// chainJobResponse is the GET /v1/jobs/{id} shape. `segments` entries are
// tolerated as bare strings or objects keyed by id/segment_id.
type chainJobResponse struct {
Status string `json:"status"`
Segment int `json:"segment"`
Total int `json:"total"`
Segments []json.RawMessage `json:"segments"`
}
// ChainStatus implements videogen.Chainer.
func (m *chainerModel) ChainStatus(ctx context.Context, jobID string) (*videogen.ChainJob, error) {
path, err := m.jobPath(jobID, "")
if err != nil {
return nil, err
}
var raw json.RawMessage
if err := m.p.doJSON(ctx, http.MethodGet, path, m.id, nil, &raw); err != nil {
return nil, err
}
var out chainJobResponse
if err := json.Unmarshal(raw, &out); err != nil {
return nil, fmt.Errorf("llama-swap: decode chain job status: %w", err)
}
job := &videogen.ChainJob{
Status: out.Status,
Segment: out.Segment,
Total: out.Total,
Raw: raw,
}
// Entries that carry no usable id — JSON null (which unmarshals into a
// string as a no-op, leaving ""), an empty string, or an object with
// neither key — are SKIPPED, never appended as "": SegmentIDs promises
// fetchable artifacts, and the full payload stays in Raw for callers
// that want the unfiltered list.
for _, entry := range out.Segments {
var s string
if json.Unmarshal(entry, &s) == nil {
if s != "" {
job.SegmentIDs = append(job.SegmentIDs, s)
}
continue
}
var obj struct {
ID string `json:"id"`
SegmentID string `json:"segment_id"`
}
if json.Unmarshal(entry, &obj) == nil {
switch {
case obj.ID != "":
job.SegmentIDs = append(job.SegmentIDs, obj.ID)
case obj.SegmentID != "":
job.SegmentIDs = append(job.SegmentIDs, obj.SegmentID)
}
}
}
if job.Status == "" {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
Message: "chain job status payload carried no status: " + truncateForError(raw)}
}
return job, nil
}
// ChainResult implements videogen.Chainer.
func (m *chainerModel) ChainResult(ctx context.Context, jobID string) (*videogen.Result, error) {
path, err := m.jobPath(jobID, "/result")
if err != nil {
return nil, err
}
raw, respType, err := m.p.doRaw(ctx, http.MethodGet, path, m.id, "", nil, maxVideoResponseBytes)
if err != nil {
return nil, err
}
return singleVideoResult(m.p.name, m.id, "video chain result", raw, respType)
}
// ChainSegmentResult implements videogen.Chainer.
func (m *chainerModel) ChainSegmentResult(ctx context.Context, jobID string, n int) (*videogen.Result, error) {
if n < 0 {
return nil, fmt.Errorf("%w: chain segment index must be >= 0, got %d", llm.ErrUnsupported, n)
}
path, err := m.jobPath(jobID, "/segments/"+strconv.Itoa(n))
if err != nil {
return nil, err
}
raw, respType, err := m.p.doRaw(ctx, http.MethodGet, path, m.id, "", nil, maxVideoResponseBytes)
if err != nil {
return nil, err
}
return singleVideoResult(m.p.name, m.id, "video chain segment", raw, respType)
}
// jobPath builds /upstream/<model>/v1/jobs/<jobID><suffix>, refusing job ids
// that carry path structure. The id is SERVER-SUPPLIED (echoed back from
// SubmitChain), so like upstreamPath's rest-component checks this rejects
// rather than escapes — a hostile/buggy upstream must not be able to steer
// the follow-up request at another proxy endpoint.
func (m *chainerModel) jobPath(jobID, suffix string) (string, error) {
if strings.TrimSpace(jobID) == "" {
return "", fmt.Errorf("llama-swap: chain job call requires a job id")
}
// '%' is rejected alongside the literal path characters: job ids never
// legitimately carry percent-escapes, and %2F/%2E%2E would decode back
// into path structure server-side.
if strings.ContainsAny(jobID, "/?#%") || strings.Contains(jobID, "..") {
return "", fmt.Errorf("llama-swap: invalid chain job id %q (contains path structure)", jobID)
}
return upstreamPath(m.id, "/v1/jobs/"+jobID+suffix)
}
+322
View File
@@ -0,0 +1,322 @@
package llamaswap
import (
"context"
"encoding/base64"
"encoding/json"
"errors"
"math"
"net/http"
"net/http/httptest"
"reflect"
"testing"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/majordomo/videogen"
)
func TestSubmitChain(t *testing.T) {
var gotPath string
var gotBody map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
_ = json.NewDecoder(r.Body).Decode(&gotBody)
_, _ = w.Write([]byte(`{"job_id":"job-123"}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
ch, err := p.ChainerModel("videoutils")
if err != nil {
t.Fatalf("ChainerModel: %v", err)
}
jobID, err := ch.SubmitChain(context.Background(), videogen.ChainRequest{
Segments: []videogen.ChainSegment{
{Prompt: "a cat walks in", Seconds: 5},
{Prompt: "the cat sits down", Seconds: 5},
},
InitImage: []byte("IMG"),
SmoothJoins: true,
Size: "1280x704",
})
if err != nil {
t.Fatalf("SubmitChain: %v", err)
}
if jobID != "job-123" {
t.Errorf("jobID = %q", jobID)
}
if gotPath != "/upstream/videoutils/v1/video/chain" {
t.Errorf("path = %q", gotPath)
}
segments, _ := gotBody["segments"].([]any)
if len(segments) != 2 {
t.Fatalf("segments = %v", gotBody["segments"])
}
first, _ := segments[0].(map[string]any)
if first["prompt"] != "a cat walks in" || first["seconds"] != 5.0 {
t.Errorf("segment[0] = %v", first)
}
if gotBody["init_image_b64"] != base64.StdEncoding.EncodeToString([]byte("IMG")) {
t.Errorf("init_image_b64 = %v", gotBody["init_image_b64"])
}
if gotBody["smooth_joins"] != true || gotBody["size"] != "1280x704" {
t.Errorf("smooth_joins/size = %v/%v", gotBody["smooth_joins"], gotBody["size"])
}
}
func TestSubmitChainOmitsUnsetFields(t *testing.T) {
var gotBody map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_ = json.NewDecoder(r.Body).Decode(&gotBody)
_, _ = w.Write([]byte(`{"data":{"job_id":"job-9"}}`)) // data-wrapped envelope
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
ch, _ := p.ChainerModel("videoutils")
jobID, err := ch.SubmitChain(context.Background(), videogen.ChainRequest{
Segments: []videogen.ChainSegment{{Prompt: "a dog"}},
})
if err != nil {
t.Fatalf("SubmitChain: %v", err)
}
if jobID != "job-9" {
t.Errorf("jobID = %q, want data-wrapped id", jobID)
}
for _, k := range []string{"init_image_b64", "smooth_joins", "size"} {
if v, ok := gotBody[k]; ok {
t.Errorf("unset request sent %q = %v, want omitted", k, v)
}
}
seg, _ := gotBody["segments"].([]any)
if first, _ := seg[0].(map[string]any); first == nil {
t.Fatalf("segments = %v", gotBody["segments"])
} else if _, ok := first["seconds"]; ok {
t.Error("zero seconds sent; want omitted")
}
}
func TestSubmitChainRejectsBadArgs(t *testing.T) {
p := New(WithBaseURL("http://unused"))
ch, _ := p.ChainerModel("videoutils")
if _, err := ch.SubmitChain(context.Background(), videogen.ChainRequest{}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("no segments: err = %v, want ErrUnsupported", err)
}
if _, err := ch.SubmitChain(context.Background(), videogen.ChainRequest{
Segments: []videogen.ChainSegment{{Prompt: " "}},
}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("empty prompt: err = %v, want ErrUnsupported", err)
}
if _, err := ch.SubmitChain(context.Background(), videogen.ChainRequest{
Segments: []videogen.ChainSegment{{Prompt: "x", Seconds: -1}},
}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("negative seconds: err = %v, want ErrUnsupported", err)
}
for name, bad := range map[string]float64{
"NaN": math.NaN(),
"+Inf": math.Inf(1),
"-Inf": math.Inf(-1),
} {
if _, err := ch.SubmitChain(context.Background(), videogen.ChainRequest{
Segments: []videogen.ChainSegment{{Prompt: "x", Seconds: bad}},
}); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("%s seconds: err = %v, want ErrUnsupported", name, err)
}
}
}
func TestSubmitChainRejectsMissingJobID(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{"ok":true}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
ch, _ := p.ChainerModel("videoutils")
_, err := ch.SubmitChain(context.Background(), videogen.ChainRequest{
Segments: []videogen.ChainSegment{{Prompt: "x"}},
})
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want APIError for missing job_id", err)
}
}
func TestChainStatus(t *testing.T) {
var gotPath string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
_, _ = w.Write([]byte(`{"status":"running","segment":2,"total":3,"segments":["seg-0"]}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
ch, _ := p.ChainerModel("videoutils")
job, err := ch.ChainStatus(context.Background(), "job-123")
if err != nil {
t.Fatalf("ChainStatus: %v", err)
}
if gotPath != "/upstream/videoutils/v1/jobs/job-123" {
t.Errorf("path = %q", gotPath)
}
if job.Status != "running" || job.Segment != 2 || job.Total != 3 {
t.Errorf("job = %+v", job)
}
if !reflect.DeepEqual(job.SegmentIDs, []string{"seg-0"}) {
t.Errorf("SegmentIDs = %v", job.SegmentIDs)
}
if job.Raw == nil {
t.Error("Raw = nil, want raw payload")
}
}
func TestChainStatusToleratesObjectSegments(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{"status":"done","segment":2,"total":2,"segments":[{"id":"seg-0"},{"segment_id":"seg-1"}]}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
ch, _ := p.ChainerModel("videoutils")
job, err := ch.ChainStatus(context.Background(), "job-123")
if err != nil {
t.Fatalf("ChainStatus: %v", err)
}
if !reflect.DeepEqual(job.SegmentIDs, []string{"seg-0", "seg-1"}) {
t.Errorf("SegmentIDs = %v", job.SegmentIDs)
}
}
func TestChainStatusSkipsIDLessSegments(t *testing.T) {
// null, "", an id-less object, and a mistyped entry must all be
// skipped — never appended as "" (SegmentIDs promises fetchable
// artifacts; the unfiltered list stays in Raw).
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{"status":"running","segment":3,"total":5,` +
`"segments":[null,"seg-0","",{},{"id":"seg-1"},{"segment_id":""},42]}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
ch, _ := p.ChainerModel("videoutils")
job, err := ch.ChainStatus(context.Background(), "job-123")
if err != nil {
t.Fatalf("ChainStatus: %v", err)
}
if !reflect.DeepEqual(job.SegmentIDs, []string{"seg-0", "seg-1"}) {
t.Errorf("SegmentIDs = %v, want [seg-0 seg-1]", job.SegmentIDs)
}
}
func TestChainStatusRejectsStatuslessPayload(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{"detail":"no such job"}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
ch, _ := p.ChainerModel("videoutils")
_, err := ch.ChainStatus(context.Background(), "job-123")
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want APIError for statusless payload", err)
}
}
func TestChainJobPathRejectsHostileIDs(t *testing.T) {
p := New(WithBaseURL("http://unused"))
ch, _ := p.ChainerModel("videoutils")
for _, bad := range []string{"", "a/b", "a?b", "a#b", "..", "a..b", "a%2Fb", "%2e%2e", "a%b"} {
if _, err := ch.ChainStatus(context.Background(), bad); err == nil {
t.Errorf("ChainStatus(%q) succeeded; want error", bad)
}
if _, err := ch.ChainResult(context.Background(), bad); err == nil {
t.Errorf("ChainResult(%q) succeeded; want error", bad)
}
}
}
func TestChainResult(t *testing.T) {
var gotPath string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
w.Header().Set("Content-Type", "video/mp4")
_, _ = w.Write(mp4Fixture())
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
ch, _ := p.ChainerModel("videoutils")
res, err := ch.ChainResult(context.Background(), "job-123")
if err != nil {
t.Fatalf("ChainResult: %v", err)
}
if gotPath != "/upstream/videoutils/v1/jobs/job-123/result" {
t.Errorf("path = %q", gotPath)
}
if res.Video.MIME != "video/mp4" || len(res.Video.Data) == 0 {
t.Fatalf("video = %q/%d bytes", res.Video.MIME, len(res.Video.Data))
}
}
func TestChainSegmentResult(t *testing.T) {
var gotPath string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.Path
w.Header().Set("Content-Type", "video/mp4")
_, _ = w.Write(mp4Fixture())
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
ch, _ := p.ChainerModel("videoutils")
res, err := ch.ChainSegmentResult(context.Background(), "job-123", 1)
if err != nil {
t.Fatalf("ChainSegmentResult: %v", err)
}
if gotPath != "/upstream/videoutils/v1/jobs/job-123/segments/1" {
t.Errorf("path = %q", gotPath)
}
if res.Video.MIME != "video/mp4" {
t.Fatalf("video = %q", res.Video.MIME)
}
if _, err := ch.ChainSegmentResult(context.Background(), "job-123", -1); !errors.Is(err, llm.ErrUnsupported) {
t.Errorf("negative segment: err = %v, want ErrUnsupported", err)
}
}
func TestChainResultRejectsNonVideoResponse(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"status":"running"}`)) // a status page, not the clip
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
ch, _ := p.ChainerModel("videoutils")
_, err := ch.ChainResult(context.Background(), "job-123")
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %v, want APIError for non-video body", err)
}
}
func TestChainSurfacesAPIError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNotFound)
_, _ = w.Write([]byte(`{"error":{"message":"job not found"}}`))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
ch, _ := p.ChainerModel("videoutils")
_, err := ch.ChainStatus(context.Background(), "job-void")
var apiErr *llm.APIError
if !errors.As(err, &apiErr) {
t.Fatalf("err = %T %v, want *llm.APIError", err, err)
}
if apiErr.Status != http.StatusNotFound || apiErr.Message != "job not found" || apiErr.Model != "videoutils" {
t.Errorf("apiErr = %+v", apiErr)
}
}
+139
View File
@@ -0,0 +1,139 @@
// videoutil.go implements the videogen.VideoBackgroundRemovalProvider and
// videogen.VideoUpscaleProvider surfaces against the mediautils shim reached
// through llama-swap's /upstream passthrough (ADR-0025):
//
// matte POST /upstream/<id>/v1/video/matte (Robust Video Matting)
// upscale POST /upstream/<id>/v1/video/upscale (per-frame Real-ESRGAN)
//
// Both are one-file multipart in, encoded clip out — the video siblings of
// mediautil.go's still-image surfaces.
package llamaswap
import (
"context"
"fmt"
"mime"
"net/http"
"strconv"
"strings"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/majordomo/videogen"
)
// --- video background removal (matting) ---
// VideoBackgroundRemoverModel implements
// videogen.VideoBackgroundRemovalProvider against the mediautils shim's
// POST /v1/video/matte. The id selects which upstream llama-swap loads.
func (p *Provider) VideoBackgroundRemoverModel(id string, opts ...videogen.VideoBackgroundRemoverModelOption) (videogen.VideoBackgroundRemover, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
_ = videogen.ApplyVideoBackgroundRemoverModelOptions(opts)
return &videoMatteModel{p: p, id: id}, nil
}
type videoMatteModel struct {
p *Provider
id string
}
// RemoveVideoBackground implements videogen.VideoBackgroundRemover.
func (m *videoMatteModel) RemoveVideoBackground(ctx context.Context, req videogen.VideoBackgroundRemovalRequest, opts ...videogen.VideoBackgroundRemovalOption) (*videogen.Result, error) {
req = req.Apply(opts...)
if len(req.Video) == 0 {
return nil, fmt.Errorf("%w: video background removal requires a video", llm.ErrUnsupported)
}
if req.Output != "" && req.Output != "greenscreen_mp4" && req.Output != "alpha_webm" {
return nil, fmt.Errorf("%w: video matte output must be \"greenscreen_mp4\" or \"alpha_webm\", got %q", llm.ErrUnsupported, req.Output)
}
path, err := upstreamPath(m.id, "/v1/video/matte")
if err != nil {
return nil, err
}
body, contentType, err := buildMultipart("build video-matte form",
filePart{field: "file", filename: videoInputFilename(req.Filename, req.MIME), data: req.Video},
[]formField{{"output", req.Output, false}})
if err != nil {
return nil, err
}
raw, respType, err := m.p.doRaw(ctx, http.MethodPost, path, m.id, contentType, body, maxVideoResponseBytes)
if err != nil {
return nil, err
}
return singleVideoResult(m.p.name, m.id, "video matte", raw, respType)
}
// --- video upscale ---
// VideoUpscalerModel implements videogen.VideoUpscaleProvider against the
// mediautils shim's POST /v1/video/upscale.
func (p *Provider) VideoUpscalerModel(id string, opts ...videogen.VideoUpscalerModelOption) (videogen.VideoUpscaler, error) {
if err := p.requireBaseURL(); err != nil {
return nil, err
}
_ = videogen.ApplyVideoUpscalerModelOptions(opts)
return &videoUpscaleModel{p: p, id: id}, nil
}
type videoUpscaleModel struct {
p *Provider
id string
}
// UpscaleVideo implements videogen.VideoUpscaler.
func (m *videoUpscaleModel) UpscaleVideo(ctx context.Context, req videogen.VideoUpscaleRequest, opts ...videogen.VideoUpscaleOption) (*videogen.Result, error) {
req = req.Apply(opts...)
if len(req.Video) == 0 {
return nil, fmt.Errorf("%w: video upscale requires a video", llm.ErrUnsupported)
}
if req.Scale != 0 && req.Scale != 2 && req.Scale != 4 {
return nil, fmt.Errorf("%w: video upscale scale must be 2 or 4, got %d", llm.ErrUnsupported, req.Scale)
}
path, err := upstreamPath(m.id, "/v1/video/upscale")
if err != nil {
return nil, err
}
scale := ""
if req.Scale != 0 {
scale = strconv.Itoa(req.Scale)
}
body, contentType, err := buildMultipart("build video-upscale form",
filePart{field: "file", filename: videoInputFilename(req.Filename, req.MIME), data: req.Video},
[]formField{{"scale", scale, false}})
if err != nil {
return nil, err
}
raw, respType, err := m.p.doRaw(ctx, http.MethodPost, path, m.id, contentType, body, maxVideoResponseBytes)
if err != nil {
return nil, err
}
return singleVideoResult(m.p.name, m.id, "video upscale", raw, respType)
}
// videoInputFilename picks the multipart filename hint for a caller-supplied
// video: the caller's (sanitized — upload metadata is untrusted), else one
// derived from the MIME subtype ("video.mp4"), else "video". Mirrors
// transcriptionFilename.
func videoInputFilename(filename, mimeType string) string {
if name := sanitizeFilename(filename); name != "" {
return name
}
mt := strings.ToLower(strings.TrimSpace(mimeType))
if parsed, _, err := mime.ParseMediaType(mt); err == nil {
mt = parsed
}
switch mt {
case "video/mp4":
return "video.mp4"
case "video/webm":
return "video.webm"
case "video/quicktime":
return "video.mov"
case "video/x-matroska":
return "video.mkv"
default:
return "video"
}
}
+1 -1
View File
@@ -82,7 +82,7 @@ func (m *model) do(ctx context.Context, req llm.Request, stream bool) (*http.Res
Model: m.id,
Status: http.StatusUnauthorized,
Code: "missing_api_key",
Message: "no API key configured: set OPENAI_API_KEY or use WithAPIKey",
Message: "no API key configured: set " + m.p.apiKeyName + " or use WithAPIKey",
}
}
body, err := json.Marshal(m.buildRequest(req, stream))
+15 -5
View File
@@ -34,6 +34,7 @@ const defaultBaseURL = "https://api.openai.com/v1"
type Provider struct {
name string
apiKey string
apiKeyName string
baseURL string
client *http.Client
caps llm.Capabilities
@@ -64,6 +65,14 @@ func WithHTTPClient(c *http.Client) Option {
}
}
// WithAPIKeyName sets the environment-variable name shown in the missing-key
// error (default "OPENAI_API_KEY"). Why: the same client serves compat
// endpoints keyed by other env vars (e.g. KIMI_API_KEY), and the error should
// name the one the operator actually needs to set.
func WithAPIKeyName(name string) Option {
return func(p *Provider) { p.apiKeyName = name }
}
// WithName overrides the registry name ("openai" by default). Why: the same
// client serves many OpenAI-compatible endpoints, and each needs a distinct
// name in "provider/model" specs and error reporting.
@@ -104,11 +113,12 @@ func defaultCapabilities() llm.Capabilities {
// 401-style *llm.APIError at request time, not at construction.
func New(opts ...Option) *Provider {
p := &Provider{
name: "openai",
apiKey: os.Getenv("OPENAI_API_KEY"),
baseURL: defaultBaseURL,
client: http.DefaultClient,
caps: defaultCapabilities(),
name: "openai",
apiKey: os.Getenv("OPENAI_API_KEY"),
apiKeyName: "OPENAI_API_KEY",
baseURL: defaultBaseURL,
client: http.DefaultClient,
caps: defaultCapabilities(),
}
for _, opt := range opts {
opt(p)
+1 -1
View File
@@ -263,7 +263,7 @@ func (r *Registry) providerFor(name string) (llm.Provider, error) {
return nil, envErr
}
envKey := "LLM_" + strings.ToUpper(strings.ReplaceAll(name, "-", "_"))
envKey := envKeyForProvider(name)
envVal := r.envLookup(envKey)
if envVal == "" {
return nil, fmt.Errorf("%w: %q (checked registry and %s env var)", ErrUnknownProvider, name, envKey)
+103
View File
@@ -0,0 +1,103 @@
package videogen
import "context"
// ChainSegment is one prompt in a multi-segment ("long video") chain.
type ChainSegment struct {
// Prompt describes this segment. Required.
Prompt string
// Seconds is the segment's requested length; 0 = backend default.
Seconds float64
}
// ChainRequest asks a chain orchestrator to generate a long video as a
// sequence of segments, each continuing from the previous segment's last
// frame. Zero values mean "backend default" (ADR-0025).
type ChainRequest struct {
// Segments are the per-segment prompts in order. At least one required.
Segments []ChainSegment
// InitImage optionally conditions the FIRST segment on a starting frame
// (image-to-video); nil = pure text-to-video.
InitImage []byte
// SmoothJoins asks the orchestrator to interpolate across segment
// boundaries (RIFE-style) so cuts don't pop.
SmoothJoins bool
// Size is the requested resolution, e.g. "1280x704"; "" = backend
// default.
Size string
}
// ChainJob is a chain job's progress snapshot.
type ChainJob struct {
// Status is the backend's job state, passed through verbatim
// (e.g. "queued", "running", "done", "failed").
Status string
// Segment is the segment currently being generated (1-based); Total is
// the segment count.
Segment int
Total int
// SegmentIDs name the COMPLETED per-segment artifacts, in order. A
// mid-chain failure still leaves those segments retrievable via
// ChainSegmentResult — note it takes the segment's index in the chain,
// not an id string; this list tells you WHICH segments completed. So
// multi-minute GPU output is never discarded. Entries the backend
// reports without a usable id are skipped (the unfiltered list survives
// in Raw).
SegmentIDs []string
// Raw is the provider-native job payload. May be nil.
Raw any
}
// Chainer drives a multi-segment video-chain job. Unlike Model.Generate it
// is deliberately ASYNC — a chain runs through multiple GPU loads for many
// minutes, so callers submit, poll, and fetch instead of holding one
// blocking call open.
type Chainer interface {
// SubmitChain starts a chain job and returns its job id.
SubmitChain(ctx context.Context, req ChainRequest) (string, error)
// ChainStatus reports the job's progress. Polling also signals liveness
// to backends that unload idle orchestrators.
ChainStatus(ctx context.Context, jobID string) (*ChainJob, error)
// ChainResult fetches the finished, concatenated clip.
ChainResult(ctx context.Context, jobID string) (*Result, error)
// ChainSegmentResult fetches one completed segment's clip (n indexes the
// job's segment list) — the partial-delivery path when a chain dies
// mid-run.
ChainSegmentResult(ctx context.Context, jobID string, n int) (*Result, error)
}
// ChainerModelOption configures a Chainer at construction time. Reserved for
// future per-model settings.
type ChainerModelOption func(*ChainerModelConfig)
// ChainerModelConfig carries per-model construction settings.
type ChainerModelConfig struct{}
// ApplyChainerModelOptions folds options into a config.
func ApplyChainerModelOptions(opts []ChainerModelOption) ChainerModelConfig {
var cfg ChainerModelConfig
for _, opt := range opts {
opt(&cfg)
}
return cfg
}
// ChainerProvider mints Chainers bound to one backend.
type ChainerProvider interface {
// Name is the registry identifier for the provider.
Name() string
// ChainerModel returns a Chainer bound to the given id (passed through
// to the backend verbatim; no catalog validation).
ChainerModel(id string, opts ...ChainerModelOption) (Chainer, error)
}
+82
View File
@@ -0,0 +1,82 @@
package videogen
import "context"
// InterpolateRequest asks a frame-interpolation backend (RIFE-style) to
// synthesize intermediate frames in an existing clip. Zero values mean
// "backend default" (ADR-0020).
type InterpolateRequest struct {
// Video is the clip to interpolate. Required.
Video Video
// Multi is the frame multiplier (2 or 4 on the reference backend);
// 0 = backend default (2).
Multi int
// SlowMo plays the synthesized frames at the SOURCE frame rate instead of
// multiplying it: Multi-times slow motion. The backend strips audio in
// this mode (time-stretched audio is noise).
SlowMo bool
// TargetFPS resamples the smoothed clip to a specific frame rate
// (e.g. 60); 0 = Multi times the source rate. Ignored in SlowMo mode.
TargetFPS int
}
// InterpolateOption mutates an InterpolateRequest before it is sent.
type InterpolateOption func(*InterpolateRequest)
// WithInterpolateMulti sets the frame multiplier.
func WithInterpolateMulti(m int) InterpolateOption {
return func(r *InterpolateRequest) { r.Multi = m }
}
// WithSlowMo switches to slow-motion output.
func WithSlowMo() InterpolateOption { return func(r *InterpolateRequest) { r.SlowMo = true } }
// WithTargetFPS resamples the smoothed clip to a specific frame rate.
func WithTargetFPS(fps int) InterpolateOption {
return func(r *InterpolateRequest) { r.TargetFPS = fps }
}
// Apply returns a copy of the request with all options applied.
func (r InterpolateRequest) Apply(opts ...InterpolateOption) InterpolateRequest {
for _, opt := range opts {
opt(&r)
}
return r
}
// Interpolator synthesizes intermediate frames (fps boost or slow-mo). Its
// own small interface rather than a method on Model: interpolators are not
// generators — they bind to a different backend id entirely.
type Interpolator interface {
// Interpolate returns the smoothed (or slowed) clip.
Interpolate(ctx context.Context, req InterpolateRequest, opts ...InterpolateOption) (*Result, error)
}
// InterpolatorModelOption configures an Interpolator at construction time.
// Reserved for future per-model settings.
type InterpolatorModelOption func(*InterpolatorModelConfig)
// InterpolatorModelConfig carries per-model construction settings.
type InterpolatorModelConfig struct{}
// ApplyInterpolatorModelOptions folds options into a config.
func ApplyInterpolatorModelOptions(opts []InterpolatorModelOption) InterpolatorModelConfig {
var cfg InterpolatorModelConfig
for _, opt := range opts {
opt(&cfg)
}
return cfg
}
// InterpolationProvider mints Interpolators bound to one backend.
type InterpolationProvider interface {
// Name is the registry identifier for the provider.
Name() string
// InterpolatorModel returns an Interpolator bound to the given id (passed
// through to the backend verbatim; no catalog validation).
InterpolatorModel(id string, opts ...InterpolatorModelOption) (Interpolator, error)
}
+93
View File
@@ -0,0 +1,93 @@
package videogen
import "context"
// LipsyncRequest asks a talking-head backend (SadTalker style) to animate a
// still portrait so it speaks the given audio. Zero values mean "backend
// default" (ADR-0025).
type LipsyncRequest struct {
// Image is the portrait to animate. Required.
Image Image
// Audio is the encoded speech the head lip-syncs to. Required. Carried
// as bytes (never a URL), mirroring audio.TranscriptionRequest.
Audio []byte
// AudioMIME is the audio MIME type (e.g. "audio/wav"); "" = let the
// backend sniff it.
AudioMIME string
// AudioFilename is the multipart filename hint some backends key their
// format detection on; "" derives one from AudioMIME or falls back to
// "audio".
AudioFilename string
// Still reduces head motion to blinks and lip movement (less uncanny on
// formal portraits); false = backend default motion.
Still bool
// Enhance runs the backend's face enhancer over the output frames.
Enhance bool
// Preprocess selects how the backend frames the face: "crop" (animate
// the face crop) or "full" (paste the animated face back into the whole
// image); "" = backend default.
Preprocess string
}
// LipsyncOption mutates a LipsyncRequest before it is sent.
type LipsyncOption func(*LipsyncRequest)
// WithLipsyncStill reduces head motion to blinks and lip movement.
func WithLipsyncStill() LipsyncOption { return func(r *LipsyncRequest) { r.Still = true } }
// WithLipsyncEnhance runs the backend's face enhancer over the output.
func WithLipsyncEnhance() LipsyncOption { return func(r *LipsyncRequest) { r.Enhance = true } }
// WithLipsyncPreprocess selects the face framing ("crop" or "full").
func WithLipsyncPreprocess(p string) LipsyncOption {
return func(r *LipsyncRequest) { r.Preprocess = p }
}
// Apply returns a copy of the request with all options applied.
func (r LipsyncRequest) Apply(opts ...LipsyncOption) LipsyncRequest {
for _, opt := range opts {
opt(&r)
}
return r
}
// Lipsyncer animates still portraits into talking-head clips. Its own small
// interface rather than a method on Model: lip-syncers are not text-to-video
// generators — they bind to a different backend id entirely.
type Lipsyncer interface {
// Lipsync returns the talking-head clip. Generation is slow (minutes);
// bound the call with a context deadline.
Lipsync(ctx context.Context, req LipsyncRequest, opts ...LipsyncOption) (*Result, error)
}
// LipsyncModelOption configures a Lipsyncer at construction time. Reserved
// for future per-model settings.
type LipsyncModelOption func(*LipsyncModelConfig)
// LipsyncModelConfig carries per-model construction settings.
type LipsyncModelConfig struct{}
// ApplyLipsyncModelOptions folds options into a config.
func ApplyLipsyncModelOptions(opts []LipsyncModelOption) LipsyncModelConfig {
var cfg LipsyncModelConfig
for _, opt := range opts {
opt(&cfg)
}
return cfg
}
// LipsyncProvider mints Lipsyncers bound to one backend.
type LipsyncProvider interface {
// Name is the registry identifier for the provider.
Name() string
// LipsyncModel returns a Lipsyncer bound to the given id (passed through
// to the backend verbatim; no catalog validation).
LipsyncModel(id string, opts ...LipsyncModelOption) (Lipsyncer, error)
}
+80
View File
@@ -0,0 +1,80 @@
package videogen
import "context"
// VideoBackgroundRemovalRequest asks a video-matting backend (Robust Video
// Matting style) to separate the foreground subject from the background of a
// clip. Zero values mean "backend default" (ADR-0025).
type VideoBackgroundRemovalRequest struct {
// Video is the encoded clip to matte. Required. Carried as bytes (never
// a URL).
Video []byte
// MIME is the video MIME type (e.g. "video/mp4"); "" = let the backend
// sniff it.
MIME string
// Filename is the multipart filename hint some backends key their format
// detection on; "" derives one from MIME ("video.mp4") or falls back to
// "video".
Filename string
// Output selects the delivery container: "greenscreen_mp4" (subject over
// solid green, universally playable) or "alpha_webm" (true transparency,
// VP9 alpha channel); "" = backend default.
Output string
}
// VideoBackgroundRemovalOption mutates a VideoBackgroundRemovalRequest
// before it is sent.
type VideoBackgroundRemovalOption func(*VideoBackgroundRemovalRequest)
// WithVideoBackgroundOutput selects the delivery container
// ("greenscreen_mp4" or "alpha_webm").
func WithVideoBackgroundOutput(o string) VideoBackgroundRemovalOption {
return func(r *VideoBackgroundRemovalRequest) { r.Output = o }
}
// Apply returns a copy of the request with all options applied.
func (r VideoBackgroundRemovalRequest) Apply(opts ...VideoBackgroundRemovalOption) VideoBackgroundRemovalRequest {
for _, opt := range opts {
opt(&r)
}
return r
}
// VideoBackgroundRemover mattes the subject out of video clips — the moving-
// picture sibling of imagegen.BackgroundRemover.
type VideoBackgroundRemover interface {
// RemoveVideoBackground returns the matted clip. Matting is slow
// (per-frame inference); bound the call with a context deadline.
RemoveVideoBackground(ctx context.Context, req VideoBackgroundRemovalRequest, opts ...VideoBackgroundRemovalOption) (*Result, error)
}
// VideoBackgroundRemoverModelOption configures a VideoBackgroundRemover at
// construction time. Reserved for future per-model settings.
type VideoBackgroundRemoverModelOption func(*VideoBackgroundRemoverModelConfig)
// VideoBackgroundRemoverModelConfig carries per-model construction settings.
type VideoBackgroundRemoverModelConfig struct{}
// ApplyVideoBackgroundRemoverModelOptions folds options into a config.
func ApplyVideoBackgroundRemoverModelOptions(opts []VideoBackgroundRemoverModelOption) VideoBackgroundRemoverModelConfig {
var cfg VideoBackgroundRemoverModelConfig
for _, opt := range opts {
opt(&cfg)
}
return cfg
}
// VideoBackgroundRemovalProvider mints VideoBackgroundRemovers bound to one
// backend.
type VideoBackgroundRemovalProvider interface {
// Name is the registry identifier for the provider.
Name() string
// VideoBackgroundRemoverModel returns a VideoBackgroundRemover bound to
// the given id (passed through to the backend verbatim; no catalog
// validation).
VideoBackgroundRemoverModel(id string, opts ...VideoBackgroundRemoverModelOption) (VideoBackgroundRemover, error)
}
+180
View File
@@ -0,0 +1,180 @@
// Package videogen is majordomo's canonical video-generation surface. Like
// imagegen and audio, it is a deliberately separate contract from the llm
// package: video generation shares none of the chat message/tool/stream
// machinery, so it gets its own small Provider/Model interface rather than
// overloading llm.Model (ADR-0019).
//
// Zero values mean "backend default" throughout, mirroring imagegen: an empty
// Size leaves the backend's default resolution, zero NumFrames/FPS the
// backend's default clip length and rate.
//
// Text-to-video and image-to-video are one surface: a Request with a nil
// InitImage is a pure text prompt, a non-nil InitImage conditions generation
// on that frame. Hybrid models (e.g. Wan 2.2 TI2V) serve both from the same
// checkpoint, so unlike imagegen there is no separate Editor-style interface.
// LastImage extends the same surface to the other end of the clip, so one
// Request covers t2v, i2v, and first-last-frame-to-video without a mode flag.
//
// The first implementation is provider/llamaswap, which targets the blocking
// OpenAI/vLLM-Omni-style POST /v1/videos/sync endpoint: the response body is
// the encoded video itself, so one request yields exactly one clip — Result
// carries a single Video, not a batch.
package videogen
import (
"context"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
// Image is a conditioning input frame (bytes + MIME). Aliased to
// llm.ImagePart so chat-sourced images feed image-to-video without
// conversion, mirroring imagegen.Image.
type Image = llm.ImagePart
// Video is one generated video: raw encoded bytes plus a MIME type
// (e.g. "video/mp4").
type Video struct {
// Data is the encoded video container.
Data []byte
// MIME is the video MIME type, e.g. "video/mp4".
MIME string
}
// Request is a video generation request. Zero values mean "backend default" —
// for llama-swap-served models that is the per-model default baked into the
// upstream launch flags. A caller overrides only what it explicitly sets.
type Request struct {
// Prompt is the text description of the video to generate.
Prompt string
// InitImage conditions generation on a starting frame (image-to-video);
// nil = pure text-to-video.
InitImage *Image
// LastImage conditions generation on an ENDING frame. With InitImage it
// pins both ends (first-last-frame-to-video); alone it pins only the
// destination and lets the backend invent the approach.
//
// Support is per-model and NOT advertised anywhere in this contract: a
// backend that does not understand a trailing keyframe ignores it and
// returns an ordinary clip, which is indistinguishable from success.
// There is no capability bit to consult, because the contract has no way
// to learn one. A caller that needs to know whether the pin actually took
// effect must establish that out of band — by configuration it controls,
// not by inspecting the result.
LastImage *Image
// Size is the requested resolution, e.g. "1280x704"; "" = backend default.
Size string
// NumFrames is the clip length in frames; 0 = backend default.
NumFrames int
// FPS is the frame rate of the generated clip; 0 = backend default.
FPS int
// Steps is the number of diffusion steps; nil = backend default.
Steps *int
// GuidanceScale is the guidance strength; nil = backend default.
// Architecture-sensitive (distilled models want low or none), so prefer
// leaving it nil unless the caller knows the target model.
GuidanceScale *float64
// NegativePrompt steers generation away from concepts; "" = none.
NegativePrompt string
// Seed fixes the RNG seed for reproducible output; nil = random.
Seed *int64
}
// Result is the canonical video-generation result.
type Result struct {
// Video is the generated clip.
Video Video
// Raw is the provider-native response object, an escape hatch for
// provider-specific fields. May be nil; never required for normal use.
Raw any
}
// Option mutates a Request before it is sent. Options passed to Generate are
// applied to a copy of the request, so a Request value can be reused.
type Option func(*Request)
// WithInitImage conditions generation on a starting frame (image-to-video).
func WithInitImage(img Image) Option { return func(r *Request) { r.InitImage = &img } }
// WithLastImage conditions generation on an ending frame. Combined with
// WithInitImage this pins both ends of the clip.
func WithLastImage(img Image) Option { return func(r *Request) { r.LastImage = &img } }
// WithSize sets the requested resolution (e.g. "1280x704").
func WithSize(size string) Option { return func(r *Request) { r.Size = size } }
// WithNumFrames sets the clip length in frames.
func WithNumFrames(n int) Option { return func(r *Request) { r.NumFrames = n } }
// WithFPS sets the frame rate of the generated clip.
func WithFPS(fps int) Option { return func(r *Request) { r.FPS = fps } }
// WithSteps overrides the number of diffusion steps.
func WithSteps(n int) Option { return func(r *Request) { r.Steps = &n } }
// WithGuidanceScale overrides the guidance strength.
func WithGuidanceScale(s float64) Option { return func(r *Request) { r.GuidanceScale = &s } }
// WithNegativePrompt sets a negative prompt.
func WithNegativePrompt(s string) Option { return func(r *Request) { r.NegativePrompt = s } }
// WithSeed fixes the RNG seed for reproducible output.
func WithSeed(seed int64) Option { return func(r *Request) { r.Seed = &seed } }
// Apply returns a copy of the request with all options applied. Providers call
// this once at the top of Generate.
func (r Request) Apply(opts ...Option) Request {
for _, opt := range opts {
opt(&r)
}
return r
}
// Model generates a video clip from a text prompt and optional conditioning
// frame. It is intentionally narrower than llm.Model — no Stream, no
// Capabilities, no tool calls.
type Model interface {
// Generate produces one clip for the request. Generation is slow
// (minutes on consumer hardware) and the call blocks until the clip is
// ready; callers bound it with a context deadline.
Generate(ctx context.Context, req Request, opts ...Option) (*Result, error)
}
// ModelOption configures a Model at construction time (Provider.VideoModel).
// Reserved for future per-model settings; present now so the interface is
// forward-compatible.
type ModelOption func(*ModelConfig)
// ModelConfig carries per-model construction settings.
type ModelConfig struct{}
// ApplyModelOptions folds options into a config.
func ApplyModelOptions(opts []ModelOption) ModelConfig {
var cfg ModelConfig
for _, opt := range opts {
opt(&cfg)
}
return cfg
}
// Provider mints video Models bound to one backend. It mirrors llm.Provider
// but for video generation.
type Provider interface {
// Name is the registry identifier for the provider.
Name() string
// VideoModel returns a Model bound to the given id (passed through to the
// backend verbatim; no catalog validation).
VideoModel(id string, opts ...ModelOption) (Model, error)
}
+49
View File
@@ -0,0 +1,49 @@
package videogen
import "testing"
func TestApplyDoesNotMutateOriginal(t *testing.T) {
orig := Request{Prompt: "a cat"}
got := orig.Apply(
WithSize("1280x704"),
WithNumFrames(81),
WithFPS(16),
WithSteps(4),
WithGuidanceScale(1.0),
WithNegativePrompt("blurry"),
WithSeed(42),
WithInitImage(Image{MIME: "image/png", Data: []byte{1}}),
)
if orig.Size != "" || orig.NumFrames != 0 || orig.FPS != 0 || orig.Steps != nil ||
orig.GuidanceScale != nil || orig.NegativePrompt != "" || orig.Seed != nil || orig.InitImage != nil {
t.Fatalf("original mutated: %+v", orig)
}
if got.Size != "1280x704" || got.NumFrames != 81 || got.FPS != 16 {
t.Errorf("size/frames/fps = %q/%d/%d", got.Size, got.NumFrames, got.FPS)
}
if got.Steps == nil || *got.Steps != 4 {
t.Errorf("steps = %v", got.Steps)
}
if got.GuidanceScale == nil || *got.GuidanceScale != 1.0 {
t.Errorf("guidance = %v", got.GuidanceScale)
}
if got.NegativePrompt != "blurry" {
t.Errorf("negative = %q", got.NegativePrompt)
}
if got.Seed == nil || *got.Seed != 42 {
t.Errorf("seed = %v", got.Seed)
}
if got.InitImage == nil || got.InitImage.MIME != "image/png" {
t.Errorf("init image = %+v", got.InitImage)
}
}
func TestApplyModelOptions(t *testing.T) {
called := false
ApplyModelOptions([]ModelOption{func(*ModelConfig) { called = true }})
if !called {
t.Fatal("model option not applied")
}
}
+74
View File
@@ -0,0 +1,74 @@
package videogen
import "context"
// VideoUpscaleRequest asks a super-resolution backend (per-frame Real-ESRGAN
// style) to enlarge a clip. Zero values mean "backend default" (ADR-0025).
type VideoUpscaleRequest struct {
// Video is the encoded clip to upscale. Required. Carried as bytes
// (never a URL).
Video []byte
// MIME is the video MIME type (e.g. "video/mp4"); "" = let the backend
// sniff it.
MIME string
// Filename is the multipart filename hint some backends key their format
// detection on; "" derives one from MIME ("video.mp4") or falls back to
// "video".
Filename string
// Scale is the enlargement factor (2 or 4 on the reference backend);
// 0 = backend default.
Scale int
}
// VideoUpscaleOption mutates a VideoUpscaleRequest before it is sent.
type VideoUpscaleOption func(*VideoUpscaleRequest)
// WithVideoUpscaleScale sets the enlargement factor.
func WithVideoUpscaleScale(s int) VideoUpscaleOption {
return func(r *VideoUpscaleRequest) { r.Scale = s }
}
// Apply returns a copy of the request with all options applied.
func (r VideoUpscaleRequest) Apply(opts ...VideoUpscaleOption) VideoUpscaleRequest {
for _, opt := range opts {
opt(&r)
}
return r
}
// VideoUpscaler enlarges video clips frame by frame — the moving-picture
// sibling of imagegen.Upscaler.
type VideoUpscaler interface {
// UpscaleVideo returns the enlarged clip. Upscaling is slow (per-frame
// inference); bound the call with a context deadline.
UpscaleVideo(ctx context.Context, req VideoUpscaleRequest, opts ...VideoUpscaleOption) (*Result, error)
}
// VideoUpscalerModelOption configures a VideoUpscaler at construction time.
// Reserved for future per-model settings.
type VideoUpscalerModelOption func(*VideoUpscalerModelConfig)
// VideoUpscalerModelConfig carries per-model construction settings.
type VideoUpscalerModelConfig struct{}
// ApplyVideoUpscalerModelOptions folds options into a config.
func ApplyVideoUpscalerModelOptions(opts []VideoUpscalerModelOption) VideoUpscalerModelConfig {
var cfg VideoUpscalerModelConfig
for _, opt := range opts {
opt(&cfg)
}
return cfg
}
// VideoUpscaleProvider mints VideoUpscalers bound to one backend.
type VideoUpscaleProvider interface {
// Name is the registry identifier for the provider.
Name() string
// VideoUpscalerModel returns a VideoUpscaler bound to the given id
// (passed through to the backend verbatim; no catalog validation).
VideoUpscalerModel(id string, opts ...VideoUpscalerModelOption) (VideoUpscaler, error)
}