20 Commits
Author SHA1 Message Date
steveandClaude Opus 5 85561ab477 chore: ignore .claude/ (agent worktrees)
CI / Tidy (push) Successful in 9m23s
CI / Build & Test (push) Successful in 9m48s
A `git add -A` in a checkout with a worktree under .claude/worktrees/ records
it as a gitlink — a submodule pointer no clone can resolve. That happened in
gadfly today; this is the same one-line prevention, applied before it happens
here.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-12 18:58:35 -04:00
steve f837115a55 Merge pull request 'feat(qwen): Alibaba Qwen built-in over Model Studio's OpenAI-compatible mode' (#27)
CI / Tidy (push) Successful in 9m24s
CI / Build & Test (push) Successful in 11m10s
2026-08-12 21:03:34 +00:00
steveandClaude Opus 5 f8ced9c629 docs(progress): describe the shape this PR actually landed in
CI / Tidy (pull_request) Successful in 9m22s
CI / Build & Test (pull_request) Successful in 10m28s
The progress entry was written before four review rounds reshaped the change:
it credited openaiCompatScheme alone, listed the tests as six per-provider
cases, and mentioned a captureRT detail that has since moved. Rewritten to
match what merges — registerOpenAICompatBuiltin owning both halves,
envKeyForProvider as the single LLM_<NAME> definition, and the shared table
that every OpenAI-compat built-in is now checked against.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-12 16:57:33 -04:00
steveandClaude Opus 5 0760cf96d4 docs: gadfly round 4 — two doc-comment fixes
CI / Tidy (pull_request) Successful in 9m35s
CI / Build & Test (pull_request) Successful in 10m17s
DSN.Scheme's list named kimi but not qwen — the same sibling drift this PR
keeps finding, in a doc comment this time (I updated the README's scheme list
and not this one). Added qwen, and llama-swap/llama-swaps while there, since
they were missing too.

envKeyForProvider's example ran backwards: "LLM_M1 → m1" describes registry
naming, not what the function does. Now reads name → variable in one
direction.

Not taking the third: the const block's alignment is gofmt's own output
(gofmt -l is empty), and the uneven padding is forced by the doc comments that
split the block into alignment groups. glm-5.2 reached that same conclusion in
round 2 before flagging it here.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-12 16:53:05 -04:00
steveandClaude Opus 5 8670ed22be refactor: gadfly round 3 — one table owns the OpenAI-compat contract
CI / Tidy (pull_request) Successful in 9m21s
CI / Build & Test (pull_request) Successful in 10m23s
Three findings, and the first two are the same recurring shape.

envKeyForProvider (env.go) is now the single definition of the LLM_<NAME>
form. It lived in two places — lazy resolution in registry.go and the
missing-key hint in openaiCompatScheme — with a comment on the second asserting
it matched the first. A comment is not enforcement: if either had drifted, a
keyless DSN target would have named a variable that does nothing, and nothing
would have failed.

The kimi and qwen test files had become near-identical, which is round 1's
finding at the level above it: I deduped the fixtures, then left two parallel
suites asserting the same four things. They are now ONE table in
builtin_openaicompat_test.go — endpoint + credential, missing key fails closed
naming its own variable and never reaching the network, the name:// DSN
reaching another host, and a keyless DSN naming LLM_<NAME> instead of the
built-in's key. Adding an OpenAI-compat built-in is a table row that
immediately owes all four; builtin_kimi_test.go is deleted because the table
covers it. Only genuinely qwen-specific tests remain in the qwen file: the
reverse credential leak and the reasoning_effort wire claim ADR-0027 rests on.

Also trimmed ProviderQwen's doc comment, which restated the ADR-0027 rationale
already given at the registration site.

The break-check suite caught its own rot again — two mutations went stale when
these tests were renamed, and the landed-check reported them loudly instead of
passing them off as green. Now 9 cases, including one that drifts
envKeyForProvider to prove the shared helper is load-bearing. 9/9 apply and are
caught.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-12 16:40:53 -04:00
steveandClaude Opus 5 f1f2b653c3 refactor: gadfly round 2 — both halves of an OpenAI-compat built-in register together
CI / Tidy (pull_request) Successful in 9m25s
CI / Build & Test (pull_request) Successful in 9m50s
Same class of finding as round 1, one level in: I factored the DSN-scheme half
of the kimi/qwen duplication into openaiCompatScheme and left the eager
provider half copy-pasted, so a third built-in still had six lines to clone —
including both credential rules, which is exactly the pair you do not want
re-typed.

registerOpenAICompatBuiltin now installs both halves from one call. The rules
that matter hold by construction for every future caller: WithAPIKey passed
unconditionally (an unset key must not fall through to OPENAI_API_KEY), and
WithAPIKeyName naming that same variable in the 401 hint. Registering kimi and
qwen is now one line each.

Also fixed a cross-reference the ADR got wrong: Qwen's image-input caveat is
README matrix footnote ⁴, not ³ — ³ is kimi's. I wrote "³, shared with kimi"
in the ADR and then gave Qwen its own footnote in the README.

The break-check harness needed fixing before any of this could be trusted:
three of its mutations targeted lines this refactor moved, so they matched
nothing, the code was never broken, and the suite reported "test still passed"
— identical output to a test that genuinely misses the bug. Mutations are now
verified to have landed (sha before/after) and the suite fails loudly if one
doesn't. Two new cases cover the helper: dropping the unconditional WithAPIKey,
and dropping the scheme-half registration. 8/8 apply and are caught.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-12 16:25:38 -04:00
steveandClaude Opus 5 31d6b59356 refactor(test): gadfly round 1 — share the OpenAI-compat test fixtures
CI / Tidy (pull_request) Successful in 9m25s
CI / Build & Test (pull_request) Successful in 9m50s
Both findings were the same one, and both were fair: the PR that retires two
byte-identical DSN factories into openaiCompatScheme then copy-pasted the test
fixtures. qwenResponse was byte-identical to kimiResponse, and the single-key
env-lookup closure appeared three times in the new file (plus a fourth in the
kimi file, which neither reviewer was looking at).

Fixed for the class rather than for qwen: captureRT, the canned Chat
Completions body (now chatCompletionOK), and a new singleKeyEnv helper move to
builtin_openaicompat_test.go, owned by no single provider. The kimi tests adopt
them too, so the next OpenAI-compat built-in has nothing left to copy — the
same argument the production helper makes.

Also aligned the test model ids to the current Model Studio names
(qwen3.8-max / qwen3.7-plus), which the docs already cited. One reviewer called
those ids fictional and named the 2025 ones instead; they shipped 2026-08-03
and 2026-05-21 respectively, so that finding is stale model knowledge, not a
defect — but having tests and prose name the same models removes the smell that
prompted it. A dotted id also now proves it passes through verbatim.

Break-checked again after the refactor: all six mutations still fail their test.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-12 16:16:54 -04:00
steveandClaude Opus 5 02cd561eaf feat(qwen): Alibaba Qwen built-in over Model Studio's OpenAI-compatible mode
Gadfly review (reusable) / review (pull_request) Successful in 5m14s
Adversarial Review (Gadfly) / review (pull_request) Successful in 5m14s
CI / Tidy (pull_request) Successful in 9m24s
CI / Build & Test (pull_request) Successful in 9m53s
Adds the `qwen` built-in provider and the `qwen://` DSN scheme, keyed by
QWEN_API_KEY and defaulting to Model Studio's international host. Like kimi
(ADR-0026) it is `provider/openai` pointed elsewhere — no new client.

Model Studio serves the same models over two protocols, so the real decision
was which wire format to speak. ADR-0027 records why it is the OpenAI one:
down the anthropic client `ReasoningEffort` is ignored by design, structured
output rides the first-party `output_config.format` mechanism the shim does
not implement, and cached-token accounting reads Anthropic-only usage fields.
Each of those fails silently rather than loudly, which is what makes the
choice worth writing down. The shim stays reachable ad hoc via an
`anthropic://` DSN.

The kimi and qwen DSN factories were byte-identical, so they now share one
`openaiCompatScheme` helper: the "credential comes from the DSN token, and
the missing-key hint names LLM_<NAME>" rules hold by construction instead of
by copy.

Tests are hermetic and break-checked (all six fail on a deliberate mutation),
including the reverse credential leak — a visible QWEN_API_KEY must not
authenticate the openai built-in — and reasoning_effort asserted on the wire
body, which is the ADR's load-bearing claim.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-12 16:07:41 -04:00
steve e779169416 Merge pull request 'feat(videogen): LastImage — pin the trailing keyframe (first-last-frame-to-video)' (#26) from feat/videogen-last-frame into main
CI / Tidy (push) Successful in 9m23s
CI / Build & Test (push) Successful in 9m47s
2026-08-08 07:10:26 +00:00
steveandClaude Opus 5 5994d96921 refactor(llamaswap): drop initImageFilename — one caller left, and it was a rename of imageFilename
CI / Tidy (pull_request) Successful in 9m40s
CI / Build & Test (pull_request) Successful in 11m38s
The 2/4 finding is right: after writeImagePart started passing an explicit
filename stem, initImageFilename had no caller in video.go, and its
"conditioning frame" doc no longer described its one remaining user
(lipsync.go's avatar image). A one-line wrapper that survives only to be
misdescribed is not indirection worth keeping.

lipsync now calls imageFilename(mime, "frame") directly, and imageFilename's
doc lists the real bases — including WHY the video keyframes need distinct
ones: a backend that stages uploads by filename would otherwise have the
second overwrite the first.

Not taken: consolidating the first/last-frame rationale to a single canonical
site. The copies address different readers — the wire encoding (provider), the
contract's undetectable-support caveat (videogen), and the mode table (README)
— and last round's finding was a doc pointing at a note that did not exist.
Trading duplication for cross-references is what produced that.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01PLjgrxvHjm1sJgUu9zBPH9
2026-08-08 03:06:21 -04:00
steveandClaude Opus 5 588e092465 docs(videogen): gadfly — README FL2V section, and stop pointing at a note that does not exist
CI / Tidy (pull_request) Successful in 9m25s
CI / Build & Test (pull_request) Successful in 9m50s
- README documented only t2v/i2v. Now a table of the four keyframe
  combinations, plus the undetectable-support caveat, which is the one thing a
  caller cannot work out for itself.
- The LastImage doc comment said "see the note on LastImage support in
  provider/llamaswap" — there was no such note. A pointer to something that
  does not exist is worse than no pointer; the comment is now self-contained.
- Generate's doc described only input_reference; it now names
  input_reference_last and explains why an unsupporting backend returns a clip
  rather than an error.

The 2/4 finding (writeImagePart reusing the "frame" base for both parts) was
already fixed in dbc9689 — from the receiving end, where the consequence is
concrete rather than stylistic: ComfyUI stages uploads by FILENAME with
overwrite=true, so a shared name means the second clobbers the first and both
keyframes resolve to one image.

Not taken: initImageFilename's name is no longer misleading (writeImagePart
stopped calling it), and it is still used by lipsync.go so it is not dead.
The empty-LastImage test stays standalone — it mirrors the existing standalone
empty-InitImage coverage rather than a table this file does not have.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01PLjgrxvHjm1sJgUu9zBPH9
2026-08-08 02:59:12 -04:00
steveandClaude Opus 5 dbc96898ab fix(videogen): distinct FILENAMES for the two keyframes, not just distinct field names
CI / Tidy (pull_request) Successful in 9m25s
CI / Build & Test (pull_request) Successful in 10m28s
Caught while writing the receiving end. Distinct multipart field names are not
sufficient: backends stage an uploaded frame under a name derived from the
FILENAME, and our own ComfyUI shim posts to /upload/image with overwrite=true.
Both parts were sending initImageFilename(mime) — literally "frame.png" for
each — so the second upload would have clobbered the first and BOTH keyframe
inputs would have resolved to the same stored image.

The failure mode is the worst kind: a clip pinned at both ends to the same
frame renders cleanly, returns 200, and looks like the feature not working
rather than like a bug. Nothing upstream or downstream would report a fault.

writeImagePart now takes the filename stem (frame / frame_last), and the test
asserts the two arrive under different filenames.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01PLjgrxvHjm1sJgUu9zBPH9
2026-08-08 02:50:43 -04:00
steveandClaude Opus 5 44fcfbb273 feat(videogen): LastImage — pin the trailing keyframe (first-last-frame-to-video)
Gadfly review (reusable) / review (pull_request) Successful in 3m41s
Adversarial Review (Gadfly) / review (pull_request) Successful in 3m41s
CI / Tidy (pull_request) Successful in 9m26s
CI / Build & Test (pull_request) Successful in 9m48s
videogen.Request gains LastImage alongside InitImage, so one Request covers
t2v, i2v and FL2V without a mode flag. With InitImage it pins both ends of the
clip; alone it pins the destination and lets the backend invent the approach.

The llamaswap provider sends it as a SEPARATE `input_reference_last` part
rather than a second `input_reference`. Multipart permits repeated names, but
then which frame is first and which is last depends on part ORDER — an
ordering contract invisible in the payload, that nothing notices breaking. A
backend that does not know the new name ignores the part, the same degradation
as any other unknown field.

Both parts go through one writeImagePart helper so their encoding cannot
drift, and an empty LastImage is rejected up front exactly as InitImage
already is.

Support is per-model and deliberately NOT advertised in this contract: a
backend that ignores a trailing keyframe returns an ordinary clip, which is
indistinguishable from success. The doc comment says so, because a caller that
needs to know whether the pin took effect has to establish that out of band —
and the mort side gates on a convar for exactly this reason.

Motivated by mort's #1567 (long-form video): with both ends pinned, drift
becomes structurally bounded inside each shot instead of compounding across an
autoregressive chain.

Tests break-checked: sending the last frame under the shared name fails both
the distinct-name assertion and the last-alone case.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01PLjgrxvHjm1sJgUu9zBPH9
2026-08-08 02:47:39 -04:00
steve 203895696c Merge pull request 'fix(agent): recover the front-loaded answer over a summary closer' (#25) from fix/finalize-summary-closer into main
CI / Tidy (push) Successful in 9m22s
CI / Build & Test (push) Successful in 9m44s
2026-08-06 00:39:39 +00:00
steveandClaude Fable 5 1bbbdaa1e5 refactor(agent): gadfly round 2 — shared leadingMarkers, explicit mode, comment altitude
CI / Tidy (pull_request) Successful in 9m24s
CI / Build & Test (pull_request) Successful in 9m52s
All tidiness, no behavior change: the leading-marker class is one shared
constant for citationLabelRe and summaryCloserRe (hand-copying it is how
'+' went missing the first time); the deliberate 'all' duplication across
summaryCopulas/summaryArticle is now stated at both sites; the weak-final
switch case assigns modeBackRef explicitly; test comments state the
constraint they guard instead of which reviewer asked for them.

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-08-05 20:29:19 -04:00
steveandClaude Fable 5 21b4775d16 fix(agent): gadfly round 1 — user-boundary scan, back-ref precedence, regex legibility
CI / Tidy (pull_request) Successful in 9m39s
CI / Build & Test (pull_request) Successful in 10m5s
Two behavioral fixes from the review:

- modeSummary's backward scan now stops at the most recent user message.
  With the dwarf ratio rejecting the current turn's 1x-3x answer, the old
  unbounded scan could walk into WithHistory content and resurrect a stale
  answer to a DIFFERENT question — strictly worse than keeping the closer
  (opus, correctness). Other modes keep their historical unbounded scan.
- A terminal matching BOTH the ack shape and a back-reference is now
  classified back-ref: it carries no answer content, so the looser bar is
  the right one (opus, error-handling).

Plus the nits: summaryCloserRe assembled from named fragments, the leading
marker class gains '+' (parity with citationLabelRe), verb-first form takes
'all the', dwarf ratio hoisted into one named local, and the 151-vs-153
char/byte comment inaccuracy corrected.

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-08-05 20:17:56 -04:00
steveandClaude Fable 5 127966bb3a fix(agent): recover the front-loaded answer over a summary closer
CI / Build & Test (pull_request) Successful in 9m49s
CI / Tidy (pull_request) Successful in 10m28s
Gadfly review (reusable) / review (pull_request) Successful in 11m49s
Adversarial Review (Gadfly) / review (pull_request) Successful in 11m49s
A third degenerate terminal shape from the glm-5.2 cite pattern: the model
front-loads its full answer into the cite-call turn, then closes with a
bookkeeping ack plus a one-line compression ("Citations are logged. Short
version: ..."). mort run b3cb9ee9 delivered 151 chars of a 2,089-char
answer this way — the closer was neither a back-reference (over the 120
cap, no back-ref phrase) nor a citations addendum (no label-colon, no
links), so finalOutput let it stand.

isSummaryCloser keys on the ack sentence alone (the verb must end the
sentence, so prose about citations never matches; a compression marker
without the ack is deliberately out of scope), and the new modeSummary
recovery bar makes the 3x dwarf ratio mandatory at every length: unlike a
back-reference this closer carries real answer content, so it is only
displaced by the clearly-fuller original it compressed.

The citations/back-ref bool becomes a three-way recoveryMode; existing
behavior for both old modes is unchanged.

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-08-05 08:47:40 -04:00
steve 0bd14e01b3 Merge pull request 'feat(faceswap): report whether the likeness actually transferred' (#24) from feat/imagegen-faceswap into main
CI / Tidy (push) Successful in 9m31s
CI / Build & Test (push) Successful in 10m38s
2026-07-31 21:51:12 +00:00
steve 6995a8dee1 feat(faceswap): expose yaw on enumeration too
Gadfly review (reusable) / review (pull_request) Successful in 4m54s
Adversarial Review (Gadfly) / review (pull_request) Successful in 4m55s
CI / Tidy (pull_request) Successful in 9m23s
CI / Build & Test (pull_request) Successful in 9m48s
ListFaces now carries head yaw, so a caller choosing WHICH face to swap can
see the thing that decides whether the swap will read — not only learn it
afterwards from the swap report. In the run that prompted this the target's
three faces sat at -82, -8 and -11 degrees; only the first was hopeless, and
nothing in a bounding box said so.
2026-07-31 17:50:27 -04:00
steve ae2615ca68 feat(faceswap): carry the measured outcome, not just the image
A face swap always returns an image and always looks like success. Whether the
likeness actually transferred is a different question, and until now nothing in
the response answered it — so a caller wanting to know went and asked a vision
model instead. That is wrong in precisely the cases that matter: shown a jogger
in a Georgetown cap holding McDonald's cups, a VLM answers "Bill Clinton"
whoever's face is on him. In the run that prompted this it reported failure on
six consecutive CORRECT swaps (measured afterwards at 0.79-0.84 cosine), and
the caller burned 21 minutes chasing a problem that did not exist.

Result.SwappedFaces now carries, per replaced face: pixel size, the target
image's dimensions, head yaw, and cosine similarity between the source face and
the face actually present in the output.

Yaw and FractionOfImage are the two that explain the complaint. The swap in
question replaced a 138px face in a 1010px-wide photo — 14% of the width,
correct and invisible at a glance — and elsewhere a face turned -82 degrees,
where the features carrying identity are edge-on and any swap reads as a
generic person. Same code on a 168px face in a 385px picture (44%, yaw 2) is
unmistakable. None of that was inferable from a bounding box.

Typed on Result rather than stuffed into Raw: a caller has to act on this, and
a value reachable only by type-asserting an `any` is one nobody finds in time.

doRawHeaders is doRaw with the whole header instead of only Content-Type; doRaw
delegates to it, so the other 25 call sites are untouched and there is still
one place where the status check and the size cap live.

A missing or malformed header yields nil, not an error — an older shim sends no
header, and a swap that produced a good image must not fail because the
diagnostics beside it were unreadable. Covered for absent/garbage/wrong-type,
and the parse is break-checked.
2026-07-31 17:49:27 -04:00
23 changed files with 1348 additions and 266 deletions
+1
View File
@@ -7,6 +7,7 @@ OLLAMA_API_KEY=your-ollama-cloud-key-here
# Built-in provider keys (each optional; only needed for the providers you use).
#OPENAI_API_KEY=sk-...
#KIMI_API_KEY=sk-... # Moonshot AI (Kimi); provider name "kimi"
#QWEN_API_KEY=sk-... # Alibaba Model Studio (Qwen); provider name "qwen"
#ANTHROPIC_API_KEY=sk-ant-...
#GOOGLE_API_KEY=...
+3
View File
@@ -28,3 +28,6 @@ go.work.sum
# macOS
.DS_Store
# Local worktrees created for agent work — never part of the repo.
.claude/
+41 -6
View File
@@ -122,6 +122,7 @@ Chains are health-tracked per target:
|----------|-----------|-------------|------------------|
| OpenAI (+compatible) | `openai` | `OPENAI_API_KEY` | https://api.openai.com/v1 |
| Kimi (Moonshot AI) | `kimi` | `KIMI_API_KEY` | https://api.moonshot.ai/v1 |
| Qwen (Alibaba) | `qwen` | `QWEN_API_KEY` | https://dashscope-intl.aliyuncs.com/compatible-mode/v1 |
| Anthropic (+compatible) | `anthropic` | `ANTHROPIC_API_KEY` | https://api.anthropic.com |
| Google (Gemini) | `google` | `GOOGLE_API_KEY` / `GEMINI_API_KEY` | Gemini API (official SDK) |
| Ollama Cloud | `ollama-cloud` | `OLLAMA_API_KEY` | https://ollama.com |
@@ -134,6 +135,19 @@ the openai client (like llama-swap). The `kimi` built-in defaults to the
international endpoint; reach the China endpoint (or any other host) with a
`kimi://` DSN, e.g. `LLM_KCN=kimi://[email protected]/v1`.
Qwen is the same shape: Alibaba Model Studio's OpenAI-compatible mode, reusing
the openai client. The `qwen` built-in defaults to the international
(Singapore) host; reach the China host or a workspace-scoped regional one with
a `qwen://` DSN, e.g.
`LLM_QCN=qwen://[email protected]/compatible-mode/v1`. Model Studio
also fronts the same models with an Anthropic-compatible `/v1/messages` shim —
majordomo does **not** use it, because on that surface `reasoning_effort` is
dropped, `Request.Schema` stops being enforced, and cached-token accounting
disappears; see [ADR-0027](docs/adr/0027-qwen-builtin.md). Two Alibaba-side
quirks are worth knowing: thinking is on by default for some models (e.g.
`qwen3.7-plus`), and the Qwen3 open-source models require streaming while
thinking, so buffered `Generate` calls want a Max/Plus model.
OpenAI-compatible / Anthropic-compatible endpoints: construct the provider
with a name and base URL and register it —
@@ -165,7 +179,7 @@ m, _ := reg.Parse("m5/qwen3:30b,m1/qwen3:30b,thinking")
```
DSN format: `scheme://[token@]host[/path]`, scheme ∈ `foreman`, `ollama`,
`ollama-cloud`, `openai`, `kimi`, `anthropic`, `google`/`gemini`, `llama-swap`,
`ollama-cloud`, `openai`, `kimi`, `qwen`, `anthropic`, `google`/`gemini`, `llama-swap`,
`llama-swaps`, or any scheme you add with `RegisterScheme`. The token is the
credential (bearer token / API key); the base URL is always `https://host[/path]`
— except `llama-swap`, which builds `http://host[:port]` since it's local-first
@@ -272,17 +286,30 @@ tr, err := tm.Transcribe(ctx, audio.TranscriptionRequest{
voices, err := ls.ListVoices(ctx, "kokoro") // []string of voice ids
```
## Video: text-to-video + image-to-video
## Video: text-to-video, image-to-video, first-last-frame
Video generation lives in the `videogen` package (ADR-0019), mirroring
imagegen/audio: one small `Model` contract, zero values mean backend
defaults, bytes in/out. Text-to-video and image-to-video are one surface
a nil `InitImage` is a pure text prompt; setting it conditions generation
on that frame (hybrid checkpoints like Wan 2.2 TI2V serve both). First
backend: llama-swap (blocking `/v1/videos/sync`, vLLM-Omni style — the
defaults, bytes in/out. All modes are one surface, selected by which
keyframes are set rather than by a mode flag:
| `InitImage` | `LastImage` | mode |
|---|---|---|
| nil | nil | text-to-video |
| set | nil | image-to-video (hybrid checkpoints like Wan 2.2 TI2V serve both) |
| set | set | first-last-frame — both ends pinned |
| nil | set | pin the destination, model invents the approach |
First backend: llama-swap (blocking `/v1/videos/sync`, vLLM-Omni style — the
response body is the encoded clip, so `Result` carries a single `Video`).
Generation runs for minutes; bound the call with a context deadline.
**`LastImage` support is per-model and cannot be detected.** A backend that
does not understand a trailing keyframe ignores the part and returns an
ordinary clip — indistinguishable from success. There is no capability bit,
because the contract has no way to learn one, so a caller depending on the
pin must establish support out of band.
```go
vm, _ := ls.VideoModel("videogen-wan22-5b")
res, err := vm.Generate(ctx, videogen.Request{Prompt: "a cat surfing"},
@@ -407,6 +434,7 @@ to build one.
|----------------------|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| OpenAI (+compatible) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Kimi (Moonshot AI) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅³ | ✅ |
| Qwen (Alibaba) | ✅ | ✅ | ✅ | ✅ | ✅⁴ | ✅⁴ | ✅ |
| Anthropic (+compat) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Google (Gemini) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Ollama Cloud | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
@@ -431,6 +459,13 @@ probe and management methods on `*llamaswap.Provider`.
level; whether a call succeeds depends on the Moonshot model — only the vision
variants (e.g. `moonshot-v1-8k-vision-preview`) accept images.
⁴ Qwen also reuses the openai client (ADR-0027), so both columns are present at
the client level and gated by the Model Studio model you name: `json_schema`
structured output is on the Max/Plus families, image inputs on the `qwen-vl-*`
/ `qwen3-vl-*` models. `reasoning_effort` rides through as a top-level field —
one reason the built-in speaks OpenAI-compat rather than Model Studio's
Anthropic-compat shim.
Notes: Ollama has no native tool_choice — `"none"` drops the tools;
`"required"`/named choices are best-effort ignored there. Ollama Cloud
ignores the `format` field (verified live), so the provider also states
+145 -25
View File
@@ -14,8 +14,8 @@ import (
// their answer to the final, tool-free turn. But some models — notably several
// open-weight ones — "front-load" their full answer into an earlier turn that
// ALSO calls a tool (e.g. answer text alongside a citation call), then close
// with a degenerate terminal turn that is not itself the answer. Two shapes are
// recovered from the transcript (zero extra model calls):
// with a degenerate terminal turn that is not itself the answer. Three shapes
// are recovered from the transcript (zero extra model calls):
//
// - a trivial back-reference ("(Already answered above.)", "see above", …):
// the real answer sits earlier, so recover it and DISCARD the worthless
@@ -25,25 +25,44 @@ import (
// glm-5.2 "cite" pattern behind mort issue #1418). The citations are real,
// useful content — unlike a back-reference — so recover the prior answer and
// KEEP the citations, appended below it.
// - a bookkeeping closer ("Citations are logged. Short version: …"): the
// model acknowledged the citation round and compressed the answer it had
// already written into a one-liner (mort run b3cb9ee9 — a 2,089-char answer
// shrank to a 153-byte closer at delivery). The compression is strictly
// poorer than the front-loaded answer, so recover the prior turn and
// DISCARD the closer — but only when the prior turn clearly dwarfs it,
// because unlike a back-reference this closer DOES carry answer content
// (see modeSummary).
//
// A citations addendum is tested first and wins over the back-reference test (a
// short terminal can be both), so its links are never discarded. When the
// terminal text stands on its own it is returned unchanged; when it is
// degenerate but nothing better can be recovered, it is returned as-is (a bare
// sources list still beats nothing).
// A citations addendum is tested first and wins over the other two (a short
// terminal can match more than one shape), so its links are never discarded.
// The back-reference test wins over the summary-closer test: a terminal
// matching both ("Citations are logged. As I said above…") carries no answer
// content of its own, so the looser back-ref recovery bar — not the summary
// closer's dwarf ratio — is the right one. When the terminal text stands
// on its own it is returned unchanged; when it is degenerate but nothing
// better can be recovered, it is returned as-is (a compressed answer still
// beats nothing).
//
// msgs must already include the terminal assistant message as its last element
// (the loop appends it before calling this); terminal is that message's text.
func finalOutput(msgs []llm.Message, terminal string) string {
citations := isCitationsOnly(terminal)
if !citations && !isWeakFinal(terminal) {
mode := modeBackRef
switch {
case isCitationsOnly(terminal):
mode = modeCitations
case isWeakFinal(terminal):
mode = modeBackRef
case isSummaryCloser(terminal):
mode = modeSummary
default:
return terminal
}
rec, ok := lastSubstantiveAssistantText(msgs, terminal, citations)
rec, ok := lastSubstantiveAssistantText(msgs, terminal, mode)
if !ok {
return terminal
}
if citations {
if mode == modeCitations {
// Preserve the citations addendum below the recovered answer, unless the
// recovered turn already carries it (guards against a duplicate sources
// block when the front-loaded turn included its own citations). The
@@ -65,11 +84,64 @@ func stripURLAngles(s string) string {
return strings.NewReplacer("<", "", ">", "").Replace(s)
}
// recoveryMode selects the bar a prior assistant turn must clear to replace
// the terminal turn (see isSubstantiveAnswer) and what finalOutput does with
// the terminal once recovery succeeds.
type recoveryMode int
const (
// modeBackRef: the terminal is empty or a pure back-reference — worthless
// on its own, so any real prior answer replaces it and it is discarded.
modeBackRef recoveryMode = iota
// modeCitations: the terminal is a sources-only addendum — not a rival
// answer, so the dwarf ratio is skipped and the addendum is kept, appended
// below the recovered answer.
modeCitations
// modeSummary: the terminal acknowledges the citation round and may carry
// a short compression of the front-loaded answer. Unlike a back-reference
// it DOES contain answer content, so it is only replaced when a prior turn
// clearly dwarfs it — the ratio is mandatory at every length, the recovery
// scan stops at the most recent user message (a compression can only be of
// THIS turn's answer; never resurrect one from an earlier question), and
// the closer is discarded (its content is a strict subset of what it
// replaced).
modeSummary
)
// backRefRe matches a terminal turn that merely points back to an earlier
// message instead of stating the answer ("(Already answered above.)",
// "see above", "as I said", ...).
var backRefRe = regexp.MustCompile(`(?i)(already answered|see above|as (i )?(said|mentioned|stated|noted)|answered (that )?above|per my (previous|earlier))`)
// summaryCloserRe matches a terminal turn that OPENS with a bookkeeping
// acknowledgment of the citation round — "Citations are logged.", "Sources
// cited.", "Logged the citations." — the shape a model produces when it
// front-loaded its answer into an earlier cite-call turn and closes by
// acknowledging the tool results, often followed by a "Short version: …"
// compression of the answer it already wrote. The ack clause must end at a
// sentence terminator ([.!]) DIRECTLY after the verb: "The citations are
// recorded in the court transcript…" is a real answer about citations, not
// bookkeeping, and must never match. A compression marker without the ack
// ("Short version: no.") is deliberately out of scope — a user who asked for
// brevity would be answered with exactly that shape, and misclassifying it
// would hijack a legitimate answer; an unmatched closer merely keeps today's
// behavior (fail closed). Assembled from named fragments so the alternations
// stay legible and extendable.
const (
summaryPreface = `((done|all set|ok(ay)?)[\s,.!:—-]+)?` // optional "Done —" style opener
summaryNouns = `(citations?|sources?|references?|claims?)`
// "all" appears here AND in summaryArticle on purpose: as a quantifier
// between noun and verb ("Citations all logged.") and as a determiner
// before the noun ("All claims cited.", "Logged all the citations.").
summaryCopulas = `((are|were|have\s+been|all)\s+)*`
summaryVerbs = `(logged|recorded|cited|saved|noted|captured|filed)`
summaryArticle = `((all|the)\s+)*` // star, not ?: "Logged all the citations."
)
var summaryCloserRe = regexp.MustCompile(`(?i)^` + leadingMarkers + summaryPreface +
`(` + summaryArticle + summaryNouns + `\s+` + summaryCopulas + summaryVerbs +
`|logged\s+` + summaryArticle + summaryNouns + `)[.!]`)
// preambleRe matches intent-announcing prefixes ("Let me search...", "I'll
// check...") so a preamble is never mistaken for the answer during recovery.
var preambleRe = regexp.MustCompile(`(?i)^(let me|let'?s|i'?ll|i will|first[, ]|sure[,. ]|okay[,. ]|on it|checking)`)
@@ -83,7 +155,15 @@ var preambleRe = regexp.MustCompile(`(?i)^(let me|let'?s|i'?ll|i will|first[, ]|
// the colon/dash separator. Anchored at ^ so a normal answer that merely
// mentions "sources" mid-sentence, or ends with a "Sources:" section AFTER its
// prose, is never matched.
var citationLabelRe = regexp.MustCompile(`(?i)^[\s>#*_+-]*(sources?|references?|citations?|works cited|further reading)\b[\s*_]*[:\-—]`)
var citationLabelRe = regexp.MustCompile(`(?i)^` + leadingMarkers +
`(sources?|references?|citations?|works cited|further reading)\b[\s*_]*[:\-—]`)
// leadingMarkers tolerates markdown noise before a label: emphasis (*, _),
// list (-, +, *), block-quote (>), and ATX-heading (#) markers, with their
// whitespace. Shared by citationLabelRe and summaryCloserRe so the two
// classifiers cannot drift apart (the first draft of the summary class
// dropped '+' by hand-copying this set).
const leadingMarkers = `[\s>#*_+-]*`
// linkRe matches a whole markdown link "[label](url)" or a bare URL. Used both
// to require that a citations terminal carries at least one link and to strip
@@ -114,6 +194,12 @@ const (
// be at most len/N of the whole, so a prose answer that merely opens with
// "Source:" and cites a URL mid-sentence is not mistaken for a bare list.
citationDominatedDivisor = 3
// summaryCloserMaxChars bounds a summary closer: room for the ack sentence
// plus a couple of compression sentences (the b3cb9ee9 closer was 153
// bytes — Go len(), which is what every threshold here compares). Beyond
// this the "short version" is substantial enough that replacing it risks
// losing content the front-loaded turn never had.
summaryCloserMaxChars = 300
)
// isWeakFinal reports whether a terminal turn's text fails to stand on its own
@@ -152,14 +238,39 @@ func isCitationsOnly(s string) bool {
return len(residue) <= len(t)/citationDominatedDivisor
}
// isSummaryCloser reports whether a terminal turn is a bookkeeping closer: it
// opens with a complete "citations are logged"-style ack sentence (see
// summaryCloserRe) and is short enough that whatever follows the ack can only
// be a compression of an earlier, fuller answer. Whether that fuller answer
// actually exists is modeSummary's job — the dwarf ratio in
// isSubstantiveAnswer keeps a matching closer in place when nothing earlier
// clearly outweighs it.
func isSummaryCloser(s string) bool {
t := strings.TrimSpace(s)
if t == "" || len(t) > summaryCloserMaxChars {
return false
}
return summaryCloserRe.MatchString(t)
}
// lastSubstantiveAssistantText scans msgs newest→oldest (skipping the terminal
// turn and empty tool-only turns) for the most recent assistant turn whose text
// reads like a real answer. citations selects the recovery bar (see
// reads like a real answer. mode selects the recovery bar (see
// isSubstantiveAnswer). Returns ("", false) when nothing qualifies.
func lastSubstantiveAssistantText(msgs []llm.Message, terminal string, citations bool) (string, bool) {
func lastSubstantiveAssistantText(msgs []llm.Message, terminal string, mode recoveryMode) (string, bool) {
tt := strings.TrimSpace(terminal)
for i := len(msgs) - 1; i >= 0; i-- {
m := msgs[i]
if mode == modeSummary && m.Role == llm.RoleUser {
// A summary closer compresses THIS turn's front-loaded answer, so
// the scan must not cross into an earlier question: once the dwarf
// ratio has rejected the current turn's text, walking further back
// would resurrect a stale answer to a DIFFERENT question — strictly
// worse than keeping the closer. (A mid-run steer message is also a
// user-role boundary; recovery then fails closed, which is fine.)
// The other modes keep their historical unbounded scan.
break
}
if m.Role != llm.RoleAssistant {
continue
}
@@ -167,7 +278,7 @@ func lastSubstantiveAssistantText(msgs []llm.Message, terminal string, citations
if txt == "" || txt == tt {
continue // the terminal turn itself, or an empty tool-only turn
}
if isSubstantiveAnswer(txt, tt, citations) {
if isSubstantiveAnswer(txt, tt, mode) {
return txt, true
}
}
@@ -177,20 +288,29 @@ func lastSubstantiveAssistantText(msgs []llm.Message, terminal string, citations
// isSubstantiveAnswer reports whether txt (a prior assistant turn) reads like a
// real answer rather than a preamble, relative to the terminal text.
//
// A sufficiently long turn (>= recoverMinChars) is accepted unconditionally: a
// multi-hundred-char turn is an answer even when it opens conversationally
// ("Sure, here's…", "Let me explain: …"), so the preamble filter is NOT applied
// to it — applying it there would drop a legitimate long front-loaded answer.
// Only in the borderline band does a turn have to clear a floor, not read like a
// short planning preamble ("Let me look that up…"), and — unless the terminal is
// a citations addendum (not a rival answer, so its length is irrelevant) — also
// clearly dwarf the terminal.
func isSubstantiveAnswer(txt, terminal string, citations bool) bool {
// modeSummary demands the dwarf ratio FIRST, at every length: a summary closer
// carries a real (compressed) answer, so replacing it is only justified when
// the prior turn is clearly the fuller original it was compressed from.
//
// A sufficiently long turn (>= recoverMinChars) is otherwise accepted
// unconditionally: a multi-hundred-char turn is an answer even when it opens
// conversationally ("Sure, here's…", "Let me explain: …"), so the preamble
// filter is NOT applied to it — applying it there would drop a legitimate long
// front-loaded answer. Only in the borderline band does a turn have to clear a
// floor, not read like a short planning preamble ("Let me look that up…"),
// and — for modeBackRef only — also clearly dwarf the terminal (a citations
// addendum is not a rival answer, so its length is irrelevant; a summary
// closer already proved the ratio above).
func isSubstantiveAnswer(txt, terminal string, mode recoveryMode) bool {
dwarfs := len(txt) >= recoverRatio*len(terminal)
if mode == modeSummary && !dwarfs {
return false
}
if len(txt) >= recoverMinChars {
return true
}
if len(txt) < recoverFloorChars || preambleRe.MatchString(txt) {
return false
}
return citations || len(txt) >= recoverRatio*len(terminal)
return mode != modeBackRef || dwarfs
}
+179 -1
View File
@@ -72,6 +72,46 @@ func TestIsCitationsOnly(t *testing.T) {
}
}
// b3cb9ee9Closer is the verbatim terminal turn from mort run b3cb9ee9: a
// 2,089-char answer was front-loaded into the cite-call turn and this 153-byte
// compression (151 runes — the em dash is 3 bytes, and byte length is what the
// thresholds compare) was all that got delivered.
const b3cb9ee9Closer = "Citations are logged. Short version: the bulk of that ~$64M was AIPAC and dark-money super PACs, not the party committees — and it still wasn't enough."
func TestIsSummaryCloser(t *testing.T) {
cases := []struct {
name string
in string
want bool
}{
{"b3cb9ee9-verbatim", b3cb9ee9Closer, true},
{"ack-only", "Citations are logged.", true},
{"ack-no-copula", "Citations logged.", true},
{"claims-cited", "All claims cited.", true},
{"verb-first", "Logged the citations.", true},
{"done-prefix", "Done — citations logged.", true},
{"ack-then-tldr", "Sources have been recorded! TL;DR: the GPU was the bottleneck.", true},
{"references-noted", "References noted. In short: yes, it ships Tuesday.", true},
{"plus-list-marker", "+ Citations are logged.", true},
{"logged-all-the", "Logged all the citations.", true},
{"empty", "", false},
{"ack-continues-midsentence", "The citations are recorded in the court transcript, which shows the filing dates.", false},
{"ack-verb-then-clause", "Citations are logged in Zotero whenever you click the save button.", false},
{"compression-without-ack", "Short version: yes.", false}, // deliberately out of scope
{"mentions-citations-midsentence", "The paper's citations are what got it retracted.", false},
{"crisp-number", "42", false},
{"over-cap", "Citations are logged. " + strings.Repeat("The long version has many more details worth keeping. ", 6), false}, // >300: too substantial to replace
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
if got := isSummaryCloser(c.in); got != c.want {
t.Errorf("isSummaryCloser(%q) = %v, want %v", c.in, got, c.want)
}
})
}
}
func asst(text string, tools ...llm.ToolCall) llm.Message {
m := llm.Message{Role: llm.RoleAssistant}
if text != "" {
@@ -83,7 +123,8 @@ func asst(text string, tools ...llm.ToolCall) llm.Message {
func TestFinalOutput(t *testing.T) {
cite := []llm.ToolCall{{ID: "c1", Name: "cite", Arguments: json.RawMessage(`{}`)}}
longAnswer := strings.TrimSpace(strings.Repeat("Free group calls are capped at sixty minutes. ", 6)) // >200
longAnswer := strings.TrimSpace(strings.Repeat("Free group calls are capped at sixty minutes. ", 6)) // >200
hugeAnswer := strings.TrimSpace(strings.Repeat("Free group calls are capped at sixty minutes. ", 12)) // >3x the b3cb9ee9 closer
// A sources/citations-only terminal — the glm-5.2 "cite" shape behind mort
// issue #1418: the prose answer was front-loaded into the tool-call turn and
// the terminal turn carried only the citations.
@@ -103,6 +144,9 @@ func TestFinalOutput(t *testing.T) {
// A >=200-byte real answer that merely OPENS with a conversational word
// ("Sure,"). The preamble filter must NOT veto it (gadfly regression guard).
longConversationalAnswer := "Sure, here's the rundown: it currently sells for about $2,700 used on eBay, typically $2,400 to $2,900 depending on condition and bundle, with the sealed Founders Edition commanding the top of that range while used AIB cards go a bit lower."
// Matches BOTH the summary ack and backRefRe, within the 120-byte weak cap,
// and long enough (>~92 bytes) that longAnswer would fail the summary bar.
bothMatchCloser := "Citations are logged. As I mentioned above, the full detail on the money sources is in my earlier message."
tests := []struct {
name string
@@ -256,6 +300,109 @@ func TestFinalOutput(t *testing.T) {
terminal: sources,
want: longConversationalAnswer + "\n\n" + sources,
},
{
// The b3cb9ee9 shape: full answer front-loaded into the cite turn,
// then a summary closer. The closer is discarded — its content is a
// strict compression of the recovered answer.
name: "summary closer discarded when the front-loaded answer dwarfs it",
msgs: []llm.Message{
llm.UserText("where did the $64M come from?"),
asst(hugeAnswer, cite...),
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
asst(b3cb9ee9Closer),
},
terminal: b3cb9ee9Closer,
want: hugeAnswer,
},
{
// The dwarf ratio is mandatory for a summary closer at EVERY length:
// a prior turn that is longer but not clearly the fuller original
// (here ~275 chars vs a 151-char closer, under the 3x bar) must not
// displace a closer that carries real answer content.
name: "summary closer kept when the prior turn does not dwarf it",
msgs: []llm.Message{
llm.UserText("q?"),
asst(longAnswer, cite...),
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
asst(b3cb9ee9Closer),
},
terminal: b3cb9ee9Closer,
want: b3cb9ee9Closer,
},
{
// An ack-only closer ("Citations are logged.") is tiny, so even a
// modest front-loaded answer clears the ratio and replaces it.
name: "ack-only summary closer recovered over a modest answer",
msgs: []llm.Message{
llm.UserText("q?"),
asst(longAnswer, cite...),
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
asst("Citations are logged."),
},
terminal: "Citations are logged.",
want: longAnswer,
},
{
name: "summary closer with only a preamble prior keeps the closer",
msgs: []llm.Message{
llm.UserText("q?"),
asst("Let me gather the numbers.", cite...),
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
asst(b3cb9ee9Closer),
},
terminal: b3cb9ee9Closer,
want: b3cb9ee9Closer,
},
{
// The modeSummary scan must stop at the most recent user message.
// Here the current turn's answer sits in the 1x-3x band (rejected
// by the ratio) while a dwarfing answer to a DIFFERENT question
// sits in history — resurrecting it would be strictly worse than
// keeping the closer.
name: "summary closer never resurrects a stale answer across the user boundary",
msgs: []llm.Message{
llm.UserText("earlier, unrelated question?"),
asst(hugeAnswer),
llm.UserText("q?"),
asst(conciseAnswer, cite...),
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
asst(b3cb9ee9Closer),
},
terminal: b3cb9ee9Closer,
want: b3cb9ee9Closer,
},
{
// The boundary must not break the legitimate multi-turn case: the
// dwarfing front-loaded answer in THIS turn's window is recovered
// even with history behind it.
name: "summary closer recovery still works with history present",
msgs: []llm.Message{
llm.UserText("earlier, unrelated question?"),
asst(longAnswer),
llm.UserText("q?"),
asst(hugeAnswer, cite...),
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
asst(b3cb9ee9Closer),
},
terminal: b3cb9ee9Closer,
want: hugeAnswer,
},
{
// A closer matching BOTH the ack shape and a back-reference
// carries no answer content, so the back-ref test must win and the
// ordinary recovery bar apply — under the summary bar this
// ~106-byte terminal would demand a ~318-byte prior and wrongly
// keep the closer over longAnswer.
name: "back-reference wins over the summary ack when both match",
msgs: []llm.Message{
llm.UserText("q?"),
asst(longAnswer, cite...),
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
asst(bothMatchCloser),
},
terminal: bothMatchCloser,
want: longAnswer,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
@@ -326,6 +473,37 @@ func TestRun_HealthyTerminalUnchanged(t *testing.T) {
}
}
// TestRun_RecoversFrontLoadedAnswerOverSummaryCloser reproduces mort run
// b3cb9ee9 end-to-end: the model front-loads its full answer into the
// cite-call turn, the cite results come back, and the terminal turn is only a
// bookkeeping ack plus a one-line compression. The delivered output must be
// the front-loaded answer, with no extra model call.
func TestRun_RecoversFrontLoadedAnswerOverSummaryCloser(t *testing.T) {
hugeAnswer := strings.TrimSpace(strings.Repeat("Free group calls are capped at sixty minutes. ", 12))
fp := fake.New("fp")
fp.Enqueue("test-model",
fake.ReplyWith(llm.Response{
Parts: []llm.Part{llm.Text(hugeAnswer)},
ToolCalls: []llm.ToolCall{{ID: "c1", Name: "cite", Arguments: json.RawMessage(`{}`)}},
FinishReason: llm.FinishToolCalls,
Usage: llm.Usage{InputTokens: 10, OutputTokens: 5},
}),
fake.Reply(b3cb9ee9Closer),
)
a := New(newModel(t, fp), "sys", WithToolbox(citeToolbox(t)))
res, err := a.Run(context.Background(), "where did the $64M come from?")
if err != nil {
t.Fatalf("Run: %v", err)
}
if res.Output != hugeAnswer {
t.Errorf("Output = %q, want recovered front-loaded answer", res.Output)
}
if n := len(fp.Calls()); n != 2 {
t.Errorf("model calls = %d, want 2 (no extra nudge turn)", n)
}
}
// TestRun_RecoversFrontLoadedAnswerWithCitations reproduces mort issue #1418
// end-to-end: the model front-loads the prose answer into the tool-call turn
// and closes with a sources-only terminal turn. The delivered output must be
+69 -28
View File
@@ -2,7 +2,6 @@ package majordomo
import (
"net/http"
"strings"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/majordomo/provider/anthropic"
@@ -18,7 +17,13 @@ const (
// ProviderKimi is Moonshot AI's Kimi models over their OpenAI-compatible
// Chat Completions endpoint. Reuses the openai client (like llama-swap);
// keyed by KIMI_API_KEY, default base URL kimiBaseURL.
ProviderKimi = "kimi"
ProviderKimi = "kimi"
// ProviderQwen is Alibaba's Qwen models over Model Studio's
// OpenAI-compatible Chat Completions endpoint. Reuses the openai client
// (like kimi and llama-swap); keyed by QWEN_API_KEY, default base URL
// qwenBaseURL. ADR-0027 records why the OpenAI surface and not the
// Anthropic-compatible one Model Studio also exposes.
ProviderQwen = "qwen"
ProviderAnthropic = "anthropic"
ProviderGoogle = "google"
ProviderOllama = "ollama"
@@ -37,6 +42,55 @@ const (
// China endpoint (api.moonshot.cn/v1) is reachable via a kimi:// LLM_* DSN.
const kimiBaseURL = "https://api.moonshot.ai/v1"
// qwenBaseURL is Alibaba Model Studio's international (Singapore) endpoint in
// OpenAI-compatible mode. The China endpoint
// (dashscope.aliyuncs.com/compatible-mode/v1) and any regional host are
// reachable via a qwen:// LLM_* DSN.
const qwenBaseURL = "https://dashscope-intl.aliyuncs.com/compatible-mode/v1"
// openaiCompatScheme builds the DSN factory shared by every built-in that is
// "the openai client pointed somewhere else" (kimi, qwen, ...). The provider
// is named after the LLM_<NAME> var that defined it, takes its credential from
// the DSN token — not the built-in's own env var, which does nothing for a
// DSN-defined provider — and so names that same LLM_<NAME> var in the
// missing-key hint, matching the lazy-resolution key form in providerFor.
//
// wrap is the caller's option-decorator (it injects the registry's HTTP
// client), so a DSN provider is built exactly like the eager built-ins.
func openaiCompatScheme(wrap func(...openai.Option) []openai.Option) SchemeFactory {
return func(name string, dsn DSN) (llm.Provider, error) {
return openai.New(wrap(
openai.WithName(name),
openai.WithBaseURL(dsn.BaseURL()),
openai.WithAPIKey(dsn.Token),
openai.WithAPIKeyName(envKeyForProvider(name)),
)...), nil
}
}
// registerOpenAICompatBuiltin installs BOTH halves of an OpenAI-compat
// built-in: the eager provider under name (credential from keyEnv) and the
// matching name:// DSN scheme. Why both in one call: the two halves are a pair
// — a built-in whose scheme is missing resolves as a spec but not from an
// LLM_* DSN, and the credential rules below have to hold identically in each.
// Adding the next one is a single line rather than six lines to copy.
//
// The two credential rules, holding by construction for every caller:
// - WithAPIKey is passed UNCONDITIONALLY, even when the lookup comes back
// empty. openai.New defaults its key to OPENAI_API_KEY, so anything less
// lets an unset keyEnv silently authenticate as OpenAI.
// - WithAPIKeyName makes the synthetic-401 hint name keyEnv, so a keyless
// call tells the operator the variable that actually fixes it.
func registerOpenAICompatBuiltin(r *Registry, wrap func(...openai.Option) []openai.Option, name, baseURL, keyEnv string) {
r.providers[name] = openai.New(wrap(
openai.WithName(name),
openai.WithBaseURL(baseURL),
openai.WithAPIKey(r.envLookup(keyEnv)),
openai.WithAPIKeyName(keyEnv),
)...)
r.schemes[name] = openaiCompatScheme(wrap)
}
// registerBuiltins installs the built-in providers and env-DSN scheme
// factories into a fresh registry. httpClient, when non-nil, is used by
// every provider and factory the registry itself constructs.
@@ -83,32 +137,19 @@ func registerBuiltins(r *Registry, httpClient *http.Client) {
)...), nil
}
// Kimi (Moonshot AI): OpenAI-compatible Chat Completions, so it reuses the
// openai client (like llama-swap). Defaults to Moonshot's international
// endpoint and the KIMI_API_KEY credential. WithAPIKey is passed
// unconditionally — even empty — so an unset KIMI_API_KEY can never fall
// through to the openai client's OPENAI_API_KEY default; WithAPIKeyName
// makes the missing-key error name KIMI_API_KEY.
r.providers[ProviderKimi] = openai.New(openaiOpts(
openai.WithName(ProviderKimi),
openai.WithBaseURL(kimiBaseURL),
openai.WithAPIKey(r.envLookup("KIMI_API_KEY")),
openai.WithAPIKeyName("KIMI_API_KEY"),
)...)
// kimi:// DSN scheme: an OpenAI-compatible target labeled kimi, base URL
// from the DSN host (e.g. kimi://[email protected]/v1 for China). Its
// credential is the DSN token, not KIMI_API_KEY, so the missing-key hint
// names the LLM_<NAME> env var that defines this provider (matching the
// lazy-resolution key form in providerFor) — the fix for a keyless target
// here is adding a token to that DSN.
r.schemes[ProviderKimi] = func(name string, dsn DSN) (llm.Provider, error) {
return openai.New(openaiOpts(
openai.WithName(name),
openai.WithBaseURL(dsn.BaseURL()),
openai.WithAPIKey(dsn.Token),
openai.WithAPIKeyName("LLM_"+strings.ToUpper(strings.ReplaceAll(name, "-", "_"))),
)...), nil
}
// Third-party endpoints that ARE the openai client at another base URL —
// no new package, mirroring llama-swap's chat path. Each gets the eager
// built-in plus its name:// DSN scheme, and the credential rules hold by
// construction (see registerOpenAICompatBuiltin).
//
// kimi (ADR-0026): Moonshot's international endpoint; China host via
// kimi://[email protected]/v1.
registerOpenAICompatBuiltin(r, openaiOpts, ProviderKimi, kimiBaseURL, "KIMI_API_KEY")
// qwen (ADR-0027): Alibaba Model Studio's international host. Model Studio
// also exposes an Anthropic-compatible endpoint; the ADR records why the
// OpenAI one is the built-in. China / workspace-scoped regional hosts via
// qwen://[email protected]/compatible-mode/v1.
registerOpenAICompatBuiltin(r, openaiOpts, ProviderQwen, qwenBaseURL, "QWEN_API_KEY")
// llama-swap: OpenAI-compatible chat + image generation + management
// endpoints over a model-swapping proxy. Chat reuses the openai client
-171
View File
@@ -1,171 +0,0 @@
package majordomo
import (
"context"
"errors"
"io"
"net/http"
"strings"
"testing"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
// kimiResponse is a minimal valid Chat Completions body so Generate returns a
// non-empty response (an empty one would trigger failover, not a clean pass).
const kimiResponse = `{"id":"c1","object":"chat.completion","choices":[` +
`{"index":0,"message":{"role":"assistant","content":"ok"},"finish_reason":"stop"}]}`
// captureRT records the last request and returns a canned response without
// touching the network, so these tests stay hermetic while still exercising
// the real openai client the kimi built-in reuses (base URL + auth header).
type captureRT struct {
req *http.Request
body string
}
func (c *captureRT) RoundTrip(r *http.Request) (*http.Response, error) {
c.req = r
return &http.Response{
StatusCode: http.StatusOK,
Body: io.NopCloser(strings.NewReader(c.body)),
Header: make(http.Header),
Request: r,
}, nil
}
// TestKimiBuiltin: the built-in "kimi" provider resolves in Parse, targets
// Moonshot's default endpoint, and authenticates with KIMI_API_KEY.
func TestKimiBuiltin(t *testing.T) {
rt := &captureRT{body: kimiResponse}
r := newTestRegistry(t,
WithEnvLookup(func(k string) string {
if k == "KIMI_API_KEY" {
return "kimi-secret"
}
return ""
}),
WithHTTPClient(&http.Client{Transport: rt}),
)
if p, ok := r.Provider(ProviderKimi); !ok {
t.Fatal("built-in kimi provider not registered")
} else if p.Name() != ProviderKimi {
t.Errorf("name = %q, want %q", p.Name(), ProviderKimi)
}
m, err := r.Parse("kimi/kimi-k2-0711-preview")
if err != nil {
t.Fatalf("Parse: %v", err)
}
if got := targetsOf(t, m); len(got) != 1 || got[0] != "kimi/kimi-k2-0711-preview" {
t.Fatalf("targets = %v", got)
}
if _, err := m.Generate(context.Background(), llm.Request{Messages: []llm.Message{llm.UserText("hi")}}); err != nil {
t.Fatalf("Generate: %v", err)
}
if rt.req == nil {
t.Fatal("no request captured")
}
if want := "https://api.moonshot.ai/v1/chat/completions"; rt.req.URL.String() != want {
t.Errorf("URL = %q, want %q", rt.req.URL.String(), want)
}
if want := "Bearer kimi-secret"; rt.req.Header.Get("Authorization") != want {
t.Errorf("Authorization = %q, want %q", rt.req.Header.Get("Authorization"), want)
}
}
// TestKimiBuiltinMissingKey: with no KIMI_API_KEY the built-in fails fast with a
// synthetic 401 whose hint names KIMI_API_KEY — never OPENAI_API_KEY (proving
// the credential does not fall through to the openai client's default), and
// without hitting the network.
func TestKimiBuiltinMissingKey(t *testing.T) {
rt := &captureRT{body: kimiResponse}
r := newTestRegistry(t, WithHTTPClient(&http.Client{Transport: rt}))
m, err := r.Parse("kimi/kimi-k2-0711-preview")
if err != nil {
t.Fatalf("Parse: %v", err)
}
_, err = m.Generate(context.Background(), llm.Request{Messages: []llm.Message{llm.UserText("hi")}})
apiErr, ok := errors.AsType[*llm.APIError](err)
if !ok {
t.Fatalf("err = %v (%T), want *llm.APIError", err, err)
}
if apiErr.Status != http.StatusUnauthorized || apiErr.Code != "missing_api_key" {
t.Errorf("Status/Code = %d/%q, want 401/missing_api_key", apiErr.Status, apiErr.Code)
}
if !strings.Contains(apiErr.Message, "KIMI_API_KEY") {
t.Errorf("message = %q, want it to name KIMI_API_KEY", apiErr.Message)
}
if strings.Contains(apiErr.Message, "OPENAI_API_KEY") {
t.Errorf("message = %q, must not name OPENAI_API_KEY", apiErr.Message)
}
if rt.req != nil {
t.Error("network was hit despite missing key")
}
}
// TestKimiScheme: a kimi:// LLM_* DSN defines a named provider on any Moonshot
// host (here the China endpoint) that is first-class in Parse and carries the
// DSN token as its bearer credential.
func TestKimiScheme(t *testing.T) {
rt := &captureRT{body: kimiResponse}
r := newTestRegistry(t, WithHTTPClient(&http.Client{Transport: rt}))
if err := r.LoadEnv(map[string]string{
"LLM_KCN": "kimi://[email protected]/v1",
}); err != nil {
t.Fatalf("LoadEnv: %v", err)
}
m, err := r.Parse("kcn/moonshot-v1-8k")
if err != nil {
t.Fatalf("Parse: %v", err)
}
if _, err := m.Generate(context.Background(), llm.Request{Messages: []llm.Message{llm.UserText("hi")}}); err != nil {
t.Fatalf("Generate: %v", err)
}
if rt.req == nil {
t.Fatal("no request captured")
}
if want := "https://api.moonshot.cn/v1/chat/completions"; rt.req.URL.String() != want {
t.Errorf("URL = %q, want %q", rt.req.URL.String(), want)
}
if want := "Bearer tok"; rt.req.Header.Get("Authorization") != want {
t.Errorf("Authorization = %q, want %q", rt.req.Header.Get("Authorization"), want)
}
}
// TestKimiSchemeMissingToken: a kimi:// DSN with no token is fixed by adding one
// to the DSN, not by setting KIMI_API_KEY — so the missing-key hint names the
// defining LLM_<NAME> env var, never KIMI_API_KEY (which does nothing for a
// DSN-defined provider).
func TestKimiSchemeMissingToken(t *testing.T) {
rt := &captureRT{body: kimiResponse}
r := newTestRegistry(t, WithHTTPClient(&http.Client{Transport: rt}))
if err := r.LoadEnv(map[string]string{
"LLM_KCN": "kimi://api.moonshot.cn/v1", // no token
}); err != nil {
t.Fatalf("LoadEnv: %v", err)
}
m, err := r.Parse("kcn/moonshot-v1-8k")
if err != nil {
t.Fatalf("Parse: %v", err)
}
_, err = m.Generate(context.Background(), llm.Request{Messages: []llm.Message{llm.UserText("hi")}})
apiErr, ok := errors.AsType[*llm.APIError](err)
if !ok {
t.Fatalf("err = %v (%T), want *llm.APIError", err, err)
}
if !strings.Contains(apiErr.Message, "LLM_KCN") {
t.Errorf("message = %q, want it to name LLM_KCN", apiErr.Message)
}
if strings.Contains(apiErr.Message, "KIMI_API_KEY") {
t.Errorf("message = %q, must not name KIMI_API_KEY for a DSN provider", apiErr.Message)
}
if rt.req != nil {
t.Error("network was hit despite missing token")
}
}
+240
View File
@@ -0,0 +1,240 @@
package majordomo
import (
"context"
"errors"
"io"
"net/http"
"strings"
"testing"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
// Shared fixtures and the shared contract for the built-ins that are "the
// openai client pointed somewhere else" (kimi, qwen, ...). They live here
// rather than in any one provider's test file so a new OpenAI-compat built-in
// has nothing to copy — the same reason registerOpenAICompatBuiltin exists on
// the production side.
// chatCompletionOK is a minimal valid Chat Completions body, so Generate
// returns a non-empty response (an empty one would trigger failover, not a
// clean pass).
const chatCompletionOK = `{"id":"c1","object":"chat.completion","choices":[` +
`{"index":0,"message":{"role":"assistant","content":"ok"},"finish_reason":"stop"}]}`
// captureRT records the last request (and the bytes of its body) and returns a
// canned response without touching the network, so these tests stay hermetic
// while still exercising the real openai client the built-ins reuse: base URL,
// auth header, and the JSON actually put on the wire.
type captureRT struct {
req *http.Request
reqBody []byte
body string
}
func (c *captureRT) RoundTrip(r *http.Request) (*http.Response, error) {
c.req = r
// Drain and close the request body: a RoundTripper owns it, and those
// bytes are what wire-shape assertions read.
c.reqBody = nil
if r.Body != nil {
c.reqBody, _ = io.ReadAll(r.Body)
_ = r.Body.Close()
}
return &http.Response{
StatusCode: http.StatusOK,
Body: io.NopCloser(strings.NewReader(c.body)),
Header: make(http.Header),
Request: r,
}, nil
}
// singleKeyEnv builds a WithEnvLookup function that knows exactly one variable
// and returns "" for everything else. The empty default has teeth: a built-in
// that reached for any other variable name gets nothing, so the request 401s
// and the test fails rather than quietly authenticating off the wrong key.
func singleKeyEnv(key, value string) func(string) string {
return func(k string) string {
if k == key {
return value
}
return ""
}
}
// openAICompatBuiltin describes one built-in for the shared contract below.
// Adding an OpenAI-compat built-in means adding a row here — not copying a
// test file, which is how kimi's and qwen's suites became near-identical.
type openAICompatBuiltin struct {
name string // registry name and spec prefix
keyEnv string // the credential variable this built-in reads
model string // a current model id for that endpoint
wantURL string // chat-completions URL the default endpoint must produce
// The name:// DSN case: an alternate host (regional/China endpoint)
// reached through an LLM_<dsnVar> definition.
dsnVar string
dsnHost string
wantDSNURL string
}
var openAICompatBuiltins = []openAICompatBuiltin{
{
name: ProviderKimi,
keyEnv: "KIMI_API_KEY",
model: "kimi-k2-0711-preview",
wantURL: "https://api.moonshot.ai/v1/chat/completions",
dsnVar: "LLM_KCN",
dsnHost: "api.moonshot.cn/v1",
wantDSNURL: "https://api.moonshot.cn/v1/chat/completions",
},
{
name: ProviderQwen,
keyEnv: "QWEN_API_KEY",
model: "qwen3.8-max",
wantURL: "https://dashscope-intl.aliyuncs.com/compatible-mode/v1/chat/completions",
dsnVar: "LLM_QCN",
dsnHost: "dashscope.aliyuncs.com/compatible-mode/v1",
wantDSNURL: "https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions",
},
}
// TestOpenAICompatBuiltins is the whole contract an OpenAI-compat built-in
// owes, asserted identically for every one of them: it resolves in Parse and
// targets its own endpoint with its own key; a missing key fails closed naming
// the right variable and never reaching the network; its name:// DSN reaches
// any other host on the DSN token; and a keyless DSN names the LLM_<NAME> that
// actually fixes it rather than the built-in's variable, which does nothing
// for a DSN-defined provider.
func TestOpenAICompatBuiltins(t *testing.T) {
for _, tc := range openAICompatBuiltins {
t.Run(tc.name+"/builtin", func(t *testing.T) {
rt := &captureRT{body: chatCompletionOK}
secret := tc.name + "-secret"
r := newTestRegistry(t,
WithEnvLookup(singleKeyEnv(tc.keyEnv, secret)),
WithHTTPClient(&http.Client{Transport: rt}),
)
if p, ok := r.Provider(tc.name); !ok {
t.Fatalf("built-in %q not registered", tc.name)
} else if p.Name() != tc.name {
t.Errorf("name = %q, want %q", p.Name(), tc.name)
}
spec := tc.name + "/" + tc.model
m, err := r.Parse(spec)
if err != nil {
t.Fatalf("Parse(%q): %v", spec, err)
}
if got := targetsOf(t, m); len(got) != 1 || got[0] != spec {
t.Fatalf("targets = %v, want [%q]", got, spec)
}
if _, err := m.Generate(context.Background(), llm.Request{Messages: []llm.Message{llm.UserText("hi")}}); err != nil {
t.Fatalf("Generate: %v", err)
}
if rt.req == nil {
t.Fatal("no request captured")
}
if rt.req.URL.String() != tc.wantURL {
t.Errorf("URL = %q, want %q", rt.req.URL.String(), tc.wantURL)
}
if want := "Bearer " + secret; rt.req.Header.Get("Authorization") != want {
t.Errorf("Authorization = %q, want %q", rt.req.Header.Get("Authorization"), want)
}
})
t.Run(tc.name+"/builtin missing key", func(t *testing.T) {
rt := &captureRT{body: chatCompletionOK}
r := newTestRegistry(t, WithHTTPClient(&http.Client{Transport: rt}))
m, err := r.Parse(tc.name + "/" + tc.model)
if err != nil {
t.Fatalf("Parse: %v", err)
}
_, err = m.Generate(context.Background(), llm.Request{Messages: []llm.Message{llm.UserText("hi")}})
apiErr, ok := errors.AsType[*llm.APIError](err)
if !ok {
t.Fatalf("err = %v (%T), want *llm.APIError", err, err)
}
if apiErr.Status != http.StatusUnauthorized || apiErr.Code != "missing_api_key" {
t.Errorf("Status/Code = %d/%q, want 401/missing_api_key", apiErr.Status, apiErr.Code)
}
if !strings.Contains(apiErr.Message, tc.keyEnv) {
t.Errorf("message = %q, want it to name %s", apiErr.Message, tc.keyEnv)
}
// The load-bearing half: openai.New defaults its key to
// OPENAI_API_KEY, so a built-in that stopped passing WithAPIKey
// unconditionally would authenticate as OpenAI instead of failing.
if strings.Contains(apiErr.Message, "OPENAI_API_KEY") {
t.Errorf("message = %q, must not name OPENAI_API_KEY", apiErr.Message)
}
if rt.req != nil {
t.Error("network was hit despite missing key")
}
})
t.Run(tc.name+"/dsn scheme", func(t *testing.T) {
rt := &captureRT{body: chatCompletionOK}
r := newTestRegistry(t, WithHTTPClient(&http.Client{Transport: rt}))
if err := r.LoadEnv(map[string]string{
tc.dsnVar: tc.name + "://tok@" + tc.dsnHost,
}); err != nil {
t.Fatalf("LoadEnv: %v", err)
}
dsnName := strings.ToLower(strings.TrimPrefix(tc.dsnVar, "LLM_"))
m, err := r.Parse(dsnName + "/" + tc.model)
if err != nil {
t.Fatalf("Parse: %v", err)
}
if _, err := m.Generate(context.Background(), llm.Request{Messages: []llm.Message{llm.UserText("hi")}}); err != nil {
t.Fatalf("Generate: %v", err)
}
if rt.req == nil {
t.Fatal("no request captured")
}
if rt.req.URL.String() != tc.wantDSNURL {
t.Errorf("URL = %q, want %q", rt.req.URL.String(), tc.wantDSNURL)
}
if want := "Bearer tok"; rt.req.Header.Get("Authorization") != want {
t.Errorf("Authorization = %q, want %q", rt.req.Header.Get("Authorization"), want)
}
})
t.Run(tc.name+"/dsn scheme missing token", func(t *testing.T) {
rt := &captureRT{body: chatCompletionOK}
r := newTestRegistry(t, WithHTTPClient(&http.Client{Transport: rt}))
if err := r.LoadEnv(map[string]string{
tc.dsnVar: tc.name + "://" + tc.dsnHost, // no token
}); err != nil {
t.Fatalf("LoadEnv: %v", err)
}
dsnName := strings.ToLower(strings.TrimPrefix(tc.dsnVar, "LLM_"))
m, err := r.Parse(dsnName + "/" + tc.model)
if err != nil {
t.Fatalf("Parse: %v", err)
}
_, err = m.Generate(context.Background(), llm.Request{Messages: []llm.Message{llm.UserText("hi")}})
apiErr, ok := errors.AsType[*llm.APIError](err)
if !ok {
t.Fatalf("err = %v (%T), want *llm.APIError", err, err)
}
// A keyless DSN is fixed by adding a token to that DSN, so the
// hint must name the defining variable — never the built-in's own
// key, which does nothing for a DSN-defined provider.
if !strings.Contains(apiErr.Message, tc.dsnVar) {
t.Errorf("message = %q, want it to name %s", apiErr.Message, tc.dsnVar)
}
if strings.Contains(apiErr.Message, tc.keyEnv) {
t.Errorf("message = %q, must not name %s for a DSN provider", apiErr.Message, tc.keyEnv)
}
if rt.req != nil {
t.Error("network was hit despite missing token")
}
})
}
}
+86
View File
@@ -0,0 +1,86 @@
package majordomo
import (
"context"
"encoding/json"
"net/http"
"testing"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
// The contract qwen shares with every other OpenAI-compat built-in (endpoint,
// credential isolation, its qwen:// DSN) is asserted by the table in
// builtin_openaicompat_test.go. What remains here is qwen-specific: the
// reverse-leak direction, and the wire claim ADR-0027 turns on.
// TestQwenBuiltinKeyDoesNotLeakToOpenAI: QWEN_API_KEY is the qwen built-in's
// credential and nothing else's. Why this direction too: the shared table's
// missing-key case only proves qwen never borrows OPENAI_API_KEY; this proves
// the reverse — a registry that can see QWEN_API_KEY must not hand it to the
// openai built-in, which would send an Alibaba key to api.openai.com.
func TestQwenBuiltinKeyDoesNotLeakToOpenAI(t *testing.T) {
// Set before newTestRegistry: the openai built-in reads OPENAI_API_KEY at
// construction. Giving it a real key is what keeps this test honest — a
// keyless openai target would 401 before any request, and the assertion
// below would pass without a single byte reaching the wire.
t.Setenv("OPENAI_API_KEY", "openai-secret")
rt := &captureRT{body: chatCompletionOK}
r := newTestRegistry(t,
WithEnvLookup(singleKeyEnv("QWEN_API_KEY", "qwen-secret")),
WithHTTPClient(&http.Client{Transport: rt}),
)
m, err := r.Parse("openai/gpt-4o-mini")
if err != nil {
t.Fatalf("Parse: %v", err)
}
if _, err := m.Generate(context.Background(), llm.Request{Messages: []llm.Message{llm.UserText("hi")}}); err != nil {
t.Fatalf("Generate: %v", err)
}
if rt.req == nil {
t.Fatal("no request captured")
}
if want := "Bearer openai-secret"; rt.req.Header.Get("Authorization") != want {
t.Errorf("Authorization = %q, want %q — the qwen credential must not reach the openai built-in",
rt.req.Header.Get("Authorization"), want)
}
}
// TestQwenReasoningEffortReachesWire is the load-bearing test for ADR-0027's
// central claim: Model Studio's OpenAI-compatible surface takes reasoning as a
// top-level "reasoning_effort" body field, which the openai client already
// sends — so llm.WithReasoningEffort survives the trip on qwen with no
// qwen-specific code. Routing qwen through the anthropic client instead would
// drop it silently (provider/anthropic ignores ReasoningEffort by design), and
// that difference would be invisible without asserting on the wire body.
func TestQwenReasoningEffortReachesWire(t *testing.T) {
rt := &captureRT{body: chatCompletionOK}
r := newTestRegistry(t,
WithEnvLookup(singleKeyEnv("QWEN_API_KEY", "qwen-secret")),
WithHTTPClient(&http.Client{Transport: rt}),
)
m, err := r.Parse("qwen/qwen3.8-max")
if err != nil {
t.Fatalf("Parse: %v", err)
}
_, err = m.Generate(context.Background(), llm.Request{
Messages: []llm.Message{llm.UserText("hi")},
ReasoningEffort: "high",
})
if err != nil {
t.Fatalf("Generate: %v", err)
}
if rt.reqBody == nil {
t.Fatal("no request body captured")
}
var sent map[string]any
if err := json.Unmarshal(rt.reqBody, &sent); err != nil {
t.Fatalf("decode request body: %v", err)
}
if got := sent["reasoning_effort"]; got != "high" {
t.Errorf("reasoning_effort = %v, want %q (body: %s)", got, "high", rt.reqBody)
}
}
+102
View File
@@ -0,0 +1,102 @@
# ADR-0027: Qwen (Alibaba) built-in provider — OpenAI-compat, not Anthropic-compat
**Status:** Accepted — 2026-08-12
## Context
Alibaba's Qwen models (`qwen3.8-max`, `qwen3.7-plus`, the `qwen3-vl-*` vision
variants, …) are served from Model Studio / DashScope, and mort wants them as a
first-class failover tier with a dedicated `QWEN_API_KEY` — the same ergonomics
ADR-0026 gave Kimi.
Unlike Kimi, Model Studio exposes the same models over **two** protocols:
| | OpenAI-compatible | Anthropic-compatible |
|---|---|---|
| Base URL | `https://dashscope-intl.aliyuncs.com/compatible-mode/v1` | `https://dashscope-intl.aliyuncs.com/apps/anthropic` |
| Endpoints | full Chat Completions surface | `/v1/messages` only (no `/v1/models`) |
| Purpose | the documented developer API | a shim, documented around hosting Claude Code |
So the question this ADR answers is not "which client do we reuse" but
"which of Alibaba's two wire protocols does the built-in speak".
## Decision
**The `qwen` built-in and the `qwen://` DSN scheme speak OpenAI-compat**, over
`provider/openai` — no new package, mirroring ADR-0026 (kimi) and ADR-0015
(llama-swap chat). Default base URL is the international host; the China host
(`dashscope.aliyuncs.com/compatible-mode/v1`) and workspace-scoped regional
hosts are reachable with a `qwen://` DSN.
Credential handling is copied from kimi verbatim, because both of its rules
are load-bearing: `WithAPIKey` is passed unconditionally (even empty) so an
unset `QWEN_API_KEY` can never fall through to `openai.New`'s `OPENAI_API_KEY`
default, and `WithAPIKeyName("QWEN_API_KEY")` makes the synthetic-401 hint name
the variable the operator actually has to set.
The kimi and qwen DSN factories were identical, so they now share one
`openaiCompatScheme` helper — the next OpenAI-compat built-in gets the
credential and key-hint rules by construction rather than by copy.
### Why not the Anthropic-compatible endpoint
Every concrete difference favors OpenAI-compat *for this codebase*:
- **Reasoning survives the trip.** Model Studio takes `reasoning_effort` as a
top-level field on the OpenAI surface, which `provider/openai` already sends
`llm.WithReasoningEffort` works on qwen with zero qwen-specific code
(`TestQwenReasoningEffortReachesWire` asserts it on the wire). Down the
anthropic client it would be dropped in silence: `provider/anthropic`
deliberately ignores `Request.ReasoningEffort`, because first-party Claude
has no such knob.
- **Structured output would regress.** `provider/anthropic` implements
`Request.Schema` with the first-party GA `output_config.format` mechanism.
Alibaba's shim does not implement it; a compat endpoint that ignores an
unknown field returns unconstrained prose while still reporting success.
The OpenAI path sends `response_format: json_schema`, which Model Studio
supports natively on the Max/Plus families.
- **Cache accounting already lands.** Model Studio's implicit prefix cache
reports hits in `usage.prompt_tokens_details.cached_tokens`, which the openai
client already maps to `llm.Usage.CacheReadTokens`. The anthropic client
reads `cache_read_input_tokens`, a field the shim has no reason to emit.
- **Thinking content is discarded on the anthropic path anyway.**
`provider/anthropic` skips `thinking` blocks in both the buffered and
streaming decoders, so the shim's headline feature — first-class
`thinking: {type: "enabled", budget_tokens: N}` — buys majordomo nothing
today.
- **Smaller blast radius.** The anthropic client has no `WithAPIKeyName`
option, so a keyless qwen would tell the operator to set `ANTHROPIC_API_KEY`;
fixing that means changing the first-party Anthropic client to serve a
third-party shim.
- **It is the less-exercised surface.** The Anthropic endpoint is documented as
Messages-only, with a temperature range that differs from Anthropic's own
([0, 2) vs [0.0, 1.0]) — i.e. it is Qwen semantics wearing an Anthropic
envelope, not an Anthropic-equivalent target.
The one thing the Anthropic surface offers that OpenAI-compat does not is
explicit `cache_control` breakpoints reached through `Request.PromptCache`.
That is not a reason to route Qwen through it: Model Studio's implicit cache is
automatic and already metered, and if explicit breakpoints ever matter they
belong in `provider/openai` (Model Studio accepts `cache_control` on content
blocks there too), where every OpenAI-compat target would get them.
## Consequences
- `qwen/<model>` is first-class in Parse, chains, aliases, and health/failover
with no consumer wiring; model ids pass through verbatim (no catalog).
- Chat, streaming, tools, structured output, reasoning effort, and cached-token
accounting all ride the openai client and inherit its fixes.
- Image *inputs* work at the client level, but only the `qwen-vl-*` /
`qwen3-vl-*` models accept them (matrix footnote ⁴; ³ is kimi's).
- Two model-side quirks are Alibaba's, not majordomo's, and are left to the
caller rather than papered over: thinking is **on by default** on some models
(e.g. `qwen3.7-plus`), and Qwen3 *open-source* models require streaming when
thinking is enabled — a buffered `Generate` against one of those needs a
model that supports non-streaming thinking (the Max/Plus families do).
- If a future consumer genuinely needs the Anthropic surface, it is reachable
today without library changes:
`LLM_QWEN_ANTHROPIC=anthropic://[email protected]/apps/anthropic`
— with the reasoning/structured-output caveats above.
- Second third-party built-in after kimi. The ADR-0026 bar still holds: a named
consumer needs it in-config. `RegisterProvider`/`LLM_*` remain the path for
everything else.
+1
View File
@@ -30,3 +30,4 @@ One decision per file, append-only; supersede rather than rewrite.
| [0024](0024-audio-wave3-surfaces.md) | Wave-3 audio surfaces (stems, SFX, speech enhance, voice clone, translate) | Accepted |
| [0025](0025-videogen-wave3-surfaces.md) | Wave-3 video surfaces (lipsync, video matte, video upscale, chain jobs) | Accepted |
| [0026](0026-kimi-builtin.md) | Kimi (Moonshot AI) built-in provider — reuse openai client, KIMI_API_KEY | Accepted |
| [0027](0027-qwen-builtin.md) | Qwen (Alibaba) built-in provider — OpenAI-compat, not Model Studio's Anthropic-compat endpoint | Accepted |
+16 -2
View File
@@ -26,8 +26,9 @@ var ErrUnknownProvider = errors.New("unknown provider")
// authenticated with the bearer token "test-token".
type DSN struct {
// Scheme selects the provider implementation: "foreman", "ollama",
// "ollama-cloud", "openai", "kimi", "anthropic", "google"/"gemini", or
// any custom scheme registered with RegisterScheme.
// "ollama-cloud", "openai", "kimi", "qwen", "anthropic",
// "google"/"gemini", "llama-swap"/"llama-swaps", or any custom scheme
// registered with RegisterScheme.
Scheme string
// Token is the provider secret (bearer token or API key); empty = none.
Token string
@@ -40,6 +41,19 @@ type DSN struct {
// env-defined providers always speak TLS).
func (d DSN) BaseURL() string { return "https://" + d.Host }
// envKeyForProvider returns the LLM_* variable that defines the provider named
// name: "m1" → LLM_M1, "my-prov" → LLM_MY_PROV.
//
// This is the single definition on purpose. Two call sites need byte-identical
// output and would drift apart in silence: lazy resolution reads this variable
// to find an unregistered provider, and openaiCompatScheme names it in the
// missing-key hint so a keyless DSN target tells the operator which variable to
// set. Those two were separate copies with a comment asserting they matched —
// a comment is not enforcement, this function is.
func envKeyForProvider(name string) string {
return "LLM_" + strings.ToUpper(strings.ReplaceAll(name, "-", "_"))
}
// ParseDSN parses a raw DSN string. The algorithm matches go-llm exactly:
// split on "://", then an optional "@" separates the token from the host;
// trailing slashes on the host are trimmed.
+59
View File
@@ -55,6 +55,14 @@ type DetectedFace struct {
Box [4]int
// Score is the detector's confidence, 0-1.
Score float64
// Yaw is how far the head is turned from camera, in degrees, or nil when
// the provider does not report pose. Exposed on ENUMERATION, not just
// after the fact, because it is how a caller picks a face a swap will
// actually work on: past roughly ±45° the features carrying identity are
// edge-on, and the result reads as a generic person however good the
// transfer is. A bounding box cannot show this.
Yaw *float64
}
// Size returns the box dimensions. Derived rather than stored: carrying
@@ -64,6 +72,57 @@ func (f DetectedFace) Size() (w, h int) {
return f.Box[2] - f.Box[0], f.Box[3] - f.Box[1]
}
// SwappedFace is the MEASURED outcome for one face the provider replaced.
//
// It exists because "the call returned an image" and "the likeness
// transferred" are different claims that look identical from outside, and a
// caller that cannot tell them apart will go looking for another way to
// check. The one it reaches for — asking a vision model who the result looks
// like — is wrong in exactly the cases that matter: a VLM shown a jogger in a
// Georgetown cap holding McDonald's cups answers "Bill Clinton" whoever's
// face is on him, so it reports failure on a correct swap.
type SwappedFace struct {
// Index is the face's position in the provider's left-to-right ordering.
Index int
// Width, Height are the replaced face's pixel size in the TARGET.
// Meaningful only against ImageWidth/ImageHeight: a 138px face is large
// in a 400px picture and nearly invisible in a 2000px one, and it is the
// ratio, not the absolute size, that decides whether a person notices.
Width, Height int
// ImageWidth, ImageHeight are the target image's dimensions, repeated on
// every entry so a single face is self-describing without the caller
// holding onto the rest of the response.
ImageWidth, ImageHeight int
// Yaw is how far the head is turned from camera, in degrees, or nil when
// the provider does not report pose. The best single predictor of whether
// a swap will READ as the source person: past roughly ±45° the features
// carrying identity are edge-on and the result looks like a generic
// person rather than a specific one.
Yaw *float64
// IdentitySimilarity is cosine similarity between the source face and the
// face actually present in the result, 0-1, or nil when the provider
// could not measure it. Above ~0.5 the identity transferred; a LOW value
// is the only evidence that a swap genuinely failed.
IdentitySimilarity *float64
}
// FractionOfImage is the swapped face's width as a share of the image's, 0-1.
// The number that predicts whether a person will SEE the change: the swap
// that prompted all this replaced a 138px face in a 1010px-wide photo — 14%,
// correct by every measure and invisible at a glance — while the same code on
// a 168px face in a 385px picture (44%) is unmistakable. Returns 0 when the
// dimensions are unknown.
func (f SwappedFace) FractionOfImage() float64 {
if f.ImageWidth <= 0 || f.Width <= 0 {
return 0
}
return float64(f.Width) / float64(f.ImageWidth)
}
// FaceSwapper is the optional face-transfer surface. Separate interface so
// existing providers keep compiling; callers type-assert.
type FaceSwapper interface {
+9
View File
@@ -68,6 +68,15 @@ type Result struct {
// Images are the generated images, in the order the backend returned them.
Images []Image
// SwappedFaces is the measured outcome of a FaceSwap, one entry per face
// replaced. Empty for every other operation, and empty from a provider
// that does not measure. TYPED rather than tucked into Raw: a caller has
// to act on this — it is the only way to distinguish a swap that
// transferred the likeness from one that returned an image and nothing
// more — and a value reachable solely by type-asserting an `any` is one
// nobody discovers in time to use.
SwappedFaces []SwappedFace
// Raw is the provider-native response object, an escape hatch for
// provider-specific fields. May be nil; never required for normal use.
Raw any
+36
View File
@@ -285,3 +285,39 @@ tests flush out.
(footnote ³), `.env.example`, ADR-0026 (+ index; also backfilled the missing
0024/0025 index rows).
- Consumer: mort names Kimi as a failover tier.
## 2026-08-12 — Qwen (Alibaba) built-in provider (ADR-0027)
- New built-in `qwen` provider + `qwen://` DSN scheme over Alibaba Model
Studio's OpenAI-compatible mode, reusing `provider/openai` (no new client,
mirrors kimi/llama-swap). Default base URL
`https://dashscope-intl.aliyuncs.com/compatible-mode/v1`; China/regional
hosts via `LLM_QCN=qwen://[email protected]/compatible-mode/v1`.
- Credential is `QWEN_API_KEY` (via the registry's injected envLookup).
`WithAPIKey` passed unconditionally so an unset key cannot fall through to
`OPENAI_API_KEY`; `WithAPIKeyName` names `QWEN_API_KEY` in the 401 hint.
- **Chose OpenAI-compat over Model Studio's Anthropic-compatible
`/apps/anthropic` shim** (ADR-0027): on the anthropic client
`ReasoningEffort` is ignored by design, `Request.Schema` rides
`output_config.format` (which the shim does not implement), and cached-token
accounting reads Anthropic-only usage fields. The shim is still reachable
ad hoc via an `anthropic://` DSN.
- `registerOpenAICompatBuiltin` installs BOTH halves of an OpenAI-compat
built-in (eager provider + `name://` DSN scheme via the shared
`openaiCompatScheme`), so the two credential rules — unconditional
`WithAPIKey`, and `WithAPIKeyName` naming that same variable — hold by
construction. kimi and qwen are one line each.
- `envKeyForProvider` is the single definition of the `LLM_<NAME>` form,
shared by lazy resolution (`registry.go`) and the DSN missing-key hint. They
were separate copies with a comment asserting they matched.
- The shared contract is ONE table (`builtin_openaicompat_test.go`), run
identically for every OpenAI-compat built-in: endpoint + bearer, missing key
fails closed naming its own variable with no network hit, the `name://` DSN
reaching another host, and a keyless DSN naming `LLM_<NAME>` rather than the
built-in's key. Adding a built-in is a table row that immediately owes all
four; `builtin_kimi_test.go` was retired into it. Qwen-only tests: the
reverse credential leak, and `reasoning_effort` asserted on the wire body
(the ADR's load-bearing claim).
- Docs in sync: README built-in table + Qwen paragraph + DSN scheme list +
support matrix (footnote ⁴), `.env.example`, ADR-0027 (+ index).
- Consumer: mort wants Qwen as a failover tier.
+21 -7
View File
@@ -338,27 +338,41 @@ func parseVoices(raw []byte) ([]string, error) {
// varies. contentType sets the request Content-Type when body is non-nil.
// A response larger than maxBytes is an error, never a silent truncation.
func (p *Provider) doRaw(ctx context.Context, method, path, model, contentType string, body io.Reader, maxBytes int64) ([]byte, string, error) {
if err := p.requireBaseURL(); err != nil {
data, hdr, err := p.doRawHeaders(ctx, method, path, model, contentType, body, maxBytes)
if err != nil {
return nil, "", err
}
return data, hdr.Get("Content-Type"), nil
}
// doRawHeaders is doRaw with the WHOLE response header rather than just
// Content-Type. Only the face swap needs it — the shim reports whether the
// likeness actually transferred in X-Swap-Report, and that answer would be
// thrown away by a function that keeps one header — so doRaw stays the
// signature 25 other call sites use and delegates here. Two bodies would be
// two places for the size cap and the status check to drift apart.
func (p *Provider) doRawHeaders(ctx context.Context, method, path, model, contentType string, body io.Reader, maxBytes int64) ([]byte, http.Header, error) {
if err := p.requireBaseURL(); err != nil {
return nil, nil, err
}
req, err := p.newRequest(ctx, method, path, contentType, body)
if err != nil {
return nil, "", err
return nil, nil, err
}
resp, err := p.client.Do(req)
if err != nil {
return nil, "", fmt.Errorf("llama-swap: do request: %w", err)
return nil, nil, fmt.Errorf("llama-swap: do request: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode/100 != 2 {
return nil, "", p.apiError(resp, model)
return nil, nil, p.apiError(resp, model)
}
data, err := io.ReadAll(io.LimitReader(resp.Body, maxBytes+1))
if err != nil {
return nil, "", fmt.Errorf("llama-swap: read response: %w", err)
return nil, nil, fmt.Errorf("llama-swap: read response: %w", err)
}
if int64(len(data)) > maxBytes {
return nil, "", fmt.Errorf("llama-swap: response exceeds %d bytes", maxBytes)
return nil, nil, fmt.Errorf("llama-swap: response exceeds %d bytes", maxBytes)
}
return data, resp.Header.Get("Content-Type"), nil
return data, resp.Header, nil
}
+63 -11
View File
@@ -41,11 +41,12 @@ type faceSwapModel struct {
type facesResponse struct {
Count int `json:"count"`
Faces []struct {
Index int `json:"index"`
Box []int `json:"box"`
Score float64 `json:"score"`
Width int `json:"width"`
Height int `json:"height"`
Index int `json:"index"`
Box []int `json:"box"`
Score float64 `json:"score"`
Width int `json:"width"`
Height int `json:"height"`
Yaw *float64 `json:"yaw"`
} `json:"faces"`
}
@@ -74,7 +75,7 @@ func (m *faceSwapModel) ListFaces(ctx context.Context, img imagegen.Image) ([]im
}
out := make([]imagegen.DetectedFace, 0, len(parsed.Faces))
for _, f := range parsed.Faces {
df := imagegen.DetectedFace{Index: f.Index, Score: f.Score}
df := imagegen.DetectedFace{Index: f.Index, Score: f.Score, Yaw: f.Yaw}
// A short box would silently index out of range below; treat a
// malformed entry as a protocol error rather than zero-filling it,
// because a wrong box sends the caller at the wrong face.
@@ -126,10 +127,11 @@ func (m *faceSwapModel) FaceSwap(ctx context.Context, req imagegen.FaceSwapReque
if err != nil {
return nil, err
}
raw, respType, err := m.p.doRaw(ctx, http.MethodPost, path, m.id, contentType, body, maxFaceSwapResponseBytes)
raw, respHdr, err := m.p.doRawHeaders(ctx, http.MethodPost, path, m.id, contentType, body, maxFaceSwapResponseBytes)
if err != nil {
return nil, err
}
respType := respHdr.Get("Content-Type")
if len(raw) == 0 {
return nil, &llm.APIError{Provider: m.p.name, Model: m.id, Message: "face swap response contained no image"}
}
@@ -151,16 +153,66 @@ func (m *faceSwapModel) FaceSwap(ctx context.Context, req imagegen.FaceSwapReque
if hdr := mimeFromContentType(respType, "image/"); hdr != "" {
mimeType = hdr
}
return &imagegen.Result{Images: []llm.ImagePart{{MIME: mimeType, Data: raw}}}, nil
return &imagegen.Result{
Images: []llm.ImagePart{{MIME: mimeType, Data: raw}},
SwappedFaces: parseSwapReport(respHdr.Get("X-Swap-Report")),
}, nil
}
// swapReport mirrors the shim's X-Swap-Report header.
type swapReport struct {
Image []int `json:"image"`
Faces []struct {
Index int `json:"index"`
Size []int `json:"size"`
Yaw *float64 `json:"yaw"`
IdentitySimilarity *float64 `json:"identity_similarity"`
} `json:"faces"`
}
// parseSwapReport decodes the measured outcome. A missing or malformed header
// yields nil rather than an error: an older shim does not send it, and a swap
// that produced a good image must not fail because the diagnostics alongside
// it were unreadable.
func parseSwapReport(header string) []imagegen.SwappedFace {
header = strings.TrimSpace(header)
if header == "" {
return nil
}
var rep swapReport
if err := json.Unmarshal([]byte(header), &rep); err != nil {
return nil
}
out := make([]imagegen.SwappedFace, 0, len(rep.Faces))
for _, f := range rep.Faces {
sf := imagegen.SwappedFace{
Index: f.Index,
Yaw: f.Yaw,
IdentitySimilarity: f.IdentitySimilarity,
}
if len(f.Size) == 2 {
sf.Width, sf.Height = f.Size[0], f.Size[1]
}
if len(rep.Image) == 2 {
sf.ImageWidth, sf.ImageHeight = rep.Image[0], rep.Image[1]
}
out = append(out, sf)
}
if len(out) == 0 {
return nil
}
return out
}
// imageFilename picks a multipart filename for an image part. The shim reads
// bytes, not names, but a plausible extension keeps server-side sniffing and
// request logs honest. base distinguishes the parts of a multi-file form
// ("target"/"source") so a log line says which one was malformed.
// ("target"/"source", "frame"/"frame_last") so a log line says which one was
// malformed — and, for the video keyframes, so a backend that stages uploads
// by filename cannot have the second overwrite the first.
//
// initImageFilename (video.go) is this function with base fixed to "frame"
// and delegates here — two copies of one extension table is how they drift.
// Every caller routes through here: two copies of one extension table is how
// they drift.
func imageFilename(mimeType, base string) string {
if base == "" {
base = "image"
+83
View File
@@ -259,3 +259,86 @@ func TestFaceSwapAllowsNegativeIndexUnderAll(t *testing.T) {
t.Error("negative index accepted when it would actually be sent")
}
}
// TestFaceSwapParsesSwapReport: the measured outcome is the whole reason the
// header exists — a caller that cannot tell "the likeness transferred" from
// "an image came back" goes and asks a vision model, which is wrong in
// exactly the cases that matter.
func TestFaceSwapParsesSwapReport(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "image/png")
w.Header().Set("X-Swap-Report",
`{"image":[1010,1200],"faces":[{"index":2,"size":[138,172],"yaw":-82.2,"identity_similarity":0.791}]}`)
raw, _ := base64.StdEncoding.DecodeString(onePixelPNG)
_, _ = w.Write(raw)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
m, err := p.FaceSwapModel("faceswap")
if err != nil {
t.Fatalf("model: %v", err)
}
img := editInit(t)
res, err := m.FaceSwap(context.Background(), imagegen.FaceSwapRequest{Target: img, Source: img})
if err != nil {
t.Fatalf("FaceSwap: %v", err)
}
if len(res.SwappedFaces) != 1 {
t.Fatalf("SwappedFaces = %d, want 1 — the measurement was dropped", len(res.SwappedFaces))
}
f := res.SwappedFaces[0]
if f.Index != 2 || f.Width != 138 || f.Height != 172 {
t.Errorf("face = %+v, want index 2 at 138x172", f)
}
if f.Yaw == nil || *f.Yaw != -82.2 {
t.Errorf("yaw = %v, want -82.2 — the pose signal is how a caller knows a profile swap will not read", f.Yaw)
}
if f.IdentitySimilarity == nil || *f.IdentitySimilarity != 0.791 {
t.Errorf("identity_similarity = %v, want 0.791", f.IdentitySimilarity)
}
// 138/1010 — the number that says "correct, and invisible at a glance".
if got := f.FractionOfImage(); got < 0.13 || got > 0.14 {
t.Errorf("FractionOfImage = %.3f, want ~0.137", got)
}
}
// TestFaceSwapSurvivesMissingReport: an older shim sends no header at all. A
// swap that produced a good image must not fail because the diagnostics
// beside it were absent or malformed.
func TestFaceSwapSurvivesMissingReport(t *testing.T) {
for name, hdr := range map[string]string{
"absent": "",
"garbage": "not json at all",
"wrongtype": `{"faces":"nope"}`,
} {
t.Run(name, func(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "image/png")
if hdr != "" {
w.Header().Set("X-Swap-Report", hdr)
}
raw, _ := base64.StdEncoding.DecodeString(onePixelPNG)
_, _ = w.Write(raw)
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
m, err := p.FaceSwapModel("faceswap")
if err != nil {
t.Fatalf("model: %v", err)
}
img := editInit(t)
res, err := m.FaceSwap(context.Background(), imagegen.FaceSwapRequest{Target: img, Source: img})
if err != nil {
t.Fatalf("a %s report failed the whole swap: %v", name, err)
}
if len(res.Images) != 1 {
t.Fatal("image lost")
}
if res.SwappedFaces != nil {
t.Errorf("SwappedFaces = %+v, want nil for a %s report", res.SwappedFaces, name)
}
})
}
}
+1 -1
View File
@@ -56,7 +56,7 @@ func (m *lipsyncModel) Lipsync(ctx context.Context, req videogen.LipsyncRequest,
// hand (mirrors videoModel.Generate).
var buf bytes.Buffer
w := multipart.NewWriter(&buf)
fw, err := w.CreateFormFile("image", initImageFilename(req.Image.MIME))
fw, err := w.CreateFormFile("image", imageFilename(req.Image.MIME, "frame"))
if err != nil {
return nil, fmt.Errorf("llama-swap: build lipsync form: %w", err)
}
+42 -13
View File
@@ -38,10 +38,13 @@ type videoModel struct {
// bound the call with a context deadline.
//
// Parameter names follow vLLM-Omni's videos API (num_frames, fps,
// num_inference_steps, guidance_scale); the conditioning frame is sent as an
// `input_reference` file part, following OpenAI's videos API. Upstreams
// num_inference_steps, guidance_scale); the leading conditioning frame is sent
// as an `input_reference` file part, following OpenAI's videos API, and a
// trailing keyframe (Request.LastImage) as `input_reference_last`. Upstreams
// ignore fields they don't understand, and optional fields stay off the wire
// entirely so the model's own defaults apply.
// entirely so the model's own defaults apply — which is also why a backend
// without first-last-frame support returns an ordinary clip here rather than
// an error.
func (m *videoModel) Generate(ctx context.Context, req videogen.Request, opts ...videogen.Option) (*videogen.Result, error) {
req = req.Apply(opts...)
if strings.TrimSpace(req.Prompt) == "" {
@@ -56,6 +59,9 @@ func (m *videoModel) Generate(ctx context.Context, req videogen.Request, opts ..
if req.InitImage != nil && len(req.InitImage.Data) == 0 {
return nil, fmt.Errorf("%w: video init image has no bytes", llm.ErrUnsupported)
}
if req.LastImage != nil && len(req.LastImage.Data) == 0 {
return nil, fmt.Errorf("%w: video last image has no bytes", llm.ErrUnsupported)
}
width, height, err := parseSize(req.Size)
if err != nil {
return nil, fmt.Errorf("%w: %v", llm.ErrUnsupported, err)
@@ -83,12 +89,19 @@ func (m *videoModel) Generate(ctx context.Context, req videogen.Request, opts ..
return nil, err
}
if req.InitImage != nil {
fw, err := w.CreateFormFile("input_reference", initImageFilename(req.InitImage.MIME))
if err != nil {
return nil, fmt.Errorf("llama-swap: build video form: %w", err)
if err := writeImagePart(w, "input_reference", "frame", req.InitImage); err != nil {
return nil, err
}
if _, err := fw.Write(req.InitImage.Data); err != nil {
return nil, fmt.Errorf("llama-swap: build video form: %w", err)
}
// The trailing keyframe rides a SEPARATE part rather than a second
// `input_reference`: multipart permits repeated names, but the receiving
// end would then have to rely on part ORDER to tell first from last, and
// an ordering contract that is invisible in the field name is one nobody
// can see they have broken. A backend that does not know the name ignores
// the part, which is the same degradation as any other unknown field.
if req.LastImage != nil {
if err := writeImagePart(w, "input_reference_last", "frame_last", req.LastImage); err != nil {
return nil, err
}
}
if err := w.Close(); err != nil {
@@ -134,11 +147,27 @@ func singleVideoResult(provider, model, verb string, raw []byte, contentType str
return &videogen.Result{Video: videogen.Video{Data: raw, MIME: mimeType}}, nil
}
// initImageFilename picks the multipart filename hint for the conditioning
// frame from its MIME subtype. The name is provider-chosen (never
// caller-supplied), so no sanitization is needed.
func initImageFilename(mimeType string) string {
return imageFilename(mimeType, "frame")
// writeImagePart attaches one conditioning frame under the given field name,
// with a filename derived from nameStem. Shared by the first- and last-frame
// parts so the two cannot drift in how they encode, which is the usual way a
// second copy of a block goes wrong.
//
// The two frames MUST carry DISTINCT filenames, not merely distinct field
// names. Backends commonly stage an uploaded frame under a name derived from
// the filename — our own ComfyUI shim posts to /upload/image with
// overwrite=true — so two parts sharing "frame.png" would have the second
// clobber the first, and BOTH keyframe inputs would then resolve to the same
// stored image. The clip would render clean, pinned at both ends to the same
// frame, with nothing anywhere reporting a problem.
func writeImagePart(w *multipart.Writer, field, nameStem string, img *videogen.Image) error {
fw, err := w.CreateFormFile(field, imageFilename(img.MIME, nameStem))
if err != nil {
return fmt.Errorf("llama-swap: build video form: %w", err)
}
if _, err := fw.Write(img.Data); err != nil {
return fmt.Errorf("llama-swap: build video form: %w", err)
}
return nil
}
// formatInt renders an optional int pointer for a form field; nil = "" (omit).
+131
View File
@@ -223,3 +223,134 @@ func TestVideoGenerateNonVideoBodyErrors(t *testing.T) {
t.Errorf("message = %q, want mention of non-video body", apiErr.Message)
}
}
// Both keyframes reach the wire, under DISTINCT field names.
//
// The distinct-name property is the actual contract with the backend shim: the
// two frames could have shared one repeated `input_reference` name, and then
// which is first and which is last would depend on multipart part ORDER — an
// ordering contract invisible in the payload, that nothing would notice
// breaking. Asserting the names is what pins it.
func TestVideoGenerateSendsBothKeyframes(t *testing.T) {
var gotFirst, gotLast []byte
var firstName, lastName string
var sawLastPart bool
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if err := r.ParseMultipartForm(32 << 20); err != nil {
t.Errorf("parse form: %v", err)
return
}
if f, hdr, err := r.FormFile("input_reference"); err == nil {
gotFirst, _ = io.ReadAll(f)
firstName = hdr.Filename
f.Close()
}
if f, hdr, err := r.FormFile("input_reference_last"); err == nil {
sawLastPart = true
gotLast, _ = io.ReadAll(f)
lastName = hdr.Filename
f.Close()
}
w.Header().Set("Content-Type", "video/mp4")
_, _ = w.Write([]byte("fake-mp4-bytes"))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
vm, err := p.VideoModel("videogen-minimax-h3")
if err != nil {
t.Fatalf("VideoModel: %v", err)
}
first, _ := base64.StdEncoding.DecodeString(onePixelPNG)
last := append(append([]byte{}, first...), 0x00) // distinguishable from first
if _, err := vm.Generate(context.Background(), videogen.Request{
Prompt: "a cat surfing",
InitImage: &videogen.Image{MIME: "image/png", Data: first},
LastImage: &videogen.Image{MIME: "image/png", Data: last},
}); err != nil {
t.Fatalf("Generate: %v", err)
}
if !sawLastPart {
t.Fatal("input_reference_last was not sent — a pinned end frame would be silently dropped")
}
if string(gotFirst) != string(first) {
t.Errorf("input_reference = %d bytes, want %d", len(gotFirst), len(first))
}
if string(gotLast) != string(last) {
t.Errorf("input_reference_last = %d bytes, want %d", len(gotLast), len(last))
}
// The two must not be the same bytes, or a swap/aliasing bug reads as a pass.
if string(gotFirst) == string(gotLast) {
t.Error("both parts carry identical bytes — the frames are being aliased")
}
// DISTINCT FILENAMES, not just distinct field names. Backends stage an
// uploaded frame under a name derived from the filename (our ComfyUI shim
// posts to /upload/image with overwrite=true), so two parts sharing
// "frame.png" would have the second clobber the first and BOTH keyframes
// would resolve to the same stored image — a clip pinned at both ends to
// the same frame, rendering cleanly with nothing reporting a fault.
if firstName == "" || lastName == "" {
t.Fatalf("filenames = %q / %q, want both set", firstName, lastName)
}
if firstName == lastName {
t.Errorf("both parts use filename %q — the second upload would clobber the first", firstName)
}
}
// LastImage alone (no InitImage) is a legitimate request: pin the destination
// and let the model invent the approach. It must not require a first frame.
func TestVideoGenerateLastImageAloneIsAllowed(t *testing.T) {
var sawFirst, sawLast bool
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if err := r.ParseMultipartForm(32 << 20); err != nil {
t.Errorf("parse form: %v", err)
return
}
if f, _, err := r.FormFile("input_reference"); err == nil {
sawFirst = true
f.Close()
}
if f, _, err := r.FormFile("input_reference_last"); err == nil {
sawLast = true
f.Close()
}
w.Header().Set("Content-Type", "video/mp4")
_, _ = w.Write([]byte("fake-mp4-bytes"))
}))
defer srv.Close()
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
vm, _ := p.VideoModel("videogen-minimax-h3")
frame, _ := base64.StdEncoding.DecodeString(onePixelPNG)
if _, err := vm.Generate(context.Background(),
videogen.Request{Prompt: "arrive here"},
videogen.WithLastImage(videogen.Image{MIME: "image/png", Data: frame}),
); err != nil {
t.Fatalf("Generate: %v", err)
}
if sawFirst {
t.Error("input_reference sent, want omitted")
}
if !sawLast {
t.Error("input_reference_last omitted, want sent")
}
}
// An empty LastImage is rejected before the request is built, matching
// InitImage's existing contract — a zero-byte frame reaching the backend is a
// confusing upstream error instead of a clear local one.
func TestVideoGenerateRejectsEmptyLastImage(t *testing.T) {
p := New(WithBaseURL("http://unused"))
vm, _ := p.VideoModel("videogen-minimax-h3")
_, err := vm.Generate(context.Background(), videogen.Request{
Prompt: "x",
LastImage: &videogen.Image{MIME: "image/png"},
})
if !errors.Is(err, llm.ErrUnsupported) {
t.Fatalf("err = %v, want llm.ErrUnsupported", err)
}
}
+1 -1
View File
@@ -263,7 +263,7 @@ func (r *Registry) providerFor(name string) (llm.Provider, error) {
return nil, envErr
}
envKey := "LLM_" + strings.ToUpper(strings.ReplaceAll(name, "-", "_"))
envKey := envKeyForProvider(name)
envVal := r.envLookup(envKey)
if envVal == "" {
return nil, fmt.Errorf("%w: %q (checked registry and %s env var)", ErrUnknownProvider, name, envKey)
+19
View File
@@ -12,6 +12,8 @@
// InitImage is a pure text prompt, a non-nil InitImage conditions generation
// on that frame. Hybrid models (e.g. Wan 2.2 TI2V) serve both from the same
// checkpoint, so unlike imagegen there is no separate Editor-style interface.
// LastImage extends the same surface to the other end of the clip, so one
// Request covers t2v, i2v, and first-last-frame-to-video without a mode flag.
//
// The first implementation is provider/llamaswap, which targets the blocking
// OpenAI/vLLM-Omni-style POST /v1/videos/sync endpoint: the response body is
@@ -51,6 +53,19 @@ type Request struct {
// nil = pure text-to-video.
InitImage *Image
// LastImage conditions generation on an ENDING frame. With InitImage it
// pins both ends (first-last-frame-to-video); alone it pins only the
// destination and lets the backend invent the approach.
//
// Support is per-model and NOT advertised anywhere in this contract: a
// backend that does not understand a trailing keyframe ignores it and
// returns an ordinary clip, which is indistinguishable from success.
// There is no capability bit to consult, because the contract has no way
// to learn one. A caller that needs to know whether the pin actually took
// effect must establish that out of band — by configuration it controls,
// not by inspecting the result.
LastImage *Image
// Size is the requested resolution, e.g. "1280x704"; "" = backend default.
Size string
@@ -92,6 +107,10 @@ type Option func(*Request)
// WithInitImage conditions generation on a starting frame (image-to-video).
func WithInitImage(img Image) Option { return func(r *Request) { r.InitImage = &img } }
// WithLastImage conditions generation on an ending frame. Combined with
// WithInitImage this pins both ends of the clip.
func WithLastImage(img Image) Option { return func(r *Request) { r.LastImage = &img } }
// WithSize sets the requested resolution (e.g. "1280x704").
func WithSize(size string) Option { return func(r *Request) { r.Size = size } }