Compare commits
25
Commits
a941f5ff4a
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
85561ab477 | ||
|
|
f837115a55 | ||
|
|
f8ced9c629 | ||
|
|
0760cf96d4 | ||
|
|
8670ed22be | ||
|
|
f1f2b653c3 | ||
|
|
31d6b59356 | ||
|
|
02cd561eaf | ||
|
|
e779169416 | ||
|
|
5994d96921 | ||
|
|
588e092465 | ||
|
|
dbc96898ab | ||
|
|
44fcfbb273 | ||
|
|
203895696c | ||
|
|
1bbbdaa1e5 | ||
|
|
21b4775d16 | ||
|
|
127966bb3a | ||
|
|
0bd14e01b3 | ||
|
|
6995a8dee1 | ||
|
|
ae2615ca68 | ||
|
|
ff832cb6b5 | ||
|
|
372bf826aa | ||
|
|
0ff90d80f6 | ||
|
|
316a430116 | ||
|
|
2c70d32fd4 |
@@ -7,6 +7,7 @@ OLLAMA_API_KEY=your-ollama-cloud-key-here
|
||||
# Built-in provider keys (each optional; only needed for the providers you use).
|
||||
#OPENAI_API_KEY=sk-...
|
||||
#KIMI_API_KEY=sk-... # Moonshot AI (Kimi); provider name "kimi"
|
||||
#QWEN_API_KEY=sk-... # Alibaba Model Studio (Qwen); provider name "qwen"
|
||||
#ANTHROPIC_API_KEY=sk-ant-...
|
||||
#GOOGLE_API_KEY=...
|
||||
|
||||
|
||||
@@ -28,3 +28,6 @@ go.work.sum
|
||||
|
||||
# macOS
|
||||
.DS_Store
|
||||
|
||||
# Local worktrees created for agent work — never part of the repo.
|
||||
.claude/
|
||||
|
||||
@@ -122,6 +122,7 @@ Chains are health-tracked per target:
|
||||
|----------|-----------|-------------|------------------|
|
||||
| OpenAI (+compatible) | `openai` | `OPENAI_API_KEY` | https://api.openai.com/v1 |
|
||||
| Kimi (Moonshot AI) | `kimi` | `KIMI_API_KEY` | https://api.moonshot.ai/v1 |
|
||||
| Qwen (Alibaba) | `qwen` | `QWEN_API_KEY` | https://dashscope-intl.aliyuncs.com/compatible-mode/v1 |
|
||||
| Anthropic (+compatible) | `anthropic` | `ANTHROPIC_API_KEY` | https://api.anthropic.com |
|
||||
| Google (Gemini) | `google` | `GOOGLE_API_KEY` / `GEMINI_API_KEY` | Gemini API (official SDK) |
|
||||
| Ollama Cloud | `ollama-cloud` | `OLLAMA_API_KEY` | https://ollama.com |
|
||||
@@ -134,6 +135,19 @@ the openai client (like llama-swap). The `kimi` built-in defaults to the
|
||||
international endpoint; reach the China endpoint (or any other host) with a
|
||||
`kimi://` DSN, e.g. `LLM_KCN=kimi://[email protected]/v1`.
|
||||
|
||||
Qwen is the same shape: Alibaba Model Studio's OpenAI-compatible mode, reusing
|
||||
the openai client. The `qwen` built-in defaults to the international
|
||||
(Singapore) host; reach the China host or a workspace-scoped regional one with
|
||||
a `qwen://` DSN, e.g.
|
||||
`LLM_QCN=qwen://[email protected]/compatible-mode/v1`. Model Studio
|
||||
also fronts the same models with an Anthropic-compatible `/v1/messages` shim —
|
||||
majordomo does **not** use it, because on that surface `reasoning_effort` is
|
||||
dropped, `Request.Schema` stops being enforced, and cached-token accounting
|
||||
disappears; see [ADR-0027](docs/adr/0027-qwen-builtin.md). Two Alibaba-side
|
||||
quirks are worth knowing: thinking is on by default for some models (e.g.
|
||||
`qwen3.7-plus`), and the Qwen3 open-source models require streaming while
|
||||
thinking, so buffered `Generate` calls want a Max/Plus model.
|
||||
|
||||
OpenAI-compatible / Anthropic-compatible endpoints: construct the provider
|
||||
with a name and base URL and register it —
|
||||
|
||||
@@ -165,7 +179,7 @@ m, _ := reg.Parse("m5/qwen3:30b,m1/qwen3:30b,thinking")
|
||||
```
|
||||
|
||||
DSN format: `scheme://[token@]host[/path]`, scheme ∈ `foreman`, `ollama`,
|
||||
`ollama-cloud`, `openai`, `kimi`, `anthropic`, `google`/`gemini`, `llama-swap`,
|
||||
`ollama-cloud`, `openai`, `kimi`, `qwen`, `anthropic`, `google`/`gemini`, `llama-swap`,
|
||||
`llama-swaps`, or any scheme you add with `RegisterScheme`. The token is the
|
||||
credential (bearer token / API key); the base URL is always `https://host[/path]`
|
||||
— except `llama-swap`, which builds `http://host[:port]` since it's local-first
|
||||
@@ -272,17 +286,30 @@ tr, err := tm.Transcribe(ctx, audio.TranscriptionRequest{
|
||||
voices, err := ls.ListVoices(ctx, "kokoro") // []string of voice ids
|
||||
```
|
||||
|
||||
## Video: text-to-video + image-to-video
|
||||
## Video: text-to-video, image-to-video, first-last-frame
|
||||
|
||||
Video generation lives in the `videogen` package (ADR-0019), mirroring
|
||||
imagegen/audio: one small `Model` contract, zero values mean backend
|
||||
defaults, bytes in/out. Text-to-video and image-to-video are one surface —
|
||||
a nil `InitImage` is a pure text prompt; setting it conditions generation
|
||||
on that frame (hybrid checkpoints like Wan 2.2 TI2V serve both). First
|
||||
backend: llama-swap (blocking `/v1/videos/sync`, vLLM-Omni style — the
|
||||
defaults, bytes in/out. All modes are one surface, selected by which
|
||||
keyframes are set rather than by a mode flag:
|
||||
|
||||
| `InitImage` | `LastImage` | mode |
|
||||
|---|---|---|
|
||||
| nil | nil | text-to-video |
|
||||
| set | nil | image-to-video (hybrid checkpoints like Wan 2.2 TI2V serve both) |
|
||||
| set | set | first-last-frame — both ends pinned |
|
||||
| nil | set | pin the destination, model invents the approach |
|
||||
|
||||
First backend: llama-swap (blocking `/v1/videos/sync`, vLLM-Omni style — the
|
||||
response body is the encoded clip, so `Result` carries a single `Video`).
|
||||
Generation runs for minutes; bound the call with a context deadline.
|
||||
|
||||
**`LastImage` support is per-model and cannot be detected.** A backend that
|
||||
does not understand a trailing keyframe ignores the part and returns an
|
||||
ordinary clip — indistinguishable from success. There is no capability bit,
|
||||
because the contract has no way to learn one, so a caller depending on the
|
||||
pin must establish support out of band.
|
||||
|
||||
```go
|
||||
vm, _ := ls.VideoModel("videogen-wan22-5b")
|
||||
res, err := vm.Generate(ctx, videogen.Request{Prompt: "a cat surfing"},
|
||||
@@ -407,6 +434,7 @@ to build one.
|
||||
|----------------------|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|
||||
| OpenAI (+compatible) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
|
||||
| Kimi (Moonshot AI) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅³ | ✅ |
|
||||
| Qwen (Alibaba) | ✅ | ✅ | ✅ | ✅ | ✅⁴ | ✅⁴ | ✅ |
|
||||
| Anthropic (+compat) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
|
||||
| Google (Gemini) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
|
||||
| Ollama Cloud | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
|
||||
@@ -431,6 +459,13 @@ probe and management methods on `*llamaswap.Provider`.
|
||||
level; whether a call succeeds depends on the Moonshot model — only the vision
|
||||
variants (e.g. `moonshot-v1-8k-vision-preview`) accept images.
|
||||
|
||||
⁴ Qwen also reuses the openai client (ADR-0027), so both columns are present at
|
||||
the client level and gated by the Model Studio model you name: `json_schema`
|
||||
structured output is on the Max/Plus families, image inputs on the `qwen-vl-*`
|
||||
/ `qwen3-vl-*` models. `reasoning_effort` rides through as a top-level field —
|
||||
one reason the built-in speaks OpenAI-compat rather than Model Studio's
|
||||
Anthropic-compat shim.
|
||||
|
||||
Notes: Ollama has no native tool_choice — `"none"` drops the tools;
|
||||
`"required"`/named choices are best-effort ignored there. Ollama Cloud
|
||||
ignores the `format` field (verified live), so the provider also states
|
||||
|
||||
+145
-25
@@ -14,8 +14,8 @@ import (
|
||||
// their answer to the final, tool-free turn. But some models — notably several
|
||||
// open-weight ones — "front-load" their full answer into an earlier turn that
|
||||
// ALSO calls a tool (e.g. answer text alongside a citation call), then close
|
||||
// with a degenerate terminal turn that is not itself the answer. Two shapes are
|
||||
// recovered from the transcript (zero extra model calls):
|
||||
// with a degenerate terminal turn that is not itself the answer. Three shapes
|
||||
// are recovered from the transcript (zero extra model calls):
|
||||
//
|
||||
// - a trivial back-reference ("(Already answered above.)", "see above", …):
|
||||
// the real answer sits earlier, so recover it and DISCARD the worthless
|
||||
@@ -25,25 +25,44 @@ import (
|
||||
// glm-5.2 "cite" pattern behind mort issue #1418). The citations are real,
|
||||
// useful content — unlike a back-reference — so recover the prior answer and
|
||||
// KEEP the citations, appended below it.
|
||||
// - a bookkeeping closer ("Citations are logged. Short version: …"): the
|
||||
// model acknowledged the citation round and compressed the answer it had
|
||||
// already written into a one-liner (mort run b3cb9ee9 — a 2,089-char answer
|
||||
// shrank to a 153-byte closer at delivery). The compression is strictly
|
||||
// poorer than the front-loaded answer, so recover the prior turn and
|
||||
// DISCARD the closer — but only when the prior turn clearly dwarfs it,
|
||||
// because unlike a back-reference this closer DOES carry answer content
|
||||
// (see modeSummary).
|
||||
//
|
||||
// A citations addendum is tested first and wins over the back-reference test (a
|
||||
// short terminal can be both), so its links are never discarded. When the
|
||||
// terminal text stands on its own it is returned unchanged; when it is
|
||||
// degenerate but nothing better can be recovered, it is returned as-is (a bare
|
||||
// sources list still beats nothing).
|
||||
// A citations addendum is tested first and wins over the other two (a short
|
||||
// terminal can match more than one shape), so its links are never discarded.
|
||||
// The back-reference test wins over the summary-closer test: a terminal
|
||||
// matching both ("Citations are logged. As I said above…") carries no answer
|
||||
// content of its own, so the looser back-ref recovery bar — not the summary
|
||||
// closer's dwarf ratio — is the right one. When the terminal text stands
|
||||
// on its own it is returned unchanged; when it is degenerate but nothing
|
||||
// better can be recovered, it is returned as-is (a compressed answer still
|
||||
// beats nothing).
|
||||
//
|
||||
// msgs must already include the terminal assistant message as its last element
|
||||
// (the loop appends it before calling this); terminal is that message's text.
|
||||
func finalOutput(msgs []llm.Message, terminal string) string {
|
||||
citations := isCitationsOnly(terminal)
|
||||
if !citations && !isWeakFinal(terminal) {
|
||||
mode := modeBackRef
|
||||
switch {
|
||||
case isCitationsOnly(terminal):
|
||||
mode = modeCitations
|
||||
case isWeakFinal(terminal):
|
||||
mode = modeBackRef
|
||||
case isSummaryCloser(terminal):
|
||||
mode = modeSummary
|
||||
default:
|
||||
return terminal
|
||||
}
|
||||
rec, ok := lastSubstantiveAssistantText(msgs, terminal, citations)
|
||||
rec, ok := lastSubstantiveAssistantText(msgs, terminal, mode)
|
||||
if !ok {
|
||||
return terminal
|
||||
}
|
||||
if citations {
|
||||
if mode == modeCitations {
|
||||
// Preserve the citations addendum below the recovered answer, unless the
|
||||
// recovered turn already carries it (guards against a duplicate sources
|
||||
// block when the front-loaded turn included its own citations). The
|
||||
@@ -65,11 +84,64 @@ func stripURLAngles(s string) string {
|
||||
return strings.NewReplacer("<", "", ">", "").Replace(s)
|
||||
}
|
||||
|
||||
// recoveryMode selects the bar a prior assistant turn must clear to replace
|
||||
// the terminal turn (see isSubstantiveAnswer) and what finalOutput does with
|
||||
// the terminal once recovery succeeds.
|
||||
type recoveryMode int
|
||||
|
||||
const (
|
||||
// modeBackRef: the terminal is empty or a pure back-reference — worthless
|
||||
// on its own, so any real prior answer replaces it and it is discarded.
|
||||
modeBackRef recoveryMode = iota
|
||||
// modeCitations: the terminal is a sources-only addendum — not a rival
|
||||
// answer, so the dwarf ratio is skipped and the addendum is kept, appended
|
||||
// below the recovered answer.
|
||||
modeCitations
|
||||
// modeSummary: the terminal acknowledges the citation round and may carry
|
||||
// a short compression of the front-loaded answer. Unlike a back-reference
|
||||
// it DOES contain answer content, so it is only replaced when a prior turn
|
||||
// clearly dwarfs it — the ratio is mandatory at every length, the recovery
|
||||
// scan stops at the most recent user message (a compression can only be of
|
||||
// THIS turn's answer; never resurrect one from an earlier question), and
|
||||
// the closer is discarded (its content is a strict subset of what it
|
||||
// replaced).
|
||||
modeSummary
|
||||
)
|
||||
|
||||
// backRefRe matches a terminal turn that merely points back to an earlier
|
||||
// message instead of stating the answer ("(Already answered above.)",
|
||||
// "see above", "as I said", ...).
|
||||
var backRefRe = regexp.MustCompile(`(?i)(already answered|see above|as (i )?(said|mentioned|stated|noted)|answered (that )?above|per my (previous|earlier))`)
|
||||
|
||||
// summaryCloserRe matches a terminal turn that OPENS with a bookkeeping
|
||||
// acknowledgment of the citation round — "Citations are logged.", "Sources
|
||||
// cited.", "Logged the citations." — the shape a model produces when it
|
||||
// front-loaded its answer into an earlier cite-call turn and closes by
|
||||
// acknowledging the tool results, often followed by a "Short version: …"
|
||||
// compression of the answer it already wrote. The ack clause must end at a
|
||||
// sentence terminator ([.!]) DIRECTLY after the verb: "The citations are
|
||||
// recorded in the court transcript…" is a real answer about citations, not
|
||||
// bookkeeping, and must never match. A compression marker without the ack
|
||||
// ("Short version: no.") is deliberately out of scope — a user who asked for
|
||||
// brevity would be answered with exactly that shape, and misclassifying it
|
||||
// would hijack a legitimate answer; an unmatched closer merely keeps today's
|
||||
// behavior (fail closed). Assembled from named fragments so the alternations
|
||||
// stay legible and extendable.
|
||||
const (
|
||||
summaryPreface = `((done|all set|ok(ay)?)[\s,.!:—-]+)?` // optional "Done —" style opener
|
||||
summaryNouns = `(citations?|sources?|references?|claims?)`
|
||||
// "all" appears here AND in summaryArticle on purpose: as a quantifier
|
||||
// between noun and verb ("Citations all logged.") and as a determiner
|
||||
// before the noun ("All claims cited.", "Logged all the citations.").
|
||||
summaryCopulas = `((are|were|have\s+been|all)\s+)*`
|
||||
summaryVerbs = `(logged|recorded|cited|saved|noted|captured|filed)`
|
||||
summaryArticle = `((all|the)\s+)*` // star, not ?: "Logged all the citations."
|
||||
)
|
||||
|
||||
var summaryCloserRe = regexp.MustCompile(`(?i)^` + leadingMarkers + summaryPreface +
|
||||
`(` + summaryArticle + summaryNouns + `\s+` + summaryCopulas + summaryVerbs +
|
||||
`|logged\s+` + summaryArticle + summaryNouns + `)[.!]`)
|
||||
|
||||
// preambleRe matches intent-announcing prefixes ("Let me search...", "I'll
|
||||
// check...") so a preamble is never mistaken for the answer during recovery.
|
||||
var preambleRe = regexp.MustCompile(`(?i)^(let me|let'?s|i'?ll|i will|first[, ]|sure[,. ]|okay[,. ]|on it|checking)`)
|
||||
@@ -83,7 +155,15 @@ var preambleRe = regexp.MustCompile(`(?i)^(let me|let'?s|i'?ll|i will|first[, ]|
|
||||
// the colon/dash separator. Anchored at ^ so a normal answer that merely
|
||||
// mentions "sources" mid-sentence, or ends with a "Sources:" section AFTER its
|
||||
// prose, is never matched.
|
||||
var citationLabelRe = regexp.MustCompile(`(?i)^[\s>#*_+-]*(sources?|references?|citations?|works cited|further reading)\b[\s*_]*[::\-—]`)
|
||||
var citationLabelRe = regexp.MustCompile(`(?i)^` + leadingMarkers +
|
||||
`(sources?|references?|citations?|works cited|further reading)\b[\s*_]*[::\-—]`)
|
||||
|
||||
// leadingMarkers tolerates markdown noise before a label: emphasis (*, _),
|
||||
// list (-, +, *), block-quote (>), and ATX-heading (#) markers, with their
|
||||
// whitespace. Shared by citationLabelRe and summaryCloserRe so the two
|
||||
// classifiers cannot drift apart (the first draft of the summary class
|
||||
// dropped '+' by hand-copying this set).
|
||||
const leadingMarkers = `[\s>#*_+-]*`
|
||||
|
||||
// linkRe matches a whole markdown link "[label](url)" or a bare URL. Used both
|
||||
// to require that a citations terminal carries at least one link and to strip
|
||||
@@ -114,6 +194,12 @@ const (
|
||||
// be at most len/N of the whole, so a prose answer that merely opens with
|
||||
// "Source:" and cites a URL mid-sentence is not mistaken for a bare list.
|
||||
citationDominatedDivisor = 3
|
||||
// summaryCloserMaxChars bounds a summary closer: room for the ack sentence
|
||||
// plus a couple of compression sentences (the b3cb9ee9 closer was 153
|
||||
// bytes — Go len(), which is what every threshold here compares). Beyond
|
||||
// this the "short version" is substantial enough that replacing it risks
|
||||
// losing content the front-loaded turn never had.
|
||||
summaryCloserMaxChars = 300
|
||||
)
|
||||
|
||||
// isWeakFinal reports whether a terminal turn's text fails to stand on its own
|
||||
@@ -152,14 +238,39 @@ func isCitationsOnly(s string) bool {
|
||||
return len(residue) <= len(t)/citationDominatedDivisor
|
||||
}
|
||||
|
||||
// isSummaryCloser reports whether a terminal turn is a bookkeeping closer: it
|
||||
// opens with a complete "citations are logged"-style ack sentence (see
|
||||
// summaryCloserRe) and is short enough that whatever follows the ack can only
|
||||
// be a compression of an earlier, fuller answer. Whether that fuller answer
|
||||
// actually exists is modeSummary's job — the dwarf ratio in
|
||||
// isSubstantiveAnswer keeps a matching closer in place when nothing earlier
|
||||
// clearly outweighs it.
|
||||
func isSummaryCloser(s string) bool {
|
||||
t := strings.TrimSpace(s)
|
||||
if t == "" || len(t) > summaryCloserMaxChars {
|
||||
return false
|
||||
}
|
||||
return summaryCloserRe.MatchString(t)
|
||||
}
|
||||
|
||||
// lastSubstantiveAssistantText scans msgs newest→oldest (skipping the terminal
|
||||
// turn and empty tool-only turns) for the most recent assistant turn whose text
|
||||
// reads like a real answer. citations selects the recovery bar (see
|
||||
// reads like a real answer. mode selects the recovery bar (see
|
||||
// isSubstantiveAnswer). Returns ("", false) when nothing qualifies.
|
||||
func lastSubstantiveAssistantText(msgs []llm.Message, terminal string, citations bool) (string, bool) {
|
||||
func lastSubstantiveAssistantText(msgs []llm.Message, terminal string, mode recoveryMode) (string, bool) {
|
||||
tt := strings.TrimSpace(terminal)
|
||||
for i := len(msgs) - 1; i >= 0; i-- {
|
||||
m := msgs[i]
|
||||
if mode == modeSummary && m.Role == llm.RoleUser {
|
||||
// A summary closer compresses THIS turn's front-loaded answer, so
|
||||
// the scan must not cross into an earlier question: once the dwarf
|
||||
// ratio has rejected the current turn's text, walking further back
|
||||
// would resurrect a stale answer to a DIFFERENT question — strictly
|
||||
// worse than keeping the closer. (A mid-run steer message is also a
|
||||
// user-role boundary; recovery then fails closed, which is fine.)
|
||||
// The other modes keep their historical unbounded scan.
|
||||
break
|
||||
}
|
||||
if m.Role != llm.RoleAssistant {
|
||||
continue
|
||||
}
|
||||
@@ -167,7 +278,7 @@ func lastSubstantiveAssistantText(msgs []llm.Message, terminal string, citations
|
||||
if txt == "" || txt == tt {
|
||||
continue // the terminal turn itself, or an empty tool-only turn
|
||||
}
|
||||
if isSubstantiveAnswer(txt, tt, citations) {
|
||||
if isSubstantiveAnswer(txt, tt, mode) {
|
||||
return txt, true
|
||||
}
|
||||
}
|
||||
@@ -177,20 +288,29 @@ func lastSubstantiveAssistantText(msgs []llm.Message, terminal string, citations
|
||||
// isSubstantiveAnswer reports whether txt (a prior assistant turn) reads like a
|
||||
// real answer rather than a preamble, relative to the terminal text.
|
||||
//
|
||||
// A sufficiently long turn (>= recoverMinChars) is accepted unconditionally: a
|
||||
// multi-hundred-char turn is an answer even when it opens conversationally
|
||||
// ("Sure, here's…", "Let me explain: …"), so the preamble filter is NOT applied
|
||||
// to it — applying it there would drop a legitimate long front-loaded answer.
|
||||
// Only in the borderline band does a turn have to clear a floor, not read like a
|
||||
// short planning preamble ("Let me look that up…"), and — unless the terminal is
|
||||
// a citations addendum (not a rival answer, so its length is irrelevant) — also
|
||||
// clearly dwarf the terminal.
|
||||
func isSubstantiveAnswer(txt, terminal string, citations bool) bool {
|
||||
// modeSummary demands the dwarf ratio FIRST, at every length: a summary closer
|
||||
// carries a real (compressed) answer, so replacing it is only justified when
|
||||
// the prior turn is clearly the fuller original it was compressed from.
|
||||
//
|
||||
// A sufficiently long turn (>= recoverMinChars) is otherwise accepted
|
||||
// unconditionally: a multi-hundred-char turn is an answer even when it opens
|
||||
// conversationally ("Sure, here's…", "Let me explain: …"), so the preamble
|
||||
// filter is NOT applied to it — applying it there would drop a legitimate long
|
||||
// front-loaded answer. Only in the borderline band does a turn have to clear a
|
||||
// floor, not read like a short planning preamble ("Let me look that up…"),
|
||||
// and — for modeBackRef only — also clearly dwarf the terminal (a citations
|
||||
// addendum is not a rival answer, so its length is irrelevant; a summary
|
||||
// closer already proved the ratio above).
|
||||
func isSubstantiveAnswer(txt, terminal string, mode recoveryMode) bool {
|
||||
dwarfs := len(txt) >= recoverRatio*len(terminal)
|
||||
if mode == modeSummary && !dwarfs {
|
||||
return false
|
||||
}
|
||||
if len(txt) >= recoverMinChars {
|
||||
return true
|
||||
}
|
||||
if len(txt) < recoverFloorChars || preambleRe.MatchString(txt) {
|
||||
return false
|
||||
}
|
||||
return citations || len(txt) >= recoverRatio*len(terminal)
|
||||
return mode != modeBackRef || dwarfs
|
||||
}
|
||||
|
||||
@@ -72,6 +72,46 @@ func TestIsCitationsOnly(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// b3cb9ee9Closer is the verbatim terminal turn from mort run b3cb9ee9: a
|
||||
// 2,089-char answer was front-loaded into the cite-call turn and this 153-byte
|
||||
// compression (151 runes — the em dash is 3 bytes, and byte length is what the
|
||||
// thresholds compare) was all that got delivered.
|
||||
const b3cb9ee9Closer = "Citations are logged. Short version: the bulk of that ~$64M was AIPAC and dark-money super PACs, not the party committees — and it still wasn't enough."
|
||||
|
||||
func TestIsSummaryCloser(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
in string
|
||||
want bool
|
||||
}{
|
||||
{"b3cb9ee9-verbatim", b3cb9ee9Closer, true},
|
||||
{"ack-only", "Citations are logged.", true},
|
||||
{"ack-no-copula", "Citations logged.", true},
|
||||
{"claims-cited", "All claims cited.", true},
|
||||
{"verb-first", "Logged the citations.", true},
|
||||
{"done-prefix", "Done — citations logged.", true},
|
||||
{"ack-then-tldr", "Sources have been recorded! TL;DR: the GPU was the bottleneck.", true},
|
||||
{"references-noted", "References noted. In short: yes, it ships Tuesday.", true},
|
||||
{"plus-list-marker", "+ Citations are logged.", true},
|
||||
{"logged-all-the", "Logged all the citations.", true},
|
||||
|
||||
{"empty", "", false},
|
||||
{"ack-continues-midsentence", "The citations are recorded in the court transcript, which shows the filing dates.", false},
|
||||
{"ack-verb-then-clause", "Citations are logged in Zotero whenever you click the save button.", false},
|
||||
{"compression-without-ack", "Short version: yes.", false}, // deliberately out of scope
|
||||
{"mentions-citations-midsentence", "The paper's citations are what got it retracted.", false},
|
||||
{"crisp-number", "42", false},
|
||||
{"over-cap", "Citations are logged. " + strings.Repeat("The long version has many more details worth keeping. ", 6), false}, // >300: too substantial to replace
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
if got := isSummaryCloser(c.in); got != c.want {
|
||||
t.Errorf("isSummaryCloser(%q) = %v, want %v", c.in, got, c.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func asst(text string, tools ...llm.ToolCall) llm.Message {
|
||||
m := llm.Message{Role: llm.RoleAssistant}
|
||||
if text != "" {
|
||||
@@ -84,6 +124,7 @@ func asst(text string, tools ...llm.ToolCall) llm.Message {
|
||||
func TestFinalOutput(t *testing.T) {
|
||||
cite := []llm.ToolCall{{ID: "c1", Name: "cite", Arguments: json.RawMessage(`{}`)}}
|
||||
longAnswer := strings.TrimSpace(strings.Repeat("Free group calls are capped at sixty minutes. ", 6)) // >200
|
||||
hugeAnswer := strings.TrimSpace(strings.Repeat("Free group calls are capped at sixty minutes. ", 12)) // >3x the b3cb9ee9 closer
|
||||
// A sources/citations-only terminal — the glm-5.2 "cite" shape behind mort
|
||||
// issue #1418: the prose answer was front-loaded into the tool-call turn and
|
||||
// the terminal turn carried only the citations.
|
||||
@@ -103,6 +144,9 @@ func TestFinalOutput(t *testing.T) {
|
||||
// A >=200-byte real answer that merely OPENS with a conversational word
|
||||
// ("Sure,"). The preamble filter must NOT veto it (gadfly regression guard).
|
||||
longConversationalAnswer := "Sure, here's the rundown: it currently sells for about $2,700 used on eBay, typically $2,400 to $2,900 depending on condition and bundle, with the sealed Founders Edition commanding the top of that range while used AIB cards go a bit lower."
|
||||
// Matches BOTH the summary ack and backRefRe, within the 120-byte weak cap,
|
||||
// and long enough (>~92 bytes) that longAnswer would fail the summary bar.
|
||||
bothMatchCloser := "Citations are logged. As I mentioned above, the full detail on the money sources is in my earlier message."
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
@@ -256,6 +300,109 @@ func TestFinalOutput(t *testing.T) {
|
||||
terminal: sources,
|
||||
want: longConversationalAnswer + "\n\n" + sources,
|
||||
},
|
||||
{
|
||||
// The b3cb9ee9 shape: full answer front-loaded into the cite turn,
|
||||
// then a summary closer. The closer is discarded — its content is a
|
||||
// strict compression of the recovered answer.
|
||||
name: "summary closer discarded when the front-loaded answer dwarfs it",
|
||||
msgs: []llm.Message{
|
||||
llm.UserText("where did the $64M come from?"),
|
||||
asst(hugeAnswer, cite...),
|
||||
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
|
||||
asst(b3cb9ee9Closer),
|
||||
},
|
||||
terminal: b3cb9ee9Closer,
|
||||
want: hugeAnswer,
|
||||
},
|
||||
{
|
||||
// The dwarf ratio is mandatory for a summary closer at EVERY length:
|
||||
// a prior turn that is longer but not clearly the fuller original
|
||||
// (here ~275 chars vs a 151-char closer, under the 3x bar) must not
|
||||
// displace a closer that carries real answer content.
|
||||
name: "summary closer kept when the prior turn does not dwarf it",
|
||||
msgs: []llm.Message{
|
||||
llm.UserText("q?"),
|
||||
asst(longAnswer, cite...),
|
||||
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
|
||||
asst(b3cb9ee9Closer),
|
||||
},
|
||||
terminal: b3cb9ee9Closer,
|
||||
want: b3cb9ee9Closer,
|
||||
},
|
||||
{
|
||||
// An ack-only closer ("Citations are logged.") is tiny, so even a
|
||||
// modest front-loaded answer clears the ratio and replaces it.
|
||||
name: "ack-only summary closer recovered over a modest answer",
|
||||
msgs: []llm.Message{
|
||||
llm.UserText("q?"),
|
||||
asst(longAnswer, cite...),
|
||||
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
|
||||
asst("Citations are logged."),
|
||||
},
|
||||
terminal: "Citations are logged.",
|
||||
want: longAnswer,
|
||||
},
|
||||
{
|
||||
name: "summary closer with only a preamble prior keeps the closer",
|
||||
msgs: []llm.Message{
|
||||
llm.UserText("q?"),
|
||||
asst("Let me gather the numbers.", cite...),
|
||||
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
|
||||
asst(b3cb9ee9Closer),
|
||||
},
|
||||
terminal: b3cb9ee9Closer,
|
||||
want: b3cb9ee9Closer,
|
||||
},
|
||||
{
|
||||
// The modeSummary scan must stop at the most recent user message.
|
||||
// Here the current turn's answer sits in the 1x-3x band (rejected
|
||||
// by the ratio) while a dwarfing answer to a DIFFERENT question
|
||||
// sits in history — resurrecting it would be strictly worse than
|
||||
// keeping the closer.
|
||||
name: "summary closer never resurrects a stale answer across the user boundary",
|
||||
msgs: []llm.Message{
|
||||
llm.UserText("earlier, unrelated question?"),
|
||||
asst(hugeAnswer),
|
||||
llm.UserText("q?"),
|
||||
asst(conciseAnswer, cite...),
|
||||
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
|
||||
asst(b3cb9ee9Closer),
|
||||
},
|
||||
terminal: b3cb9ee9Closer,
|
||||
want: b3cb9ee9Closer,
|
||||
},
|
||||
{
|
||||
// The boundary must not break the legitimate multi-turn case: the
|
||||
// dwarfing front-loaded answer in THIS turn's window is recovered
|
||||
// even with history behind it.
|
||||
name: "summary closer recovery still works with history present",
|
||||
msgs: []llm.Message{
|
||||
llm.UserText("earlier, unrelated question?"),
|
||||
asst(longAnswer),
|
||||
llm.UserText("q?"),
|
||||
asst(hugeAnswer, cite...),
|
||||
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
|
||||
asst(b3cb9ee9Closer),
|
||||
},
|
||||
terminal: b3cb9ee9Closer,
|
||||
want: hugeAnswer,
|
||||
},
|
||||
{
|
||||
// A closer matching BOTH the ack shape and a back-reference
|
||||
// carries no answer content, so the back-ref test must win and the
|
||||
// ordinary recovery bar apply — under the summary bar this
|
||||
// ~106-byte terminal would demand a ~318-byte prior and wrongly
|
||||
// keep the closer over longAnswer.
|
||||
name: "back-reference wins over the summary ack when both match",
|
||||
msgs: []llm.Message{
|
||||
llm.UserText("q?"),
|
||||
asst(longAnswer, cite...),
|
||||
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
|
||||
asst(bothMatchCloser),
|
||||
},
|
||||
terminal: bothMatchCloser,
|
||||
want: longAnswer,
|
||||
},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
@@ -326,6 +473,37 @@ func TestRun_HealthyTerminalUnchanged(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestRun_RecoversFrontLoadedAnswerOverSummaryCloser reproduces mort run
|
||||
// b3cb9ee9 end-to-end: the model front-loads its full answer into the
|
||||
// cite-call turn, the cite results come back, and the terminal turn is only a
|
||||
// bookkeeping ack plus a one-line compression. The delivered output must be
|
||||
// the front-loaded answer, with no extra model call.
|
||||
func TestRun_RecoversFrontLoadedAnswerOverSummaryCloser(t *testing.T) {
|
||||
hugeAnswer := strings.TrimSpace(strings.Repeat("Free group calls are capped at sixty minutes. ", 12))
|
||||
fp := fake.New("fp")
|
||||
fp.Enqueue("test-model",
|
||||
fake.ReplyWith(llm.Response{
|
||||
Parts: []llm.Part{llm.Text(hugeAnswer)},
|
||||
ToolCalls: []llm.ToolCall{{ID: "c1", Name: "cite", Arguments: json.RawMessage(`{}`)}},
|
||||
FinishReason: llm.FinishToolCalls,
|
||||
Usage: llm.Usage{InputTokens: 10, OutputTokens: 5},
|
||||
}),
|
||||
fake.Reply(b3cb9ee9Closer),
|
||||
)
|
||||
|
||||
a := New(newModel(t, fp), "sys", WithToolbox(citeToolbox(t)))
|
||||
res, err := a.Run(context.Background(), "where did the $64M come from?")
|
||||
if err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
if res.Output != hugeAnswer {
|
||||
t.Errorf("Output = %q, want recovered front-loaded answer", res.Output)
|
||||
}
|
||||
if n := len(fp.Calls()); n != 2 {
|
||||
t.Errorf("model calls = %d, want 2 (no extra nudge turn)", n)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRun_RecoversFrontLoadedAnswerWithCitations reproduces mort issue #1418
|
||||
// end-to-end: the model front-loads the prose answer into the tool-call turn
|
||||
// and closes with a sources-only terminal turn. The delivered output must be
|
||||
|
||||
+68
-27
@@ -2,7 +2,6 @@ package majordomo
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
|
||||
"gitea.stevedudenhoeffer.com/steve/majordomo/provider/anthropic"
|
||||
@@ -19,6 +18,12 @@ const (
|
||||
// Chat Completions endpoint. Reuses the openai client (like llama-swap);
|
||||
// keyed by KIMI_API_KEY, default base URL kimiBaseURL.
|
||||
ProviderKimi = "kimi"
|
||||
// ProviderQwen is Alibaba's Qwen models over Model Studio's
|
||||
// OpenAI-compatible Chat Completions endpoint. Reuses the openai client
|
||||
// (like kimi and llama-swap); keyed by QWEN_API_KEY, default base URL
|
||||
// qwenBaseURL. ADR-0027 records why the OpenAI surface and not the
|
||||
// Anthropic-compatible one Model Studio also exposes.
|
||||
ProviderQwen = "qwen"
|
||||
ProviderAnthropic = "anthropic"
|
||||
ProviderGoogle = "google"
|
||||
ProviderOllama = "ollama"
|
||||
@@ -37,6 +42,55 @@ const (
|
||||
// China endpoint (api.moonshot.cn/v1) is reachable via a kimi:// LLM_* DSN.
|
||||
const kimiBaseURL = "https://api.moonshot.ai/v1"
|
||||
|
||||
// qwenBaseURL is Alibaba Model Studio's international (Singapore) endpoint in
|
||||
// OpenAI-compatible mode. The China endpoint
|
||||
// (dashscope.aliyuncs.com/compatible-mode/v1) and any regional host are
|
||||
// reachable via a qwen:// LLM_* DSN.
|
||||
const qwenBaseURL = "https://dashscope-intl.aliyuncs.com/compatible-mode/v1"
|
||||
|
||||
// openaiCompatScheme builds the DSN factory shared by every built-in that is
|
||||
// "the openai client pointed somewhere else" (kimi, qwen, ...). The provider
|
||||
// is named after the LLM_<NAME> var that defined it, takes its credential from
|
||||
// the DSN token — not the built-in's own env var, which does nothing for a
|
||||
// DSN-defined provider — and so names that same LLM_<NAME> var in the
|
||||
// missing-key hint, matching the lazy-resolution key form in providerFor.
|
||||
//
|
||||
// wrap is the caller's option-decorator (it injects the registry's HTTP
|
||||
// client), so a DSN provider is built exactly like the eager built-ins.
|
||||
func openaiCompatScheme(wrap func(...openai.Option) []openai.Option) SchemeFactory {
|
||||
return func(name string, dsn DSN) (llm.Provider, error) {
|
||||
return openai.New(wrap(
|
||||
openai.WithName(name),
|
||||
openai.WithBaseURL(dsn.BaseURL()),
|
||||
openai.WithAPIKey(dsn.Token),
|
||||
openai.WithAPIKeyName(envKeyForProvider(name)),
|
||||
)...), nil
|
||||
}
|
||||
}
|
||||
|
||||
// registerOpenAICompatBuiltin installs BOTH halves of an OpenAI-compat
|
||||
// built-in: the eager provider under name (credential from keyEnv) and the
|
||||
// matching name:// DSN scheme. Why both in one call: the two halves are a pair
|
||||
// — a built-in whose scheme is missing resolves as a spec but not from an
|
||||
// LLM_* DSN, and the credential rules below have to hold identically in each.
|
||||
// Adding the next one is a single line rather than six lines to copy.
|
||||
//
|
||||
// The two credential rules, holding by construction for every caller:
|
||||
// - WithAPIKey is passed UNCONDITIONALLY, even when the lookup comes back
|
||||
// empty. openai.New defaults its key to OPENAI_API_KEY, so anything less
|
||||
// lets an unset keyEnv silently authenticate as OpenAI.
|
||||
// - WithAPIKeyName makes the synthetic-401 hint name keyEnv, so a keyless
|
||||
// call tells the operator the variable that actually fixes it.
|
||||
func registerOpenAICompatBuiltin(r *Registry, wrap func(...openai.Option) []openai.Option, name, baseURL, keyEnv string) {
|
||||
r.providers[name] = openai.New(wrap(
|
||||
openai.WithName(name),
|
||||
openai.WithBaseURL(baseURL),
|
||||
openai.WithAPIKey(r.envLookup(keyEnv)),
|
||||
openai.WithAPIKeyName(keyEnv),
|
||||
)...)
|
||||
r.schemes[name] = openaiCompatScheme(wrap)
|
||||
}
|
||||
|
||||
// registerBuiltins installs the built-in providers and env-DSN scheme
|
||||
// factories into a fresh registry. httpClient, when non-nil, is used by
|
||||
// every provider and factory the registry itself constructs.
|
||||
@@ -83,32 +137,19 @@ func registerBuiltins(r *Registry, httpClient *http.Client) {
|
||||
)...), nil
|
||||
}
|
||||
|
||||
// Kimi (Moonshot AI): OpenAI-compatible Chat Completions, so it reuses the
|
||||
// openai client (like llama-swap). Defaults to Moonshot's international
|
||||
// endpoint and the KIMI_API_KEY credential. WithAPIKey is passed
|
||||
// unconditionally — even empty — so an unset KIMI_API_KEY can never fall
|
||||
// through to the openai client's OPENAI_API_KEY default; WithAPIKeyName
|
||||
// makes the missing-key error name KIMI_API_KEY.
|
||||
r.providers[ProviderKimi] = openai.New(openaiOpts(
|
||||
openai.WithName(ProviderKimi),
|
||||
openai.WithBaseURL(kimiBaseURL),
|
||||
openai.WithAPIKey(r.envLookup("KIMI_API_KEY")),
|
||||
openai.WithAPIKeyName("KIMI_API_KEY"),
|
||||
)...)
|
||||
// kimi:// DSN scheme: an OpenAI-compatible target labeled kimi, base URL
|
||||
// from the DSN host (e.g. kimi://[email protected]/v1 for China). Its
|
||||
// credential is the DSN token, not KIMI_API_KEY, so the missing-key hint
|
||||
// names the LLM_<NAME> env var that defines this provider (matching the
|
||||
// lazy-resolution key form in providerFor) — the fix for a keyless target
|
||||
// here is adding a token to that DSN.
|
||||
r.schemes[ProviderKimi] = func(name string, dsn DSN) (llm.Provider, error) {
|
||||
return openai.New(openaiOpts(
|
||||
openai.WithName(name),
|
||||
openai.WithBaseURL(dsn.BaseURL()),
|
||||
openai.WithAPIKey(dsn.Token),
|
||||
openai.WithAPIKeyName("LLM_"+strings.ToUpper(strings.ReplaceAll(name, "-", "_"))),
|
||||
)...), nil
|
||||
}
|
||||
// Third-party endpoints that ARE the openai client at another base URL —
|
||||
// no new package, mirroring llama-swap's chat path. Each gets the eager
|
||||
// built-in plus its name:// DSN scheme, and the credential rules hold by
|
||||
// construction (see registerOpenAICompatBuiltin).
|
||||
//
|
||||
// kimi (ADR-0026): Moonshot's international endpoint; China host via
|
||||
// kimi://[email protected]/v1.
|
||||
registerOpenAICompatBuiltin(r, openaiOpts, ProviderKimi, kimiBaseURL, "KIMI_API_KEY")
|
||||
// qwen (ADR-0027): Alibaba Model Studio's international host. Model Studio
|
||||
// also exposes an Anthropic-compatible endpoint; the ADR records why the
|
||||
// OpenAI one is the built-in. China / workspace-scoped regional hosts via
|
||||
// qwen://[email protected]/compatible-mode/v1.
|
||||
registerOpenAICompatBuiltin(r, openaiOpts, ProviderQwen, qwenBaseURL, "QWEN_API_KEY")
|
||||
|
||||
// llama-swap: OpenAI-compatible chat + image generation + management
|
||||
// endpoints over a model-swapping proxy. Chat reuses the openai client
|
||||
|
||||
@@ -1,171 +0,0 @@
|
||||
package majordomo
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
|
||||
)
|
||||
|
||||
// kimiResponse is a minimal valid Chat Completions body so Generate returns a
|
||||
// non-empty response (an empty one would trigger failover, not a clean pass).
|
||||
const kimiResponse = `{"id":"c1","object":"chat.completion","choices":[` +
|
||||
`{"index":0,"message":{"role":"assistant","content":"ok"},"finish_reason":"stop"}]}`
|
||||
|
||||
// captureRT records the last request and returns a canned response without
|
||||
// touching the network, so these tests stay hermetic while still exercising
|
||||
// the real openai client the kimi built-in reuses (base URL + auth header).
|
||||
type captureRT struct {
|
||||
req *http.Request
|
||||
body string
|
||||
}
|
||||
|
||||
func (c *captureRT) RoundTrip(r *http.Request) (*http.Response, error) {
|
||||
c.req = r
|
||||
return &http.Response{
|
||||
StatusCode: http.StatusOK,
|
||||
Body: io.NopCloser(strings.NewReader(c.body)),
|
||||
Header: make(http.Header),
|
||||
Request: r,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// TestKimiBuiltin: the built-in "kimi" provider resolves in Parse, targets
|
||||
// Moonshot's default endpoint, and authenticates with KIMI_API_KEY.
|
||||
func TestKimiBuiltin(t *testing.T) {
|
||||
rt := &captureRT{body: kimiResponse}
|
||||
r := newTestRegistry(t,
|
||||
WithEnvLookup(func(k string) string {
|
||||
if k == "KIMI_API_KEY" {
|
||||
return "kimi-secret"
|
||||
}
|
||||
return ""
|
||||
}),
|
||||
WithHTTPClient(&http.Client{Transport: rt}),
|
||||
)
|
||||
|
||||
if p, ok := r.Provider(ProviderKimi); !ok {
|
||||
t.Fatal("built-in kimi provider not registered")
|
||||
} else if p.Name() != ProviderKimi {
|
||||
t.Errorf("name = %q, want %q", p.Name(), ProviderKimi)
|
||||
}
|
||||
|
||||
m, err := r.Parse("kimi/kimi-k2-0711-preview")
|
||||
if err != nil {
|
||||
t.Fatalf("Parse: %v", err)
|
||||
}
|
||||
if got := targetsOf(t, m); len(got) != 1 || got[0] != "kimi/kimi-k2-0711-preview" {
|
||||
t.Fatalf("targets = %v", got)
|
||||
}
|
||||
|
||||
if _, err := m.Generate(context.Background(), llm.Request{Messages: []llm.Message{llm.UserText("hi")}}); err != nil {
|
||||
t.Fatalf("Generate: %v", err)
|
||||
}
|
||||
if rt.req == nil {
|
||||
t.Fatal("no request captured")
|
||||
}
|
||||
if want := "https://api.moonshot.ai/v1/chat/completions"; rt.req.URL.String() != want {
|
||||
t.Errorf("URL = %q, want %q", rt.req.URL.String(), want)
|
||||
}
|
||||
if want := "Bearer kimi-secret"; rt.req.Header.Get("Authorization") != want {
|
||||
t.Errorf("Authorization = %q, want %q", rt.req.Header.Get("Authorization"), want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestKimiBuiltinMissingKey: with no KIMI_API_KEY the built-in fails fast with a
|
||||
// synthetic 401 whose hint names KIMI_API_KEY — never OPENAI_API_KEY (proving
|
||||
// the credential does not fall through to the openai client's default), and
|
||||
// without hitting the network.
|
||||
func TestKimiBuiltinMissingKey(t *testing.T) {
|
||||
rt := &captureRT{body: kimiResponse}
|
||||
r := newTestRegistry(t, WithHTTPClient(&http.Client{Transport: rt}))
|
||||
|
||||
m, err := r.Parse("kimi/kimi-k2-0711-preview")
|
||||
if err != nil {
|
||||
t.Fatalf("Parse: %v", err)
|
||||
}
|
||||
_, err = m.Generate(context.Background(), llm.Request{Messages: []llm.Message{llm.UserText("hi")}})
|
||||
apiErr, ok := errors.AsType[*llm.APIError](err)
|
||||
if !ok {
|
||||
t.Fatalf("err = %v (%T), want *llm.APIError", err, err)
|
||||
}
|
||||
if apiErr.Status != http.StatusUnauthorized || apiErr.Code != "missing_api_key" {
|
||||
t.Errorf("Status/Code = %d/%q, want 401/missing_api_key", apiErr.Status, apiErr.Code)
|
||||
}
|
||||
if !strings.Contains(apiErr.Message, "KIMI_API_KEY") {
|
||||
t.Errorf("message = %q, want it to name KIMI_API_KEY", apiErr.Message)
|
||||
}
|
||||
if strings.Contains(apiErr.Message, "OPENAI_API_KEY") {
|
||||
t.Errorf("message = %q, must not name OPENAI_API_KEY", apiErr.Message)
|
||||
}
|
||||
if rt.req != nil {
|
||||
t.Error("network was hit despite missing key")
|
||||
}
|
||||
}
|
||||
|
||||
// TestKimiScheme: a kimi:// LLM_* DSN defines a named provider on any Moonshot
|
||||
// host (here the China endpoint) that is first-class in Parse and carries the
|
||||
// DSN token as its bearer credential.
|
||||
func TestKimiScheme(t *testing.T) {
|
||||
rt := &captureRT{body: kimiResponse}
|
||||
r := newTestRegistry(t, WithHTTPClient(&http.Client{Transport: rt}))
|
||||
if err := r.LoadEnv(map[string]string{
|
||||
"LLM_KCN": "kimi://[email protected]/v1",
|
||||
}); err != nil {
|
||||
t.Fatalf("LoadEnv: %v", err)
|
||||
}
|
||||
|
||||
m, err := r.Parse("kcn/moonshot-v1-8k")
|
||||
if err != nil {
|
||||
t.Fatalf("Parse: %v", err)
|
||||
}
|
||||
if _, err := m.Generate(context.Background(), llm.Request{Messages: []llm.Message{llm.UserText("hi")}}); err != nil {
|
||||
t.Fatalf("Generate: %v", err)
|
||||
}
|
||||
if rt.req == nil {
|
||||
t.Fatal("no request captured")
|
||||
}
|
||||
if want := "https://api.moonshot.cn/v1/chat/completions"; rt.req.URL.String() != want {
|
||||
t.Errorf("URL = %q, want %q", rt.req.URL.String(), want)
|
||||
}
|
||||
if want := "Bearer tok"; rt.req.Header.Get("Authorization") != want {
|
||||
t.Errorf("Authorization = %q, want %q", rt.req.Header.Get("Authorization"), want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestKimiSchemeMissingToken: a kimi:// DSN with no token is fixed by adding one
|
||||
// to the DSN, not by setting KIMI_API_KEY — so the missing-key hint names the
|
||||
// defining LLM_<NAME> env var, never KIMI_API_KEY (which does nothing for a
|
||||
// DSN-defined provider).
|
||||
func TestKimiSchemeMissingToken(t *testing.T) {
|
||||
rt := &captureRT{body: kimiResponse}
|
||||
r := newTestRegistry(t, WithHTTPClient(&http.Client{Transport: rt}))
|
||||
if err := r.LoadEnv(map[string]string{
|
||||
"LLM_KCN": "kimi://api.moonshot.cn/v1", // no token
|
||||
}); err != nil {
|
||||
t.Fatalf("LoadEnv: %v", err)
|
||||
}
|
||||
|
||||
m, err := r.Parse("kcn/moonshot-v1-8k")
|
||||
if err != nil {
|
||||
t.Fatalf("Parse: %v", err)
|
||||
}
|
||||
_, err = m.Generate(context.Background(), llm.Request{Messages: []llm.Message{llm.UserText("hi")}})
|
||||
apiErr, ok := errors.AsType[*llm.APIError](err)
|
||||
if !ok {
|
||||
t.Fatalf("err = %v (%T), want *llm.APIError", err, err)
|
||||
}
|
||||
if !strings.Contains(apiErr.Message, "LLM_KCN") {
|
||||
t.Errorf("message = %q, want it to name LLM_KCN", apiErr.Message)
|
||||
}
|
||||
if strings.Contains(apiErr.Message, "KIMI_API_KEY") {
|
||||
t.Errorf("message = %q, must not name KIMI_API_KEY for a DSN provider", apiErr.Message)
|
||||
}
|
||||
if rt.req != nil {
|
||||
t.Error("network was hit despite missing token")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,240 @@
|
||||
package majordomo
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
|
||||
)
|
||||
|
||||
// Shared fixtures and the shared contract for the built-ins that are "the
|
||||
// openai client pointed somewhere else" (kimi, qwen, ...). They live here
|
||||
// rather than in any one provider's test file so a new OpenAI-compat built-in
|
||||
// has nothing to copy — the same reason registerOpenAICompatBuiltin exists on
|
||||
// the production side.
|
||||
|
||||
// chatCompletionOK is a minimal valid Chat Completions body, so Generate
|
||||
// returns a non-empty response (an empty one would trigger failover, not a
|
||||
// clean pass).
|
||||
const chatCompletionOK = `{"id":"c1","object":"chat.completion","choices":[` +
|
||||
`{"index":0,"message":{"role":"assistant","content":"ok"},"finish_reason":"stop"}]}`
|
||||
|
||||
// captureRT records the last request (and the bytes of its body) and returns a
|
||||
// canned response without touching the network, so these tests stay hermetic
|
||||
// while still exercising the real openai client the built-ins reuse: base URL,
|
||||
// auth header, and the JSON actually put on the wire.
|
||||
type captureRT struct {
|
||||
req *http.Request
|
||||
reqBody []byte
|
||||
body string
|
||||
}
|
||||
|
||||
func (c *captureRT) RoundTrip(r *http.Request) (*http.Response, error) {
|
||||
c.req = r
|
||||
// Drain and close the request body: a RoundTripper owns it, and those
|
||||
// bytes are what wire-shape assertions read.
|
||||
c.reqBody = nil
|
||||
if r.Body != nil {
|
||||
c.reqBody, _ = io.ReadAll(r.Body)
|
||||
_ = r.Body.Close()
|
||||
}
|
||||
return &http.Response{
|
||||
StatusCode: http.StatusOK,
|
||||
Body: io.NopCloser(strings.NewReader(c.body)),
|
||||
Header: make(http.Header),
|
||||
Request: r,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// singleKeyEnv builds a WithEnvLookup function that knows exactly one variable
|
||||
// and returns "" for everything else. The empty default has teeth: a built-in
|
||||
// that reached for any other variable name gets nothing, so the request 401s
|
||||
// and the test fails rather than quietly authenticating off the wrong key.
|
||||
func singleKeyEnv(key, value string) func(string) string {
|
||||
return func(k string) string {
|
||||
if k == key {
|
||||
return value
|
||||
}
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
// openAICompatBuiltin describes one built-in for the shared contract below.
|
||||
// Adding an OpenAI-compat built-in means adding a row here — not copying a
|
||||
// test file, which is how kimi's and qwen's suites became near-identical.
|
||||
type openAICompatBuiltin struct {
|
||||
name string // registry name and spec prefix
|
||||
keyEnv string // the credential variable this built-in reads
|
||||
model string // a current model id for that endpoint
|
||||
wantURL string // chat-completions URL the default endpoint must produce
|
||||
|
||||
// The name:// DSN case: an alternate host (regional/China endpoint)
|
||||
// reached through an LLM_<dsnVar> definition.
|
||||
dsnVar string
|
||||
dsnHost string
|
||||
wantDSNURL string
|
||||
}
|
||||
|
||||
var openAICompatBuiltins = []openAICompatBuiltin{
|
||||
{
|
||||
name: ProviderKimi,
|
||||
keyEnv: "KIMI_API_KEY",
|
||||
model: "kimi-k2-0711-preview",
|
||||
wantURL: "https://api.moonshot.ai/v1/chat/completions",
|
||||
dsnVar: "LLM_KCN",
|
||||
dsnHost: "api.moonshot.cn/v1",
|
||||
wantDSNURL: "https://api.moonshot.cn/v1/chat/completions",
|
||||
},
|
||||
{
|
||||
name: ProviderQwen,
|
||||
keyEnv: "QWEN_API_KEY",
|
||||
model: "qwen3.8-max",
|
||||
wantURL: "https://dashscope-intl.aliyuncs.com/compatible-mode/v1/chat/completions",
|
||||
dsnVar: "LLM_QCN",
|
||||
dsnHost: "dashscope.aliyuncs.com/compatible-mode/v1",
|
||||
wantDSNURL: "https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions",
|
||||
},
|
||||
}
|
||||
|
||||
// TestOpenAICompatBuiltins is the whole contract an OpenAI-compat built-in
|
||||
// owes, asserted identically for every one of them: it resolves in Parse and
|
||||
// targets its own endpoint with its own key; a missing key fails closed naming
|
||||
// the right variable and never reaching the network; its name:// DSN reaches
|
||||
// any other host on the DSN token; and a keyless DSN names the LLM_<NAME> that
|
||||
// actually fixes it rather than the built-in's variable, which does nothing
|
||||
// for a DSN-defined provider.
|
||||
func TestOpenAICompatBuiltins(t *testing.T) {
|
||||
for _, tc := range openAICompatBuiltins {
|
||||
t.Run(tc.name+"/builtin", func(t *testing.T) {
|
||||
rt := &captureRT{body: chatCompletionOK}
|
||||
secret := tc.name + "-secret"
|
||||
r := newTestRegistry(t,
|
||||
WithEnvLookup(singleKeyEnv(tc.keyEnv, secret)),
|
||||
WithHTTPClient(&http.Client{Transport: rt}),
|
||||
)
|
||||
|
||||
if p, ok := r.Provider(tc.name); !ok {
|
||||
t.Fatalf("built-in %q not registered", tc.name)
|
||||
} else if p.Name() != tc.name {
|
||||
t.Errorf("name = %q, want %q", p.Name(), tc.name)
|
||||
}
|
||||
|
||||
spec := tc.name + "/" + tc.model
|
||||
m, err := r.Parse(spec)
|
||||
if err != nil {
|
||||
t.Fatalf("Parse(%q): %v", spec, err)
|
||||
}
|
||||
if got := targetsOf(t, m); len(got) != 1 || got[0] != spec {
|
||||
t.Fatalf("targets = %v, want [%q]", got, spec)
|
||||
}
|
||||
|
||||
if _, err := m.Generate(context.Background(), llm.Request{Messages: []llm.Message{llm.UserText("hi")}}); err != nil {
|
||||
t.Fatalf("Generate: %v", err)
|
||||
}
|
||||
if rt.req == nil {
|
||||
t.Fatal("no request captured")
|
||||
}
|
||||
if rt.req.URL.String() != tc.wantURL {
|
||||
t.Errorf("URL = %q, want %q", rt.req.URL.String(), tc.wantURL)
|
||||
}
|
||||
if want := "Bearer " + secret; rt.req.Header.Get("Authorization") != want {
|
||||
t.Errorf("Authorization = %q, want %q", rt.req.Header.Get("Authorization"), want)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run(tc.name+"/builtin missing key", func(t *testing.T) {
|
||||
rt := &captureRT{body: chatCompletionOK}
|
||||
r := newTestRegistry(t, WithHTTPClient(&http.Client{Transport: rt}))
|
||||
|
||||
m, err := r.Parse(tc.name + "/" + tc.model)
|
||||
if err != nil {
|
||||
t.Fatalf("Parse: %v", err)
|
||||
}
|
||||
_, err = m.Generate(context.Background(), llm.Request{Messages: []llm.Message{llm.UserText("hi")}})
|
||||
apiErr, ok := errors.AsType[*llm.APIError](err)
|
||||
if !ok {
|
||||
t.Fatalf("err = %v (%T), want *llm.APIError", err, err)
|
||||
}
|
||||
if apiErr.Status != http.StatusUnauthorized || apiErr.Code != "missing_api_key" {
|
||||
t.Errorf("Status/Code = %d/%q, want 401/missing_api_key", apiErr.Status, apiErr.Code)
|
||||
}
|
||||
if !strings.Contains(apiErr.Message, tc.keyEnv) {
|
||||
t.Errorf("message = %q, want it to name %s", apiErr.Message, tc.keyEnv)
|
||||
}
|
||||
// The load-bearing half: openai.New defaults its key to
|
||||
// OPENAI_API_KEY, so a built-in that stopped passing WithAPIKey
|
||||
// unconditionally would authenticate as OpenAI instead of failing.
|
||||
if strings.Contains(apiErr.Message, "OPENAI_API_KEY") {
|
||||
t.Errorf("message = %q, must not name OPENAI_API_KEY", apiErr.Message)
|
||||
}
|
||||
if rt.req != nil {
|
||||
t.Error("network was hit despite missing key")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run(tc.name+"/dsn scheme", func(t *testing.T) {
|
||||
rt := &captureRT{body: chatCompletionOK}
|
||||
r := newTestRegistry(t, WithHTTPClient(&http.Client{Transport: rt}))
|
||||
if err := r.LoadEnv(map[string]string{
|
||||
tc.dsnVar: tc.name + "://tok@" + tc.dsnHost,
|
||||
}); err != nil {
|
||||
t.Fatalf("LoadEnv: %v", err)
|
||||
}
|
||||
|
||||
dsnName := strings.ToLower(strings.TrimPrefix(tc.dsnVar, "LLM_"))
|
||||
m, err := r.Parse(dsnName + "/" + tc.model)
|
||||
if err != nil {
|
||||
t.Fatalf("Parse: %v", err)
|
||||
}
|
||||
if _, err := m.Generate(context.Background(), llm.Request{Messages: []llm.Message{llm.UserText("hi")}}); err != nil {
|
||||
t.Fatalf("Generate: %v", err)
|
||||
}
|
||||
if rt.req == nil {
|
||||
t.Fatal("no request captured")
|
||||
}
|
||||
if rt.req.URL.String() != tc.wantDSNURL {
|
||||
t.Errorf("URL = %q, want %q", rt.req.URL.String(), tc.wantDSNURL)
|
||||
}
|
||||
if want := "Bearer tok"; rt.req.Header.Get("Authorization") != want {
|
||||
t.Errorf("Authorization = %q, want %q", rt.req.Header.Get("Authorization"), want)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run(tc.name+"/dsn scheme missing token", func(t *testing.T) {
|
||||
rt := &captureRT{body: chatCompletionOK}
|
||||
r := newTestRegistry(t, WithHTTPClient(&http.Client{Transport: rt}))
|
||||
if err := r.LoadEnv(map[string]string{
|
||||
tc.dsnVar: tc.name + "://" + tc.dsnHost, // no token
|
||||
}); err != nil {
|
||||
t.Fatalf("LoadEnv: %v", err)
|
||||
}
|
||||
|
||||
dsnName := strings.ToLower(strings.TrimPrefix(tc.dsnVar, "LLM_"))
|
||||
m, err := r.Parse(dsnName + "/" + tc.model)
|
||||
if err != nil {
|
||||
t.Fatalf("Parse: %v", err)
|
||||
}
|
||||
_, err = m.Generate(context.Background(), llm.Request{Messages: []llm.Message{llm.UserText("hi")}})
|
||||
apiErr, ok := errors.AsType[*llm.APIError](err)
|
||||
if !ok {
|
||||
t.Fatalf("err = %v (%T), want *llm.APIError", err, err)
|
||||
}
|
||||
// A keyless DSN is fixed by adding a token to that DSN, so the
|
||||
// hint must name the defining variable — never the built-in's own
|
||||
// key, which does nothing for a DSN-defined provider.
|
||||
if !strings.Contains(apiErr.Message, tc.dsnVar) {
|
||||
t.Errorf("message = %q, want it to name %s", apiErr.Message, tc.dsnVar)
|
||||
}
|
||||
if strings.Contains(apiErr.Message, tc.keyEnv) {
|
||||
t.Errorf("message = %q, must not name %s for a DSN provider", apiErr.Message, tc.keyEnv)
|
||||
}
|
||||
if rt.req != nil {
|
||||
t.Error("network was hit despite missing token")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
package majordomo
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"testing"
|
||||
|
||||
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
|
||||
)
|
||||
|
||||
// The contract qwen shares with every other OpenAI-compat built-in (endpoint,
|
||||
// credential isolation, its qwen:// DSN) is asserted by the table in
|
||||
// builtin_openaicompat_test.go. What remains here is qwen-specific: the
|
||||
// reverse-leak direction, and the wire claim ADR-0027 turns on.
|
||||
|
||||
// TestQwenBuiltinKeyDoesNotLeakToOpenAI: QWEN_API_KEY is the qwen built-in's
|
||||
// credential and nothing else's. Why this direction too: the shared table's
|
||||
// missing-key case only proves qwen never borrows OPENAI_API_KEY; this proves
|
||||
// the reverse — a registry that can see QWEN_API_KEY must not hand it to the
|
||||
// openai built-in, which would send an Alibaba key to api.openai.com.
|
||||
func TestQwenBuiltinKeyDoesNotLeakToOpenAI(t *testing.T) {
|
||||
// Set before newTestRegistry: the openai built-in reads OPENAI_API_KEY at
|
||||
// construction. Giving it a real key is what keeps this test honest — a
|
||||
// keyless openai target would 401 before any request, and the assertion
|
||||
// below would pass without a single byte reaching the wire.
|
||||
t.Setenv("OPENAI_API_KEY", "openai-secret")
|
||||
|
||||
rt := &captureRT{body: chatCompletionOK}
|
||||
r := newTestRegistry(t,
|
||||
WithEnvLookup(singleKeyEnv("QWEN_API_KEY", "qwen-secret")),
|
||||
WithHTTPClient(&http.Client{Transport: rt}),
|
||||
)
|
||||
|
||||
m, err := r.Parse("openai/gpt-4o-mini")
|
||||
if err != nil {
|
||||
t.Fatalf("Parse: %v", err)
|
||||
}
|
||||
if _, err := m.Generate(context.Background(), llm.Request{Messages: []llm.Message{llm.UserText("hi")}}); err != nil {
|
||||
t.Fatalf("Generate: %v", err)
|
||||
}
|
||||
if rt.req == nil {
|
||||
t.Fatal("no request captured")
|
||||
}
|
||||
if want := "Bearer openai-secret"; rt.req.Header.Get("Authorization") != want {
|
||||
t.Errorf("Authorization = %q, want %q — the qwen credential must not reach the openai built-in",
|
||||
rt.req.Header.Get("Authorization"), want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestQwenReasoningEffortReachesWire is the load-bearing test for ADR-0027's
|
||||
// central claim: Model Studio's OpenAI-compatible surface takes reasoning as a
|
||||
// top-level "reasoning_effort" body field, which the openai client already
|
||||
// sends — so llm.WithReasoningEffort survives the trip on qwen with no
|
||||
// qwen-specific code. Routing qwen through the anthropic client instead would
|
||||
// drop it silently (provider/anthropic ignores ReasoningEffort by design), and
|
||||
// that difference would be invisible without asserting on the wire body.
|
||||
func TestQwenReasoningEffortReachesWire(t *testing.T) {
|
||||
rt := &captureRT{body: chatCompletionOK}
|
||||
r := newTestRegistry(t,
|
||||
WithEnvLookup(singleKeyEnv("QWEN_API_KEY", "qwen-secret")),
|
||||
WithHTTPClient(&http.Client{Transport: rt}),
|
||||
)
|
||||
|
||||
m, err := r.Parse("qwen/qwen3.8-max")
|
||||
if err != nil {
|
||||
t.Fatalf("Parse: %v", err)
|
||||
}
|
||||
_, err = m.Generate(context.Background(), llm.Request{
|
||||
Messages: []llm.Message{llm.UserText("hi")},
|
||||
ReasoningEffort: "high",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Generate: %v", err)
|
||||
}
|
||||
if rt.reqBody == nil {
|
||||
t.Fatal("no request body captured")
|
||||
}
|
||||
var sent map[string]any
|
||||
if err := json.Unmarshal(rt.reqBody, &sent); err != nil {
|
||||
t.Fatalf("decode request body: %v", err)
|
||||
}
|
||||
if got := sent["reasoning_effort"]; got != "high" {
|
||||
t.Errorf("reasoning_effort = %v, want %q (body: %s)", got, "high", rt.reqBody)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
# ADR-0027: Qwen (Alibaba) built-in provider — OpenAI-compat, not Anthropic-compat
|
||||
|
||||
**Status:** Accepted — 2026-08-12
|
||||
|
||||
## Context
|
||||
|
||||
Alibaba's Qwen models (`qwen3.8-max`, `qwen3.7-plus`, the `qwen3-vl-*` vision
|
||||
variants, …) are served from Model Studio / DashScope, and mort wants them as a
|
||||
first-class failover tier with a dedicated `QWEN_API_KEY` — the same ergonomics
|
||||
ADR-0026 gave Kimi.
|
||||
|
||||
Unlike Kimi, Model Studio exposes the same models over **two** protocols:
|
||||
|
||||
| | OpenAI-compatible | Anthropic-compatible |
|
||||
|---|---|---|
|
||||
| Base URL | `https://dashscope-intl.aliyuncs.com/compatible-mode/v1` | `https://dashscope-intl.aliyuncs.com/apps/anthropic` |
|
||||
| Endpoints | full Chat Completions surface | `/v1/messages` only (no `/v1/models`) |
|
||||
| Purpose | the documented developer API | a shim, documented around hosting Claude Code |
|
||||
|
||||
So the question this ADR answers is not "which client do we reuse" but
|
||||
"which of Alibaba's two wire protocols does the built-in speak".
|
||||
|
||||
## Decision
|
||||
|
||||
**The `qwen` built-in and the `qwen://` DSN scheme speak OpenAI-compat**, over
|
||||
`provider/openai` — no new package, mirroring ADR-0026 (kimi) and ADR-0015
|
||||
(llama-swap chat). Default base URL is the international host; the China host
|
||||
(`dashscope.aliyuncs.com/compatible-mode/v1`) and workspace-scoped regional
|
||||
hosts are reachable with a `qwen://` DSN.
|
||||
|
||||
Credential handling is copied from kimi verbatim, because both of its rules
|
||||
are load-bearing: `WithAPIKey` is passed unconditionally (even empty) so an
|
||||
unset `QWEN_API_KEY` can never fall through to `openai.New`'s `OPENAI_API_KEY`
|
||||
default, and `WithAPIKeyName("QWEN_API_KEY")` makes the synthetic-401 hint name
|
||||
the variable the operator actually has to set.
|
||||
|
||||
The kimi and qwen DSN factories were identical, so they now share one
|
||||
`openaiCompatScheme` helper — the next OpenAI-compat built-in gets the
|
||||
credential and key-hint rules by construction rather than by copy.
|
||||
|
||||
### Why not the Anthropic-compatible endpoint
|
||||
|
||||
Every concrete difference favors OpenAI-compat *for this codebase*:
|
||||
|
||||
- **Reasoning survives the trip.** Model Studio takes `reasoning_effort` as a
|
||||
top-level field on the OpenAI surface, which `provider/openai` already sends
|
||||
— `llm.WithReasoningEffort` works on qwen with zero qwen-specific code
|
||||
(`TestQwenReasoningEffortReachesWire` asserts it on the wire). Down the
|
||||
anthropic client it would be dropped in silence: `provider/anthropic`
|
||||
deliberately ignores `Request.ReasoningEffort`, because first-party Claude
|
||||
has no such knob.
|
||||
- **Structured output would regress.** `provider/anthropic` implements
|
||||
`Request.Schema` with the first-party GA `output_config.format` mechanism.
|
||||
Alibaba's shim does not implement it; a compat endpoint that ignores an
|
||||
unknown field returns unconstrained prose while still reporting success.
|
||||
The OpenAI path sends `response_format: json_schema`, which Model Studio
|
||||
supports natively on the Max/Plus families.
|
||||
- **Cache accounting already lands.** Model Studio's implicit prefix cache
|
||||
reports hits in `usage.prompt_tokens_details.cached_tokens`, which the openai
|
||||
client already maps to `llm.Usage.CacheReadTokens`. The anthropic client
|
||||
reads `cache_read_input_tokens`, a field the shim has no reason to emit.
|
||||
- **Thinking content is discarded on the anthropic path anyway.**
|
||||
`provider/anthropic` skips `thinking` blocks in both the buffered and
|
||||
streaming decoders, so the shim's headline feature — first-class
|
||||
`thinking: {type: "enabled", budget_tokens: N}` — buys majordomo nothing
|
||||
today.
|
||||
- **Smaller blast radius.** The anthropic client has no `WithAPIKeyName`
|
||||
option, so a keyless qwen would tell the operator to set `ANTHROPIC_API_KEY`;
|
||||
fixing that means changing the first-party Anthropic client to serve a
|
||||
third-party shim.
|
||||
- **It is the less-exercised surface.** The Anthropic endpoint is documented as
|
||||
Messages-only, with a temperature range that differs from Anthropic's own
|
||||
([0, 2) vs [0.0, 1.0]) — i.e. it is Qwen semantics wearing an Anthropic
|
||||
envelope, not an Anthropic-equivalent target.
|
||||
|
||||
The one thing the Anthropic surface offers that OpenAI-compat does not is
|
||||
explicit `cache_control` breakpoints reached through `Request.PromptCache`.
|
||||
That is not a reason to route Qwen through it: Model Studio's implicit cache is
|
||||
automatic and already metered, and if explicit breakpoints ever matter they
|
||||
belong in `provider/openai` (Model Studio accepts `cache_control` on content
|
||||
blocks there too), where every OpenAI-compat target would get them.
|
||||
|
||||
## Consequences
|
||||
|
||||
- `qwen/<model>` is first-class in Parse, chains, aliases, and health/failover
|
||||
with no consumer wiring; model ids pass through verbatim (no catalog).
|
||||
- Chat, streaming, tools, structured output, reasoning effort, and cached-token
|
||||
accounting all ride the openai client and inherit its fixes.
|
||||
- Image *inputs* work at the client level, but only the `qwen-vl-*` /
|
||||
`qwen3-vl-*` models accept them (matrix footnote ⁴; ³ is kimi's).
|
||||
- Two model-side quirks are Alibaba's, not majordomo's, and are left to the
|
||||
caller rather than papered over: thinking is **on by default** on some models
|
||||
(e.g. `qwen3.7-plus`), and Qwen3 *open-source* models require streaming when
|
||||
thinking is enabled — a buffered `Generate` against one of those needs a
|
||||
model that supports non-streaming thinking (the Max/Plus families do).
|
||||
- If a future consumer genuinely needs the Anthropic surface, it is reachable
|
||||
today without library changes:
|
||||
`LLM_QWEN_ANTHROPIC=anthropic://[email protected]/apps/anthropic`
|
||||
— with the reasoning/structured-output caveats above.
|
||||
- Second third-party built-in after kimi. The ADR-0026 bar still holds: a named
|
||||
consumer needs it in-config. `RegisterProvider`/`LLM_*` remain the path for
|
||||
everything else.
|
||||
@@ -30,3 +30,4 @@ One decision per file, append-only; supersede rather than rewrite.
|
||||
| [0024](0024-audio-wave3-surfaces.md) | Wave-3 audio surfaces (stems, SFX, speech enhance, voice clone, translate) | Accepted |
|
||||
| [0025](0025-videogen-wave3-surfaces.md) | Wave-3 video surfaces (lipsync, video matte, video upscale, chain jobs) | Accepted |
|
||||
| [0026](0026-kimi-builtin.md) | Kimi (Moonshot AI) built-in provider — reuse openai client, KIMI_API_KEY | Accepted |
|
||||
| [0027](0027-qwen-builtin.md) | Qwen (Alibaba) built-in provider — OpenAI-compat, not Model Studio's Anthropic-compat endpoint | Accepted |
|
||||
|
||||
@@ -26,8 +26,9 @@ var ErrUnknownProvider = errors.New("unknown provider")
|
||||
// authenticated with the bearer token "test-token".
|
||||
type DSN struct {
|
||||
// Scheme selects the provider implementation: "foreman", "ollama",
|
||||
// "ollama-cloud", "openai", "kimi", "anthropic", "google"/"gemini", or
|
||||
// any custom scheme registered with RegisterScheme.
|
||||
// "ollama-cloud", "openai", "kimi", "qwen", "anthropic",
|
||||
// "google"/"gemini", "llama-swap"/"llama-swaps", or any custom scheme
|
||||
// registered with RegisterScheme.
|
||||
Scheme string
|
||||
// Token is the provider secret (bearer token or API key); empty = none.
|
||||
Token string
|
||||
@@ -40,6 +41,19 @@ type DSN struct {
|
||||
// env-defined providers always speak TLS).
|
||||
func (d DSN) BaseURL() string { return "https://" + d.Host }
|
||||
|
||||
// envKeyForProvider returns the LLM_* variable that defines the provider named
|
||||
// name: "m1" → LLM_M1, "my-prov" → LLM_MY_PROV.
|
||||
//
|
||||
// This is the single definition on purpose. Two call sites need byte-identical
|
||||
// output and would drift apart in silence: lazy resolution reads this variable
|
||||
// to find an unregistered provider, and openaiCompatScheme names it in the
|
||||
// missing-key hint so a keyless DSN target tells the operator which variable to
|
||||
// set. Those two were separate copies with a comment asserting they matched —
|
||||
// a comment is not enforcement, this function is.
|
||||
func envKeyForProvider(name string) string {
|
||||
return "LLM_" + strings.ToUpper(strings.ReplaceAll(name, "-", "_"))
|
||||
}
|
||||
|
||||
// ParseDSN parses a raw DSN string. The algorithm matches go-llm exactly:
|
||||
// split on "://", then an optional "@" separates the token from the host;
|
||||
// trailing slashes on the host are trimmed.
|
||||
|
||||
+31
-1
@@ -9,9 +9,32 @@ type EditRequest struct {
|
||||
// Prompt is the text description of the desired edit.
|
||||
Prompt string
|
||||
|
||||
// Init is the initial image the edit starts from. Required.
|
||||
// Init is the initial image the edit starts from. Required, EXCEPT when
|
||||
// RefImages is set — see there.
|
||||
Init Image
|
||||
|
||||
// RefImages carries reference images for INSTRUCTION-EDIT models
|
||||
// (FLUX.1 Kontext, Qwen-Image-Edit), which are a different kind of edit
|
||||
// from img2img and reach the model by a different path.
|
||||
//
|
||||
// img2img noises Init and denoises it back under the prompt: the prompt
|
||||
// describes the DESIRED IMAGE, and how much of the original survives is a
|
||||
// function of Strength. An instruction-edit model instead takes the
|
||||
// picture as conditioning and the prompt as an INSTRUCTION about it
|
||||
// ("change the sign to read OPEN"), leaving everything it was not asked
|
||||
// to touch bit-for-bit intact — no mask, no strength, no compositing.
|
||||
//
|
||||
// Sending one of these models an Init instead of a RefImage does not
|
||||
// degrade gracefully, it silently does the wrong thing: measured against
|
||||
// FLUX.1-Kontext on 2026-07-30, "change the blue rectangle to green" via
|
||||
// init_images left the rectangle blue and drifted every other region,
|
||||
// while the same prompt via a reference image turned it green and left
|
||||
// the rest of the frame numerically unchanged.
|
||||
//
|
||||
// When RefImages is non-empty, Init/Mask/Strength are IGNORED: they
|
||||
// describe a pipeline this model does not run.
|
||||
RefImages []Image
|
||||
|
||||
// Mask restricts the edit to a region (inpainting): a single-channel or
|
||||
// RGB image the same size as Init where WHITE pixels are repainted and
|
||||
// BLACK pixels are kept. Empty = whole-image edit. Backends without mask
|
||||
@@ -54,6 +77,13 @@ type EditOption func(*EditRequest)
|
||||
// WithEditMask restricts the edit to a region (white = repaint, black = keep).
|
||||
func WithEditMask(m Image) EditOption { return func(r *EditRequest) { r.Mask = m } }
|
||||
|
||||
// WithEditRefImages supplies reference images for an instruction-edit model
|
||||
// (Kontext / Qwen-Image-Edit). See EditRequest.RefImages — this selects a
|
||||
// different edit path, not a variation on img2img.
|
||||
func WithEditRefImages(imgs ...Image) EditOption {
|
||||
return func(r *EditRequest) { r.RefImages = imgs }
|
||||
}
|
||||
|
||||
// WithEditStrength sets the denoising strength in [0,1].
|
||||
func WithEditStrength(s float64) EditOption { return func(r *EditRequest) { r.Strength = &s } }
|
||||
|
||||
|
||||
@@ -0,0 +1,137 @@
|
||||
package imagegen
|
||||
|
||||
import "context"
|
||||
|
||||
// FaceSwapRequest transfers an identity from Source into Target.
|
||||
//
|
||||
// This is a DIFFERENT OPERATION from Edit, not a better-tuned one. Measured
|
||||
// against the instruction-edit models on 2026-07-31, asking a diffusion model
|
||||
// to put a specific person's face into a photo does not work by any route —
|
||||
// by name, by description, or by supplying the portrait as a reference image.
|
||||
// Face swapping is a dedicated detect/align/blend pipeline; a provider that
|
||||
// cannot do it should not pretend Edit is a substitute.
|
||||
type FaceSwapRequest struct {
|
||||
// Target is the photo to edit — the pose, expression, lighting and
|
||||
// everything outside the face are preserved from it.
|
||||
Target Image
|
||||
|
||||
// Source is a photo of the face to put in. Only the identity travels;
|
||||
// the source's own pose and expression do not.
|
||||
Source Image
|
||||
|
||||
// Index selects WHICH face in Target, in the provider's documented
|
||||
// ordering (llamaswap: left to right by box centre, as reported by
|
||||
// ListFaces). nil = the largest face, which is right for a portrait and
|
||||
// wrong for a group — enumerate first when it matters.
|
||||
Index *int
|
||||
|
||||
// All swaps every detected face and ignores Index.
|
||||
All bool
|
||||
}
|
||||
|
||||
// FaceSwapOption mutates a FaceSwapRequest before it is sent.
|
||||
type FaceSwapOption func(*FaceSwapRequest)
|
||||
|
||||
// WithFaceIndex selects which face in the target to replace.
|
||||
func WithFaceIndex(i int) FaceSwapOption { return func(r *FaceSwapRequest) { r.Index = &i } }
|
||||
|
||||
// WithAllFaces swaps every detected face.
|
||||
func WithAllFaces() FaceSwapOption { return func(r *FaceSwapRequest) { r.All = true } }
|
||||
|
||||
// Apply returns a copy of the request with all options applied.
|
||||
func (r FaceSwapRequest) Apply(opts ...FaceSwapOption) FaceSwapRequest {
|
||||
for _, opt := range opts {
|
||||
opt(&r)
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// DetectedFace is one face located in an image, in PIXEL coordinates.
|
||||
type DetectedFace struct {
|
||||
// Index is the face's position in the provider's stable ordering, and
|
||||
// the value FaceSwapRequest.Index expects.
|
||||
Index int
|
||||
// Box is [x0, y0, x1, y1].
|
||||
Box [4]int
|
||||
// Score is the detector's confidence, 0-1.
|
||||
Score float64
|
||||
|
||||
// Yaw is how far the head is turned from camera, in degrees, or nil when
|
||||
// the provider does not report pose. Exposed on ENUMERATION, not just
|
||||
// after the fact, because it is how a caller picks a face a swap will
|
||||
// actually work on: past roughly ±45° the features carrying identity are
|
||||
// edge-on, and the result reads as a generic person however good the
|
||||
// transfer is. A bounding box cannot show this.
|
||||
Yaw *float64
|
||||
}
|
||||
|
||||
// Size returns the box dimensions. Derived rather than stored: carrying
|
||||
// width/height alongside Box is two sources of truth for one fact, and the
|
||||
// pair can disagree after any transform.
|
||||
func (f DetectedFace) Size() (w, h int) {
|
||||
return f.Box[2] - f.Box[0], f.Box[3] - f.Box[1]
|
||||
}
|
||||
|
||||
// SwappedFace is the MEASURED outcome for one face the provider replaced.
|
||||
//
|
||||
// It exists because "the call returned an image" and "the likeness
|
||||
// transferred" are different claims that look identical from outside, and a
|
||||
// caller that cannot tell them apart will go looking for another way to
|
||||
// check. The one it reaches for — asking a vision model who the result looks
|
||||
// like — is wrong in exactly the cases that matter: a VLM shown a jogger in a
|
||||
// Georgetown cap holding McDonald's cups answers "Bill Clinton" whoever's
|
||||
// face is on him, so it reports failure on a correct swap.
|
||||
type SwappedFace struct {
|
||||
// Index is the face's position in the provider's left-to-right ordering.
|
||||
Index int
|
||||
|
||||
// Width, Height are the replaced face's pixel size in the TARGET.
|
||||
// Meaningful only against ImageWidth/ImageHeight: a 138px face is large
|
||||
// in a 400px picture and nearly invisible in a 2000px one, and it is the
|
||||
// ratio, not the absolute size, that decides whether a person notices.
|
||||
Width, Height int
|
||||
|
||||
// ImageWidth, ImageHeight are the target image's dimensions, repeated on
|
||||
// every entry so a single face is self-describing without the caller
|
||||
// holding onto the rest of the response.
|
||||
ImageWidth, ImageHeight int
|
||||
|
||||
// Yaw is how far the head is turned from camera, in degrees, or nil when
|
||||
// the provider does not report pose. The best single predictor of whether
|
||||
// a swap will READ as the source person: past roughly ±45° the features
|
||||
// carrying identity are edge-on and the result looks like a generic
|
||||
// person rather than a specific one.
|
||||
Yaw *float64
|
||||
|
||||
// IdentitySimilarity is cosine similarity between the source face and the
|
||||
// face actually present in the result, 0-1, or nil when the provider
|
||||
// could not measure it. Above ~0.5 the identity transferred; a LOW value
|
||||
// is the only evidence that a swap genuinely failed.
|
||||
IdentitySimilarity *float64
|
||||
}
|
||||
|
||||
// FractionOfImage is the swapped face's width as a share of the image's, 0-1.
|
||||
// The number that predicts whether a person will SEE the change: the swap
|
||||
// that prompted all this replaced a 138px face in a 1010px-wide photo — 14%,
|
||||
// correct by every measure and invisible at a glance — while the same code on
|
||||
// a 168px face in a 385px picture (44%) is unmistakable. Returns 0 when the
|
||||
// dimensions are unknown.
|
||||
func (f SwappedFace) FractionOfImage() float64 {
|
||||
if f.ImageWidth <= 0 || f.Width <= 0 {
|
||||
return 0
|
||||
}
|
||||
return float64(f.Width) / float64(f.ImageWidth)
|
||||
}
|
||||
|
||||
// FaceSwapper is the optional face-transfer surface. Separate interface so
|
||||
// existing providers keep compiling; callers type-assert.
|
||||
type FaceSwapper interface {
|
||||
// ListFaces enumerates the faces in an image, in the SAME ordering
|
||||
// FaceSwapRequest.Index uses. Exposed because a caller asked to change
|
||||
// "the man on the right" needs a way to name one face and to check its
|
||||
// own choice against pixel boxes.
|
||||
ListFaces(ctx context.Context, img Image) ([]DetectedFace, error)
|
||||
|
||||
// FaceSwap transfers Source's identity into Target.
|
||||
FaceSwap(ctx context.Context, req FaceSwapRequest, opts ...FaceSwapOption) (*Result, error)
|
||||
}
|
||||
@@ -68,6 +68,15 @@ type Result struct {
|
||||
// Images are the generated images, in the order the backend returned them.
|
||||
Images []Image
|
||||
|
||||
// SwappedFaces is the measured outcome of a FaceSwap, one entry per face
|
||||
// replaced. Empty for every other operation, and empty from a provider
|
||||
// that does not measure. TYPED rather than tucked into Raw: a caller has
|
||||
// to act on this — it is the only way to distinguish a swap that
|
||||
// transferred the likeness from one that returned an image and nothing
|
||||
// more — and a value reachable solely by type-asserting an `any` is one
|
||||
// nobody discovers in time to use.
|
||||
SwappedFaces []SwappedFace
|
||||
|
||||
// Raw is the provider-native response object, an escape hatch for
|
||||
// provider-specific fields. May be nil; never required for normal use.
|
||||
Raw any
|
||||
|
||||
+36
@@ -285,3 +285,39 @@ tests flush out.
|
||||
(footnote ³), `.env.example`, ADR-0026 (+ index; also backfilled the missing
|
||||
0024/0025 index rows).
|
||||
- Consumer: mort names Kimi as a failover tier.
|
||||
|
||||
## 2026-08-12 — Qwen (Alibaba) built-in provider (ADR-0027)
|
||||
|
||||
- New built-in `qwen` provider + `qwen://` DSN scheme over Alibaba Model
|
||||
Studio's OpenAI-compatible mode, reusing `provider/openai` (no new client,
|
||||
mirrors kimi/llama-swap). Default base URL
|
||||
`https://dashscope-intl.aliyuncs.com/compatible-mode/v1`; China/regional
|
||||
hosts via `LLM_QCN=qwen://[email protected]/compatible-mode/v1`.
|
||||
- Credential is `QWEN_API_KEY` (via the registry's injected envLookup).
|
||||
`WithAPIKey` passed unconditionally so an unset key cannot fall through to
|
||||
`OPENAI_API_KEY`; `WithAPIKeyName` names `QWEN_API_KEY` in the 401 hint.
|
||||
- **Chose OpenAI-compat over Model Studio's Anthropic-compatible
|
||||
`/apps/anthropic` shim** (ADR-0027): on the anthropic client
|
||||
`ReasoningEffort` is ignored by design, `Request.Schema` rides
|
||||
`output_config.format` (which the shim does not implement), and cached-token
|
||||
accounting reads Anthropic-only usage fields. The shim is still reachable
|
||||
ad hoc via an `anthropic://` DSN.
|
||||
- `registerOpenAICompatBuiltin` installs BOTH halves of an OpenAI-compat
|
||||
built-in (eager provider + `name://` DSN scheme via the shared
|
||||
`openaiCompatScheme`), so the two credential rules — unconditional
|
||||
`WithAPIKey`, and `WithAPIKeyName` naming that same variable — hold by
|
||||
construction. kimi and qwen are one line each.
|
||||
- `envKeyForProvider` is the single definition of the `LLM_<NAME>` form,
|
||||
shared by lazy resolution (`registry.go`) and the DSN missing-key hint. They
|
||||
were separate copies with a comment asserting they matched.
|
||||
- The shared contract is ONE table (`builtin_openaicompat_test.go`), run
|
||||
identically for every OpenAI-compat built-in: endpoint + bearer, missing key
|
||||
fails closed naming its own variable with no network hit, the `name://` DSN
|
||||
reaching another host, and a keyless DSN naming `LLM_<NAME>` rather than the
|
||||
built-in's key. Adding a built-in is a table row that immediately owes all
|
||||
four; `builtin_kimi_test.go` was retired into it. Qwen-only tests: the
|
||||
reverse credential leak, and `reasoning_effort` asserted on the wire body
|
||||
(the ADR's load-bearing claim).
|
||||
- Docs in sync: README built-in table + Qwen paragraph + DSN scheme list +
|
||||
support matrix (footnote ⁴), `.env.example`, ADR-0027 (+ index).
|
||||
- Consumer: mort wants Qwen as a failover tier.
|
||||
|
||||
@@ -338,27 +338,41 @@ func parseVoices(raw []byte) ([]string, error) {
|
||||
// varies. contentType sets the request Content-Type when body is non-nil.
|
||||
// A response larger than maxBytes is an error, never a silent truncation.
|
||||
func (p *Provider) doRaw(ctx context.Context, method, path, model, contentType string, body io.Reader, maxBytes int64) ([]byte, string, error) {
|
||||
if err := p.requireBaseURL(); err != nil {
|
||||
data, hdr, err := p.doRawHeaders(ctx, method, path, model, contentType, body, maxBytes)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
return data, hdr.Get("Content-Type"), nil
|
||||
}
|
||||
|
||||
// doRawHeaders is doRaw with the WHOLE response header rather than just
|
||||
// Content-Type. Only the face swap needs it — the shim reports whether the
|
||||
// likeness actually transferred in X-Swap-Report, and that answer would be
|
||||
// thrown away by a function that keeps one header — so doRaw stays the
|
||||
// signature 25 other call sites use and delegates here. Two bodies would be
|
||||
// two places for the size cap and the status check to drift apart.
|
||||
func (p *Provider) doRawHeaders(ctx context.Context, method, path, model, contentType string, body io.Reader, maxBytes int64) ([]byte, http.Header, error) {
|
||||
if err := p.requireBaseURL(); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
req, err := p.newRequest(ctx, method, path, contentType, body)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
return nil, nil, err
|
||||
}
|
||||
resp, err := p.client.Do(req)
|
||||
if err != nil {
|
||||
return nil, "", fmt.Errorf("llama-swap: do request: %w", err)
|
||||
return nil, nil, fmt.Errorf("llama-swap: do request: %w", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode/100 != 2 {
|
||||
return nil, "", p.apiError(resp, model)
|
||||
return nil, nil, p.apiError(resp, model)
|
||||
}
|
||||
data, err := io.ReadAll(io.LimitReader(resp.Body, maxBytes+1))
|
||||
if err != nil {
|
||||
return nil, "", fmt.Errorf("llama-swap: read response: %w", err)
|
||||
return nil, nil, fmt.Errorf("llama-swap: read response: %w", err)
|
||||
}
|
||||
if int64(len(data)) > maxBytes {
|
||||
return nil, "", fmt.Errorf("llama-swap: response exceeds %d bytes", maxBytes)
|
||||
return nil, nil, fmt.Errorf("llama-swap: response exceeds %d bytes", maxBytes)
|
||||
}
|
||||
return data, resp.Header.Get("Content-Type"), nil
|
||||
return data, resp.Header, nil
|
||||
}
|
||||
|
||||
@@ -152,3 +152,76 @@ func TestImageEditWithoutMaskOmitsField(t *testing.T) {
|
||||
t.Error("mask field sent for unmasked edit; want omitted")
|
||||
}
|
||||
}
|
||||
|
||||
// TestImageEditByReferenceUsesTxt2ImgExtraImages pins the instruction-edit
|
||||
// wire shape. It is a DIFFERENT endpoint and a DIFFERENT field from img2img,
|
||||
// and the difference is not cosmetic: measured against FLUX.1-Kontext on
|
||||
// 2026-07-30, the same prompt sent as init_images left the thing it was told
|
||||
// to change untouched and drifted everything else, while extra_images changed
|
||||
// exactly what was asked and left the rest of the frame numerically
|
||||
// unchanged. Routing a reference edit down the img2img path would look like
|
||||
// a working call and silently produce the wrong picture.
|
||||
func TestImageEditByReferenceUsesTxt2ImgExtraImages(t *testing.T) {
|
||||
var gotPath string
|
||||
var gotBody map[string]any
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
gotPath = r.URL.Path
|
||||
_ = json.NewDecoder(r.Body).Decode(&gotBody)
|
||||
_, _ = w.Write([]byte(`{"images":["` + onePixelPNG + `"]}`))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
|
||||
im, _ := p.ImageModel("imagegen-flux-kontext")
|
||||
ed := im.(imagegen.Editor)
|
||||
|
||||
ref := editInit(t)
|
||||
if _, err := ed.Edit(context.Background(), imagegen.EditRequest{
|
||||
Prompt: "make the sign read OPEN",
|
||||
// Init/Mask/Strength are set and must be IGNORED — they describe a
|
||||
// pipeline this model does not run.
|
||||
Init: ref,
|
||||
Mask: ref,
|
||||
Strength: func() *float64 { s := 0.75; return &s }(),
|
||||
}, imagegen.WithEditRefImages(ref)); err != nil {
|
||||
t.Fatalf("reference edit: %v", err)
|
||||
}
|
||||
|
||||
if gotPath != "/sdapi/v1/txt2img" {
|
||||
t.Errorf("path = %q, want /sdapi/v1/txt2img (there is no init latent to denoise)", gotPath)
|
||||
}
|
||||
extra, ok := gotBody["extra_images"].([]any)
|
||||
if !ok || len(extra) != 1 {
|
||||
t.Fatalf("extra_images = %v, want the one reference image", gotBody["extra_images"])
|
||||
}
|
||||
if _, present := gotBody["init_images"]; present {
|
||||
t.Error("init_images must NOT be sent on the reference path — it re-noises the picture")
|
||||
}
|
||||
if _, present := gotBody["denoising_strength"]; present {
|
||||
t.Error("denoising_strength must NOT be sent on the reference path")
|
||||
}
|
||||
if _, present := gotBody["mask"]; present {
|
||||
t.Error("mask must NOT be sent on the reference path")
|
||||
}
|
||||
}
|
||||
|
||||
// TestImageEditByReferenceRejectsEmptyRefs guards the case that would
|
||||
// otherwise silently become a plain txt2img: a reference edit whose only
|
||||
// reference carries no bytes has nothing to condition on, and rendering the
|
||||
// prompt from scratch is not what the caller asked for.
|
||||
func TestImageEditByReferenceRejectsEmptyRefs(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(`{"images":["` + onePixelPNG + `"]}`))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
|
||||
im, _ := p.ImageModel("imagegen-flux-kontext")
|
||||
ed := im.(imagegen.Editor)
|
||||
|
||||
_, err := ed.Edit(context.Background(), imagegen.EditRequest{Prompt: "anything"},
|
||||
imagegen.WithEditRefImages(imagegen.Image{MIME: "image/png"}))
|
||||
if !errors.Is(err, llm.ErrUnsupported) {
|
||||
t.Fatalf("err = %v, want ErrUnsupported for an all-empty reference set", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,238 @@
|
||||
// faceswap.go implements imagegen.FaceSwapper against the InsightFace shim
|
||||
// (buffalo_l + inswapper_128) reached through llama-swap's /upstream
|
||||
// passthrough (ADR-0024):
|
||||
//
|
||||
// POST /upstream/<id>/v1/faces multipart file -> JSON
|
||||
// POST /upstream/<id>/v1/faceswap multipart target, source -> PNG
|
||||
package llamaswap
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"mime"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"gitea.stevedudenhoeffer.com/steve/majordomo/imagegen"
|
||||
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
|
||||
)
|
||||
|
||||
// maxFaceSwapResponseBytes bounds the returned PNG. Generous: the shim echoes
|
||||
// the target's dimensions, and a 4K photo round-trips as a large lossless PNG.
|
||||
const maxFaceSwapResponseBytes = 64 << 20
|
||||
|
||||
// FaceSwapModel implements the face-transfer surface. The id selects which
|
||||
// upstream llama-swap loads.
|
||||
func (p *Provider) FaceSwapModel(id string) (imagegen.FaceSwapper, error) {
|
||||
if err := p.requireBaseURL(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &faceSwapModel{p: p, id: id}, nil
|
||||
}
|
||||
|
||||
type faceSwapModel struct {
|
||||
p *Provider
|
||||
id string
|
||||
}
|
||||
|
||||
// facesResponse mirrors the shim's /v1/faces body.
|
||||
type facesResponse struct {
|
||||
Count int `json:"count"`
|
||||
Faces []struct {
|
||||
Index int `json:"index"`
|
||||
Box []int `json:"box"`
|
||||
Score float64 `json:"score"`
|
||||
Width int `json:"width"`
|
||||
Height int `json:"height"`
|
||||
Yaw *float64 `json:"yaw"`
|
||||
} `json:"faces"`
|
||||
}
|
||||
|
||||
// ListFaces implements imagegen.FaceSwapper.
|
||||
func (m *faceSwapModel) ListFaces(ctx context.Context, img imagegen.Image) ([]imagegen.DetectedFace, error) {
|
||||
if len(img.Data) == 0 {
|
||||
return nil, fmt.Errorf("%w: face detection requires image bytes", llm.ErrUnsupported)
|
||||
}
|
||||
path, err := upstreamPath(m.id, "/v1/faces")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
body, contentType, err := buildMultipart("build faces form",
|
||||
filePart{field: "file", filename: imageFilename(img.MIME, "image"), data: img.Data}, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
raw, _, err := m.p.doRaw(ctx, http.MethodPost, path, m.id, contentType, body, maxFaceSwapResponseBytes)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var parsed facesResponse
|
||||
if err := json.Unmarshal(raw, &parsed); err != nil {
|
||||
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
|
||||
Message: fmt.Sprintf("faces response is not JSON: %s", truncateForError(raw))}
|
||||
}
|
||||
out := make([]imagegen.DetectedFace, 0, len(parsed.Faces))
|
||||
for _, f := range parsed.Faces {
|
||||
df := imagegen.DetectedFace{Index: f.Index, Score: f.Score, Yaw: f.Yaw}
|
||||
// A short box would silently index out of range below; treat a
|
||||
// malformed entry as a protocol error rather than zero-filling it,
|
||||
// because a wrong box sends the caller at the wrong face.
|
||||
if len(f.Box) != 4 {
|
||||
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
|
||||
Message: fmt.Sprintf("face %d has a %d-element box, want 4", f.Index, len(f.Box))}
|
||||
}
|
||||
copy(df.Box[:], f.Box)
|
||||
out = append(out, df)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// FaceSwap implements imagegen.FaceSwapper. The endpoint always answers PNG.
|
||||
func (m *faceSwapModel) FaceSwap(ctx context.Context, req imagegen.FaceSwapRequest, opts ...imagegen.FaceSwapOption) (*imagegen.Result, error) {
|
||||
req = req.Apply(opts...)
|
||||
if len(req.Target.Data) == 0 {
|
||||
return nil, fmt.Errorf("%w: face swap requires a target image", llm.ErrUnsupported)
|
||||
}
|
||||
if len(req.Source.Data) == 0 {
|
||||
return nil, fmt.Errorf("%w: face swap requires a source image", llm.ErrUnsupported)
|
||||
}
|
||||
// Only when it will actually be sent: under All the index is documented
|
||||
// as ignored, so rejecting a negative one there would fail a request that
|
||||
// is perfectly well formed.
|
||||
if !req.All && req.Index != nil && *req.Index < 0 {
|
||||
return nil, fmt.Errorf("%w: face index must be >= 0, got %d", llm.ErrUnsupported, *req.Index)
|
||||
}
|
||||
path, err := upstreamPath(m.id, "/v1/faceswap")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var fields []formField
|
||||
if req.All {
|
||||
fields = append(fields, formField{key: "all", value: "true", required: true})
|
||||
} else if req.Index != nil {
|
||||
// Only sent when NOT swapping all: the shim ignores index under
|
||||
// all=true, and sending both would imply a precedence the caller
|
||||
// cannot see.
|
||||
fields = append(fields, formField{key: "index", value: strconv.Itoa(*req.Index), required: true})
|
||||
}
|
||||
|
||||
body, contentType, err := buildMultipartFiles("build faceswap form",
|
||||
[]filePart{
|
||||
{field: "target", filename: imageFilename(req.Target.MIME, "target"), data: req.Target.Data},
|
||||
{field: "source", filename: imageFilename(req.Source.MIME, "source"), data: req.Source.Data},
|
||||
}, fields)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
raw, respHdr, err := m.p.doRawHeaders(ctx, http.MethodPost, path, m.id, contentType, body, maxFaceSwapResponseBytes)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
respType := respHdr.Get("Content-Type")
|
||||
if len(raw) == 0 {
|
||||
return nil, &llm.APIError{Provider: m.p.name, Model: m.id, Message: "face swap response contained no image"}
|
||||
}
|
||||
// Validate the BYTES, not the header. sniffImageMIME falls back to
|
||||
// image/png when detection is inconclusive, so trusting it here would
|
||||
// label a JSON error body as a PNG and return it as a successful image —
|
||||
// and a header check alone misses the case where the response carries no
|
||||
// Content-Type at all. The shim answers JSON on a semantic miss (no face
|
||||
// found), which is exactly the body that would sail through.
|
||||
detected := http.DetectContentType(raw)
|
||||
if !strings.HasPrefix(detected, "image/") {
|
||||
return nil, &llm.APIError{Provider: m.p.name, Model: m.id,
|
||||
Message: fmt.Sprintf("face swap response is not an image (sniffed %q, Content-Type %q): %s",
|
||||
detected, respType, truncateForError(raw))}
|
||||
}
|
||||
// Prefer the server's own label when it is an image type (it knows
|
||||
// subtypes the sniffer does not), else what the bytes actually are.
|
||||
mimeType := detected
|
||||
if hdr := mimeFromContentType(respType, "image/"); hdr != "" {
|
||||
mimeType = hdr
|
||||
}
|
||||
return &imagegen.Result{
|
||||
Images: []llm.ImagePart{{MIME: mimeType, Data: raw}},
|
||||
SwappedFaces: parseSwapReport(respHdr.Get("X-Swap-Report")),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// swapReport mirrors the shim's X-Swap-Report header.
|
||||
type swapReport struct {
|
||||
Image []int `json:"image"`
|
||||
Faces []struct {
|
||||
Index int `json:"index"`
|
||||
Size []int `json:"size"`
|
||||
Yaw *float64 `json:"yaw"`
|
||||
IdentitySimilarity *float64 `json:"identity_similarity"`
|
||||
} `json:"faces"`
|
||||
}
|
||||
|
||||
// parseSwapReport decodes the measured outcome. A missing or malformed header
|
||||
// yields nil rather than an error: an older shim does not send it, and a swap
|
||||
// that produced a good image must not fail because the diagnostics alongside
|
||||
// it were unreadable.
|
||||
func parseSwapReport(header string) []imagegen.SwappedFace {
|
||||
header = strings.TrimSpace(header)
|
||||
if header == "" {
|
||||
return nil
|
||||
}
|
||||
var rep swapReport
|
||||
if err := json.Unmarshal([]byte(header), &rep); err != nil {
|
||||
return nil
|
||||
}
|
||||
out := make([]imagegen.SwappedFace, 0, len(rep.Faces))
|
||||
for _, f := range rep.Faces {
|
||||
sf := imagegen.SwappedFace{
|
||||
Index: f.Index,
|
||||
Yaw: f.Yaw,
|
||||
IdentitySimilarity: f.IdentitySimilarity,
|
||||
}
|
||||
if len(f.Size) == 2 {
|
||||
sf.Width, sf.Height = f.Size[0], f.Size[1]
|
||||
}
|
||||
if len(rep.Image) == 2 {
|
||||
sf.ImageWidth, sf.ImageHeight = rep.Image[0], rep.Image[1]
|
||||
}
|
||||
out = append(out, sf)
|
||||
}
|
||||
if len(out) == 0 {
|
||||
return nil
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// imageFilename picks a multipart filename for an image part. The shim reads
|
||||
// bytes, not names, but a plausible extension keeps server-side sniffing and
|
||||
// request logs honest. base distinguishes the parts of a multi-file form
|
||||
// ("target"/"source", "frame"/"frame_last") so a log line says which one was
|
||||
// malformed — and, for the video keyframes, so a backend that stages uploads
|
||||
// by filename cannot have the second overwrite the first.
|
||||
//
|
||||
// Every caller routes through here: two copies of one extension table is how
|
||||
// they drift.
|
||||
func imageFilename(mimeType, base string) string {
|
||||
if base == "" {
|
||||
base = "image"
|
||||
}
|
||||
mt := strings.ToLower(strings.TrimSpace(mimeType))
|
||||
if parsed, _, err := mime.ParseMediaType(mt); err == nil {
|
||||
mt = parsed
|
||||
}
|
||||
switch mt {
|
||||
case "image/jpeg", "image/jpg":
|
||||
return base + ".jpg"
|
||||
case "image/webp":
|
||||
return base + ".webp"
|
||||
case "image/gif":
|
||||
return base + ".gif"
|
||||
case "image/bmp":
|
||||
return base + ".bmp"
|
||||
default:
|
||||
// PNG is the safe default: every caller in this repo either sends PNG
|
||||
// or sends something the decoder identifies by magic bytes anyway.
|
||||
return base + ".png"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,344 @@
|
||||
package llamaswap
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"errors"
|
||||
"io"
|
||||
"mime"
|
||||
"mime/multipart"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.stevedudenhoeffer.com/steve/majordomo/imagegen"
|
||||
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
|
||||
)
|
||||
|
||||
// parseParts pulls the multipart form a handler received.
|
||||
func parseParts(t *testing.T, r *http.Request) (files map[string][]byte, fields map[string]string) {
|
||||
t.Helper()
|
||||
_, params, err := mime.ParseMediaType(r.Header.Get("Content-Type"))
|
||||
if err != nil {
|
||||
t.Fatalf("content type: %v", err)
|
||||
}
|
||||
mr := multipart.NewReader(r.Body, params["boundary"])
|
||||
files, fields = map[string][]byte{}, map[string]string{}
|
||||
for {
|
||||
p, err := mr.NextPart()
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("next part: %v", err)
|
||||
}
|
||||
body, _ := io.ReadAll(p)
|
||||
if p.FileName() != "" {
|
||||
files[p.FormName()] = body
|
||||
} else {
|
||||
fields[p.FormName()] = string(body)
|
||||
}
|
||||
}
|
||||
return files, fields
|
||||
}
|
||||
|
||||
// TestFaceSwapSendsBothFiles pins the two-file wire shape. A face swap is the
|
||||
// first endpoint in this provider taking more than one file, so buildMultipart
|
||||
// grew a sibling; getting the field NAMES wrong would reach the shim as a
|
||||
// missing-argument 422 rather than anything self-explanatory.
|
||||
func TestFaceSwapSendsBothFiles(t *testing.T) {
|
||||
var gotPath string
|
||||
var files map[string][]byte
|
||||
var fields map[string]string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
gotPath = r.URL.Path
|
||||
files, fields = parseParts(t, r)
|
||||
w.Header().Set("Content-Type", "image/png")
|
||||
raw, _ := base64.StdEncoding.DecodeString(onePixelPNG)
|
||||
_, _ = w.Write(raw)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
|
||||
m, err := p.FaceSwapModel("faceswap")
|
||||
if err != nil {
|
||||
t.Fatalf("model: %v", err)
|
||||
}
|
||||
img := editInit(t)
|
||||
res, err := m.FaceSwap(context.Background(), imagegen.FaceSwapRequest{Target: img, Source: img},
|
||||
imagegen.WithFaceIndex(2))
|
||||
if err != nil {
|
||||
t.Fatalf("faceswap: %v", err)
|
||||
}
|
||||
if len(res.Images) != 1 {
|
||||
t.Fatalf("images = %d, want 1", len(res.Images))
|
||||
}
|
||||
if !strings.HasSuffix(gotPath, "/upstream/faceswap/v1/faceswap") {
|
||||
t.Errorf("path = %q", gotPath)
|
||||
}
|
||||
for _, want := range []string{"target", "source"} {
|
||||
if len(files[want]) == 0 {
|
||||
t.Errorf("no %q file part — the shim requires both", want)
|
||||
}
|
||||
}
|
||||
if fields["index"] != "2" {
|
||||
t.Errorf("index = %q, want 2", fields["index"])
|
||||
}
|
||||
if _, ok := fields["all"]; ok {
|
||||
t.Error("all sent alongside index — the shim ignores index under all=true, so sending both implies a precedence the caller cannot see")
|
||||
}
|
||||
}
|
||||
|
||||
// TestFaceSwapAllSuppressesIndex: same reasoning from the other side.
|
||||
func TestFaceSwapAllSuppressesIndex(t *testing.T) {
|
||||
var fields map[string]string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
_, fields = parseParts(t, r)
|
||||
w.Header().Set("Content-Type", "image/png")
|
||||
raw, _ := base64.StdEncoding.DecodeString(onePixelPNG)
|
||||
_, _ = w.Write(raw)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
|
||||
m, _ := p.FaceSwapModel("faceswap")
|
||||
img := editInit(t)
|
||||
if _, err := m.FaceSwap(context.Background(),
|
||||
imagegen.FaceSwapRequest{Target: img, Source: img, Index: new(int), All: true}); err != nil {
|
||||
t.Fatalf("faceswap: %v", err)
|
||||
}
|
||||
if fields["all"] != "true" {
|
||||
t.Errorf("all = %q, want true", fields["all"])
|
||||
}
|
||||
if _, ok := fields["index"]; ok {
|
||||
t.Error("index sent under all=true")
|
||||
}
|
||||
}
|
||||
|
||||
// TestFaceSwapRejectsMissingImages: both are required, and the error should
|
||||
// name which one rather than surfacing a shim 422.
|
||||
func TestFaceSwapRejectsMissingImages(t *testing.T) {
|
||||
p := New(WithBaseURL("http://example.invalid"))
|
||||
m, _ := p.FaceSwapModel("faceswap")
|
||||
img := editInit(t)
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
req imagegen.FaceSwapRequest
|
||||
want string
|
||||
}{
|
||||
{"no target", imagegen.FaceSwapRequest{Source: img}, "target"},
|
||||
{"no source", imagegen.FaceSwapRequest{Target: img}, "source"},
|
||||
} {
|
||||
_, err := m.FaceSwap(context.Background(), tc.req)
|
||||
if err == nil || !strings.Contains(err.Error(), tc.want) {
|
||||
t.Errorf("%s: err = %v, want one naming %q", tc.name, err, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestFaceSwapRejectsNonImageResponse: the shim answers JSON on a semantic
|
||||
// miss (no face found). Returning those bytes as an "image" would hand the
|
||||
// caller a file that is not a picture and call it success.
|
||||
func TestFaceSwapRejectsNonImageResponse(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{"detail":{"error":"no_face_in_source"}}`))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
|
||||
m, _ := p.FaceSwapModel("faceswap")
|
||||
img := editInit(t)
|
||||
_, err := m.FaceSwap(context.Background(), imagegen.FaceSwapRequest{Target: img, Source: img})
|
||||
if err == nil {
|
||||
t.Fatal("a JSON body was accepted as an image")
|
||||
}
|
||||
var apiErr *llm.APIError
|
||||
if !errors.As(err, &apiErr) {
|
||||
t.Errorf("err = %T, want *llm.APIError so callers can classify it", err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "no_face_in_source") {
|
||||
t.Errorf("err = %v, want it to relay the shim's reason", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestListFacesParsesOrdering: the shim's left-to-right index is the contract
|
||||
// callers select against, so it must survive decoding intact.
|
||||
func TestListFacesParsesOrdering(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if !strings.HasSuffix(r.URL.Path, "/upstream/faceswap/v1/faces") {
|
||||
t.Errorf("path = %q", r.URL.Path)
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{"count":2,"faces":[
|
||||
{"index":0,"box":[10,20,30,40],"score":0.9,"width":20,"height":20},
|
||||
{"index":1,"box":[50,20,90,60],"score":0.8,"width":40,"height":40}]}`))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
|
||||
m, _ := p.FaceSwapModel("faceswap")
|
||||
faces, err := m.ListFaces(context.Background(), editInit(t))
|
||||
if err != nil {
|
||||
t.Fatalf("list: %v", err)
|
||||
}
|
||||
if len(faces) != 2 || faces[0].Index != 0 || faces[1].Index != 1 {
|
||||
t.Fatalf("faces = %+v", faces)
|
||||
}
|
||||
if faces[1].Box != [4]int{50, 20, 90, 60} {
|
||||
t.Errorf("box = %v", faces[1].Box)
|
||||
}
|
||||
}
|
||||
|
||||
// TestListFacesRejectsShortBox: a malformed box would send a caller at the
|
||||
// wrong face, which is worse than an error.
|
||||
func TestListFacesRejectsShortBox(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{"count":1,"faces":[{"index":0,"box":[1,2],"score":0.9}]}`))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
|
||||
m, _ := p.FaceSwapModel("faceswap")
|
||||
if _, err := m.ListFaces(context.Background(), editInit(t)); err == nil {
|
||||
t.Fatal("a 2-element box was accepted")
|
||||
}
|
||||
}
|
||||
|
||||
// TestFaceSwapRejectsHeaderlessNonImage is the regression for gadfly's
|
||||
// blocking finding on #23, agreed by both models. sniffImageMIME falls back
|
||||
// to image/png when detection is inconclusive, and the original guard only
|
||||
// looked at Content-Type — so a JSON error body sent WITHOUT a Content-Type
|
||||
// header was labelled a PNG and returned as a successful image. The shim
|
||||
// answers JSON on a semantic miss, which is precisely the body that would
|
||||
// have sailed through.
|
||||
func TestFaceSwapRejectsHeaderlessNonImage(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
// Explicitly no Content-Type — Go only sets one if we write before
|
||||
// deleting it, so clear it to model a bare upstream response.
|
||||
w.Header()["Content-Type"] = nil
|
||||
_, _ = w.Write([]byte(`{"detail":{"error":"no_face_in_target"}}`))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
|
||||
m, _ := p.FaceSwapModel("faceswap")
|
||||
img := editInit(t)
|
||||
_, err := m.FaceSwap(context.Background(), imagegen.FaceSwapRequest{Target: img, Source: img})
|
||||
if err == nil {
|
||||
t.Fatal("a headerless JSON body was accepted and would have been returned as image/png")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "no_face_in_target") {
|
||||
t.Errorf("err = %v, want it to relay what actually came back", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFaceSwapAllowsNegativeIndexUnderAll: index is documented as ignored
|
||||
// when all=true, so validating it there would reject a well-formed request.
|
||||
func TestFaceSwapAllowsNegativeIndexUnderAll(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", "image/png")
|
||||
raw, _ := base64.StdEncoding.DecodeString(onePixelPNG)
|
||||
_, _ = w.Write(raw)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
|
||||
m, _ := p.FaceSwapModel("faceswap")
|
||||
img := editInit(t)
|
||||
neg := -1
|
||||
if _, err := m.FaceSwap(context.Background(),
|
||||
imagegen.FaceSwapRequest{Target: img, Source: img, Index: &neg, All: true}); err != nil {
|
||||
t.Fatalf("negative index rejected under all=true, where it is ignored: %v", err)
|
||||
}
|
||||
// ...but still rejected when it WOULD be sent.
|
||||
if _, err := m.FaceSwap(context.Background(),
|
||||
imagegen.FaceSwapRequest{Target: img, Source: img, Index: &neg}); err == nil {
|
||||
t.Error("negative index accepted when it would actually be sent")
|
||||
}
|
||||
}
|
||||
|
||||
// TestFaceSwapParsesSwapReport: the measured outcome is the whole reason the
|
||||
// header exists — a caller that cannot tell "the likeness transferred" from
|
||||
// "an image came back" goes and asks a vision model, which is wrong in
|
||||
// exactly the cases that matter.
|
||||
func TestFaceSwapParsesSwapReport(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "image/png")
|
||||
w.Header().Set("X-Swap-Report",
|
||||
`{"image":[1010,1200],"faces":[{"index":2,"size":[138,172],"yaw":-82.2,"identity_similarity":0.791}]}`)
|
||||
raw, _ := base64.StdEncoding.DecodeString(onePixelPNG)
|
||||
_, _ = w.Write(raw)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
|
||||
m, err := p.FaceSwapModel("faceswap")
|
||||
if err != nil {
|
||||
t.Fatalf("model: %v", err)
|
||||
}
|
||||
img := editInit(t)
|
||||
res, err := m.FaceSwap(context.Background(), imagegen.FaceSwapRequest{Target: img, Source: img})
|
||||
if err != nil {
|
||||
t.Fatalf("FaceSwap: %v", err)
|
||||
}
|
||||
if len(res.SwappedFaces) != 1 {
|
||||
t.Fatalf("SwappedFaces = %d, want 1 — the measurement was dropped", len(res.SwappedFaces))
|
||||
}
|
||||
f := res.SwappedFaces[0]
|
||||
if f.Index != 2 || f.Width != 138 || f.Height != 172 {
|
||||
t.Errorf("face = %+v, want index 2 at 138x172", f)
|
||||
}
|
||||
if f.Yaw == nil || *f.Yaw != -82.2 {
|
||||
t.Errorf("yaw = %v, want -82.2 — the pose signal is how a caller knows a profile swap will not read", f.Yaw)
|
||||
}
|
||||
if f.IdentitySimilarity == nil || *f.IdentitySimilarity != 0.791 {
|
||||
t.Errorf("identity_similarity = %v, want 0.791", f.IdentitySimilarity)
|
||||
}
|
||||
// 138/1010 — the number that says "correct, and invisible at a glance".
|
||||
if got := f.FractionOfImage(); got < 0.13 || got > 0.14 {
|
||||
t.Errorf("FractionOfImage = %.3f, want ~0.137", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFaceSwapSurvivesMissingReport: an older shim sends no header at all. A
|
||||
// swap that produced a good image must not fail because the diagnostics
|
||||
// beside it were absent or malformed.
|
||||
func TestFaceSwapSurvivesMissingReport(t *testing.T) {
|
||||
for name, hdr := range map[string]string{
|
||||
"absent": "",
|
||||
"garbage": "not json at all",
|
||||
"wrongtype": `{"faces":"nope"}`,
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "image/png")
|
||||
if hdr != "" {
|
||||
w.Header().Set("X-Swap-Report", hdr)
|
||||
}
|
||||
raw, _ := base64.StdEncoding.DecodeString(onePixelPNG)
|
||||
_, _ = w.Write(raw)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
|
||||
m, err := p.FaceSwapModel("faceswap")
|
||||
if err != nil {
|
||||
t.Fatalf("model: %v", err)
|
||||
}
|
||||
img := editInit(t)
|
||||
res, err := m.FaceSwap(context.Background(), imagegen.FaceSwapRequest{Target: img, Source: img})
|
||||
if err != nil {
|
||||
t.Fatalf("a %s report failed the whole swap: %v", name, err)
|
||||
}
|
||||
if len(res.Images) != 1 {
|
||||
t.Fatal("image lost")
|
||||
}
|
||||
if res.SwappedFaces != nil {
|
||||
t.Errorf("SwappedFaces = %+v, want nil for a %s report", res.SwappedFaces, name)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -135,9 +135,30 @@ type img2imgRequest struct {
|
||||
Mask string `json:"mask,omitempty"`
|
||||
}
|
||||
|
||||
// Edit implements imagegen.Editor via POST {base}/sdapi/v1/img2img.
|
||||
// refEditRequest is the wire shape for an INSTRUCTION-EDIT model. sd-server
|
||||
// exposes reference images as `extra_images` on the shared img-gen request
|
||||
// builder (routes_sdapi.cpp lands them in gen_params.ref_images — the same
|
||||
// place the CLI's -r/--ref-image goes), and that field is read on BOTH
|
||||
// /txt2img and /img2img.
|
||||
//
|
||||
// It posts to /txt2img because there is no init latent to denoise: the
|
||||
// reference IS the conditioning, so an init image plus a denoising strength
|
||||
// would only add noise to a pipeline that does not want any. Output
|
||||
// resolution follows the reference image.
|
||||
type refEditRequest struct {
|
||||
txt2imgRequest
|
||||
ExtraImages []string `json:"extra_images"`
|
||||
}
|
||||
|
||||
// Edit implements imagegen.Editor. Two different pipelines live behind it,
|
||||
// selected by the request: RefImages routes to an instruction-edit model via
|
||||
// /sdapi/v1/txt2img + extra_images, everything else is img2img. See
|
||||
// imagegen.EditRequest.RefImages for why they are not interchangeable.
|
||||
func (m *imageModel) Edit(ctx context.Context, req imagegen.EditRequest, opts ...imagegen.EditOption) (*imagegen.Result, error) {
|
||||
req = req.Apply(opts...)
|
||||
if len(req.RefImages) > 0 {
|
||||
return m.editByReference(ctx, req)
|
||||
}
|
||||
if len(req.Init.Data) == 0 {
|
||||
return nil, fmt.Errorf("%w: image edit requires an init image", llm.ErrUnsupported)
|
||||
}
|
||||
@@ -164,6 +185,34 @@ func (m *imageModel) Edit(ctx context.Context, req imagegen.EditRequest, opts ..
|
||||
return decodeImages(m.p.name, m.id, &resp)
|
||||
}
|
||||
|
||||
// editByReference runs the instruction-edit path. Mask and Strength are
|
||||
// deliberately NOT rejected when set: a caller that hands the same
|
||||
// EditRequest to whichever model is configured should get the better result
|
||||
// on a Kontext-class model, not an error, and both fields describe a
|
||||
// pipeline that simply does not exist here.
|
||||
func (m *imageModel) editByReference(ctx context.Context, req imagegen.EditRequest) (*imagegen.Result, error) {
|
||||
base, err := m.sdWire("reference edit", req.Prompt, req.NegativePrompt, req.Sampler, req.Size, req.Seed, req.Steps, req.CFGScale, req.N)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
wire := refEditRequest{txt2imgRequest: base}
|
||||
for _, ref := range req.RefImages {
|
||||
if len(ref.Data) == 0 {
|
||||
continue
|
||||
}
|
||||
wire.ExtraImages = append(wire.ExtraImages, base64.StdEncoding.EncodeToString(ref.Data))
|
||||
}
|
||||
if len(wire.ExtraImages) == 0 {
|
||||
return nil, fmt.Errorf("%w: reference edit requires at least one non-empty reference image", llm.ErrUnsupported)
|
||||
}
|
||||
|
||||
var resp txt2imgResponse
|
||||
if err := m.p.doJSON(ctx, http.MethodPost, "/sdapi/v1/txt2img", m.id, &wire, &resp); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return decodeImages(m.p.name, m.id, &resp)
|
||||
}
|
||||
|
||||
// parseSize splits a "WxH" string into width/height pointers. "" yields
|
||||
// (nil, nil) so the model's own default resolution applies.
|
||||
func parseSize(size string) (*int, *int, error) {
|
||||
|
||||
@@ -56,7 +56,7 @@ func (m *lipsyncModel) Lipsync(ctx context.Context, req videogen.LipsyncRequest,
|
||||
// hand (mirrors videoModel.Generate).
|
||||
var buf bytes.Buffer
|
||||
w := multipart.NewWriter(&buf)
|
||||
fw, err := w.CreateFormFile("image", initImageFilename(req.Image.MIME))
|
||||
fw, err := w.CreateFormFile("image", imageFilename(req.Image.MIME, "frame"))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("llama-swap: build lipsync form: %w", err)
|
||||
}
|
||||
|
||||
@@ -51,8 +51,20 @@ type filePart struct {
|
||||
// writeFormFields). wrap labels errors. Returns the body and its content
|
||||
// type.
|
||||
func buildMultipart(wrap string, file filePart, fields []formField) (*bytes.Buffer, string, error) {
|
||||
return buildMultipartFiles(wrap, []filePart{file}, fields)
|
||||
}
|
||||
|
||||
// buildMultipartFiles is buildMultipart for endpoints taking SEVERAL files
|
||||
// (face swap sends a target and a source). Files are written in the given
|
||||
// order, then the fields. One writer loop serves both so the two cannot drift
|
||||
// in how they escape names or terminate the body.
|
||||
func buildMultipartFiles(wrap string, files []filePart, fields []formField) (*bytes.Buffer, string, error) {
|
||||
if len(files) == 0 {
|
||||
return nil, "", fmt.Errorf("llama-swap: %s: no file parts", wrap)
|
||||
}
|
||||
var buf bytes.Buffer
|
||||
w := multipart.NewWriter(&buf)
|
||||
for _, file := range files {
|
||||
fw, err := w.CreateFormFile(file.field, file.filename)
|
||||
if err != nil {
|
||||
return nil, "", fmt.Errorf("llama-swap: %s: %w", wrap, err)
|
||||
@@ -60,6 +72,7 @@ func buildMultipart(wrap string, file filePart, fields []formField) (*bytes.Buff
|
||||
if _, err := fw.Write(file.data); err != nil {
|
||||
return nil, "", fmt.Errorf("llama-swap: %s: %w", wrap, err)
|
||||
}
|
||||
}
|
||||
if err := writeFormFields(w, wrap, fields); err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
|
||||
+40
-23
@@ -4,7 +4,6 @@ import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"mime"
|
||||
"mime/multipart"
|
||||
"net/http"
|
||||
"strconv"
|
||||
@@ -39,10 +38,13 @@ type videoModel struct {
|
||||
// bound the call with a context deadline.
|
||||
//
|
||||
// Parameter names follow vLLM-Omni's videos API (num_frames, fps,
|
||||
// num_inference_steps, guidance_scale); the conditioning frame is sent as an
|
||||
// `input_reference` file part, following OpenAI's videos API. Upstreams
|
||||
// num_inference_steps, guidance_scale); the leading conditioning frame is sent
|
||||
// as an `input_reference` file part, following OpenAI's videos API, and a
|
||||
// trailing keyframe (Request.LastImage) as `input_reference_last`. Upstreams
|
||||
// ignore fields they don't understand, and optional fields stay off the wire
|
||||
// entirely so the model's own defaults apply.
|
||||
// entirely so the model's own defaults apply — which is also why a backend
|
||||
// without first-last-frame support returns an ordinary clip here rather than
|
||||
// an error.
|
||||
func (m *videoModel) Generate(ctx context.Context, req videogen.Request, opts ...videogen.Option) (*videogen.Result, error) {
|
||||
req = req.Apply(opts...)
|
||||
if strings.TrimSpace(req.Prompt) == "" {
|
||||
@@ -57,6 +59,9 @@ func (m *videoModel) Generate(ctx context.Context, req videogen.Request, opts ..
|
||||
if req.InitImage != nil && len(req.InitImage.Data) == 0 {
|
||||
return nil, fmt.Errorf("%w: video init image has no bytes", llm.ErrUnsupported)
|
||||
}
|
||||
if req.LastImage != nil && len(req.LastImage.Data) == 0 {
|
||||
return nil, fmt.Errorf("%w: video last image has no bytes", llm.ErrUnsupported)
|
||||
}
|
||||
width, height, err := parseSize(req.Size)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: %v", llm.ErrUnsupported, err)
|
||||
@@ -84,12 +89,19 @@ func (m *videoModel) Generate(ctx context.Context, req videogen.Request, opts ..
|
||||
return nil, err
|
||||
}
|
||||
if req.InitImage != nil {
|
||||
fw, err := w.CreateFormFile("input_reference", initImageFilename(req.InitImage.MIME))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("llama-swap: build video form: %w", err)
|
||||
if err := writeImagePart(w, "input_reference", "frame", req.InitImage); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if _, err := fw.Write(req.InitImage.Data); err != nil {
|
||||
return nil, fmt.Errorf("llama-swap: build video form: %w", err)
|
||||
}
|
||||
// The trailing keyframe rides a SEPARATE part rather than a second
|
||||
// `input_reference`: multipart permits repeated names, but the receiving
|
||||
// end would then have to rely on part ORDER to tell first from last, and
|
||||
// an ordering contract that is invisible in the field name is one nobody
|
||||
// can see they have broken. A backend that does not know the name ignores
|
||||
// the part, which is the same degradation as any other unknown field.
|
||||
if req.LastImage != nil {
|
||||
if err := writeImagePart(w, "input_reference_last", "frame_last", req.LastImage); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
@@ -135,22 +147,27 @@ func singleVideoResult(provider, model, verb string, raw []byte, contentType str
|
||||
return &videogen.Result{Video: videogen.Video{Data: raw, MIME: mimeType}}, nil
|
||||
}
|
||||
|
||||
// initImageFilename picks the multipart filename hint for the conditioning
|
||||
// frame from its MIME subtype. The name is provider-chosen (never
|
||||
// caller-supplied), so no sanitization is needed.
|
||||
func initImageFilename(mimeType string) string {
|
||||
mt := strings.ToLower(strings.TrimSpace(mimeType))
|
||||
if parsed, _, err := mime.ParseMediaType(mt); err == nil {
|
||||
mt = parsed
|
||||
// writeImagePart attaches one conditioning frame under the given field name,
|
||||
// with a filename derived from nameStem. Shared by the first- and last-frame
|
||||
// parts so the two cannot drift in how they encode, which is the usual way a
|
||||
// second copy of a block goes wrong.
|
||||
//
|
||||
// The two frames MUST carry DISTINCT filenames, not merely distinct field
|
||||
// names. Backends commonly stage an uploaded frame under a name derived from
|
||||
// the filename — our own ComfyUI shim posts to /upload/image with
|
||||
// overwrite=true — so two parts sharing "frame.png" would have the second
|
||||
// clobber the first, and BOTH keyframe inputs would then resolve to the same
|
||||
// stored image. The clip would render clean, pinned at both ends to the same
|
||||
// frame, with nothing anywhere reporting a problem.
|
||||
func writeImagePart(w *multipart.Writer, field, nameStem string, img *videogen.Image) error {
|
||||
fw, err := w.CreateFormFile(field, imageFilename(img.MIME, nameStem))
|
||||
if err != nil {
|
||||
return fmt.Errorf("llama-swap: build video form: %w", err)
|
||||
}
|
||||
switch mt {
|
||||
case "image/jpeg", "image/jpg":
|
||||
return "frame.jpg"
|
||||
case "image/webp":
|
||||
return "frame.webp"
|
||||
default: // unknown MIME — PNG is the safe hint
|
||||
return "frame.png"
|
||||
if _, err := fw.Write(img.Data); err != nil {
|
||||
return fmt.Errorf("llama-swap: build video form: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// formatInt renders an optional int pointer for a form field; nil = "" (omit).
|
||||
|
||||
@@ -223,3 +223,134 @@ func TestVideoGenerateNonVideoBodyErrors(t *testing.T) {
|
||||
t.Errorf("message = %q, want mention of non-video body", apiErr.Message)
|
||||
}
|
||||
}
|
||||
|
||||
// Both keyframes reach the wire, under DISTINCT field names.
|
||||
//
|
||||
// The distinct-name property is the actual contract with the backend shim: the
|
||||
// two frames could have shared one repeated `input_reference` name, and then
|
||||
// which is first and which is last would depend on multipart part ORDER — an
|
||||
// ordering contract invisible in the payload, that nothing would notice
|
||||
// breaking. Asserting the names is what pins it.
|
||||
func TestVideoGenerateSendsBothKeyframes(t *testing.T) {
|
||||
var gotFirst, gotLast []byte
|
||||
var firstName, lastName string
|
||||
var sawLastPart bool
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if err := r.ParseMultipartForm(32 << 20); err != nil {
|
||||
t.Errorf("parse form: %v", err)
|
||||
return
|
||||
}
|
||||
if f, hdr, err := r.FormFile("input_reference"); err == nil {
|
||||
gotFirst, _ = io.ReadAll(f)
|
||||
firstName = hdr.Filename
|
||||
f.Close()
|
||||
}
|
||||
if f, hdr, err := r.FormFile("input_reference_last"); err == nil {
|
||||
sawLastPart = true
|
||||
gotLast, _ = io.ReadAll(f)
|
||||
lastName = hdr.Filename
|
||||
f.Close()
|
||||
}
|
||||
w.Header().Set("Content-Type", "video/mp4")
|
||||
_, _ = w.Write([]byte("fake-mp4-bytes"))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
|
||||
vm, err := p.VideoModel("videogen-minimax-h3")
|
||||
if err != nil {
|
||||
t.Fatalf("VideoModel: %v", err)
|
||||
}
|
||||
|
||||
first, _ := base64.StdEncoding.DecodeString(onePixelPNG)
|
||||
last := append(append([]byte{}, first...), 0x00) // distinguishable from first
|
||||
|
||||
if _, err := vm.Generate(context.Background(), videogen.Request{
|
||||
Prompt: "a cat surfing",
|
||||
InitImage: &videogen.Image{MIME: "image/png", Data: first},
|
||||
LastImage: &videogen.Image{MIME: "image/png", Data: last},
|
||||
}); err != nil {
|
||||
t.Fatalf("Generate: %v", err)
|
||||
}
|
||||
|
||||
if !sawLastPart {
|
||||
t.Fatal("input_reference_last was not sent — a pinned end frame would be silently dropped")
|
||||
}
|
||||
if string(gotFirst) != string(first) {
|
||||
t.Errorf("input_reference = %d bytes, want %d", len(gotFirst), len(first))
|
||||
}
|
||||
if string(gotLast) != string(last) {
|
||||
t.Errorf("input_reference_last = %d bytes, want %d", len(gotLast), len(last))
|
||||
}
|
||||
// The two must not be the same bytes, or a swap/aliasing bug reads as a pass.
|
||||
if string(gotFirst) == string(gotLast) {
|
||||
t.Error("both parts carry identical bytes — the frames are being aliased")
|
||||
}
|
||||
// DISTINCT FILENAMES, not just distinct field names. Backends stage an
|
||||
// uploaded frame under a name derived from the filename (our ComfyUI shim
|
||||
// posts to /upload/image with overwrite=true), so two parts sharing
|
||||
// "frame.png" would have the second clobber the first and BOTH keyframes
|
||||
// would resolve to the same stored image — a clip pinned at both ends to
|
||||
// the same frame, rendering cleanly with nothing reporting a fault.
|
||||
if firstName == "" || lastName == "" {
|
||||
t.Fatalf("filenames = %q / %q, want both set", firstName, lastName)
|
||||
}
|
||||
if firstName == lastName {
|
||||
t.Errorf("both parts use filename %q — the second upload would clobber the first", firstName)
|
||||
}
|
||||
}
|
||||
|
||||
// LastImage alone (no InitImage) is a legitimate request: pin the destination
|
||||
// and let the model invent the approach. It must not require a first frame.
|
||||
func TestVideoGenerateLastImageAloneIsAllowed(t *testing.T) {
|
||||
var sawFirst, sawLast bool
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if err := r.ParseMultipartForm(32 << 20); err != nil {
|
||||
t.Errorf("parse form: %v", err)
|
||||
return
|
||||
}
|
||||
if f, _, err := r.FormFile("input_reference"); err == nil {
|
||||
sawFirst = true
|
||||
f.Close()
|
||||
}
|
||||
if f, _, err := r.FormFile("input_reference_last"); err == nil {
|
||||
sawLast = true
|
||||
f.Close()
|
||||
}
|
||||
w.Header().Set("Content-Type", "video/mp4")
|
||||
_, _ = w.Write([]byte("fake-mp4-bytes"))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
p := New(WithBaseURL(srv.URL), WithHTTPClient(srv.Client()))
|
||||
vm, _ := p.VideoModel("videogen-minimax-h3")
|
||||
frame, _ := base64.StdEncoding.DecodeString(onePixelPNG)
|
||||
|
||||
if _, err := vm.Generate(context.Background(),
|
||||
videogen.Request{Prompt: "arrive here"},
|
||||
videogen.WithLastImage(videogen.Image{MIME: "image/png", Data: frame}),
|
||||
); err != nil {
|
||||
t.Fatalf("Generate: %v", err)
|
||||
}
|
||||
if sawFirst {
|
||||
t.Error("input_reference sent, want omitted")
|
||||
}
|
||||
if !sawLast {
|
||||
t.Error("input_reference_last omitted, want sent")
|
||||
}
|
||||
}
|
||||
|
||||
// An empty LastImage is rejected before the request is built, matching
|
||||
// InitImage's existing contract — a zero-byte frame reaching the backend is a
|
||||
// confusing upstream error instead of a clear local one.
|
||||
func TestVideoGenerateRejectsEmptyLastImage(t *testing.T) {
|
||||
p := New(WithBaseURL("http://unused"))
|
||||
vm, _ := p.VideoModel("videogen-minimax-h3")
|
||||
_, err := vm.Generate(context.Background(), videogen.Request{
|
||||
Prompt: "x",
|
||||
LastImage: &videogen.Image{MIME: "image/png"},
|
||||
})
|
||||
if !errors.Is(err, llm.ErrUnsupported) {
|
||||
t.Fatalf("err = %v, want llm.ErrUnsupported", err)
|
||||
}
|
||||
}
|
||||
|
||||
+1
-1
@@ -263,7 +263,7 @@ func (r *Registry) providerFor(name string) (llm.Provider, error) {
|
||||
return nil, envErr
|
||||
}
|
||||
|
||||
envKey := "LLM_" + strings.ToUpper(strings.ReplaceAll(name, "-", "_"))
|
||||
envKey := envKeyForProvider(name)
|
||||
envVal := r.envLookup(envKey)
|
||||
if envVal == "" {
|
||||
return nil, fmt.Errorf("%w: %q (checked registry and %s env var)", ErrUnknownProvider, name, envKey)
|
||||
|
||||
@@ -12,6 +12,8 @@
|
||||
// InitImage is a pure text prompt, a non-nil InitImage conditions generation
|
||||
// on that frame. Hybrid models (e.g. Wan 2.2 TI2V) serve both from the same
|
||||
// checkpoint, so unlike imagegen there is no separate Editor-style interface.
|
||||
// LastImage extends the same surface to the other end of the clip, so one
|
||||
// Request covers t2v, i2v, and first-last-frame-to-video without a mode flag.
|
||||
//
|
||||
// The first implementation is provider/llamaswap, which targets the blocking
|
||||
// OpenAI/vLLM-Omni-style POST /v1/videos/sync endpoint: the response body is
|
||||
@@ -51,6 +53,19 @@ type Request struct {
|
||||
// nil = pure text-to-video.
|
||||
InitImage *Image
|
||||
|
||||
// LastImage conditions generation on an ENDING frame. With InitImage it
|
||||
// pins both ends (first-last-frame-to-video); alone it pins only the
|
||||
// destination and lets the backend invent the approach.
|
||||
//
|
||||
// Support is per-model and NOT advertised anywhere in this contract: a
|
||||
// backend that does not understand a trailing keyframe ignores it and
|
||||
// returns an ordinary clip, which is indistinguishable from success.
|
||||
// There is no capability bit to consult, because the contract has no way
|
||||
// to learn one. A caller that needs to know whether the pin actually took
|
||||
// effect must establish that out of band — by configuration it controls,
|
||||
// not by inspecting the result.
|
||||
LastImage *Image
|
||||
|
||||
// Size is the requested resolution, e.g. "1280x704"; "" = backend default.
|
||||
Size string
|
||||
|
||||
@@ -92,6 +107,10 @@ type Option func(*Request)
|
||||
// WithInitImage conditions generation on a starting frame (image-to-video).
|
||||
func WithInitImage(img Image) Option { return func(r *Request) { r.InitImage = &img } }
|
||||
|
||||
// WithLastImage conditions generation on an ending frame. Combined with
|
||||
// WithInitImage this pins both ends of the clip.
|
||||
func WithLastImage(img Image) Option { return func(r *Request) { r.LastImage = &img } }
|
||||
|
||||
// WithSize sets the requested resolution (e.g. "1280x704").
|
||||
func WithSize(size string) Option { return func(r *Request) { r.Size = size } }
|
||||
|
||||
|
||||
Reference in New Issue
Block a user