fix(agent): recover the front-loaded answer over a summary closer
CI / Build & Test (pull_request) Successful in 9m49s
CI / Tidy (pull_request) Successful in 10m28s
Gadfly review (reusable) / review (pull_request) Successful in 11m49s
Adversarial Review (Gadfly) / review (pull_request) Successful in 11m49s

A third degenerate terminal shape from the glm-5.2 cite pattern: the model
front-loads its full answer into the cite-call turn, then closes with a
bookkeeping ack plus a one-line compression ("Citations are logged. Short
version: ..."). mort run b3cb9ee9 delivered 151 chars of a 2,089-char
answer this way — the closer was neither a back-reference (over the 120
cap, no back-ref phrase) nor a citations addendum (no label-colon, no
links), so finalOutput let it stand.

isSummaryCloser keys on the ack sentence alone (the verb must end the
sentence, so prose about citations never matches; a compression marker
without the ack is deliberately out of scope), and the new modeSummary
recovery bar makes the 3x dwarf ratio mandatory at every length: unlike a
back-reference this closer carries real answer content, so it is only
displaced by the clearly-fuller original it compressed.

The citations/back-ref bool becomes a three-way recoveryMode; existing
behavior for both old modes is unchanged.

Co-Authored-By: Claude Fable 5 <[email protected]>
This commit is contained in:
2026-08-05 08:47:40 -04:00
co-authored by Claude Fable 5
parent 0bd14e01b3
commit 127966bb3a
2 changed files with 225 additions and 25 deletions
+102 -24
View File
@@ -14,8 +14,8 @@ import (
// their answer to the final, tool-free turn. But some models — notably several
// open-weight ones — "front-load" their full answer into an earlier turn that
// ALSO calls a tool (e.g. answer text alongside a citation call), then close
// with a degenerate terminal turn that is not itself the answer. Two shapes are
// recovered from the transcript (zero extra model calls):
// with a degenerate terminal turn that is not itself the answer. Three shapes
// are recovered from the transcript (zero extra model calls):
//
// - a trivial back-reference ("(Already answered above.)", "see above", …):
// the real answer sits earlier, so recover it and DISCARD the worthless
@@ -25,25 +25,39 @@ import (
// glm-5.2 "cite" pattern behind mort issue #1418). The citations are real,
// useful content — unlike a back-reference — so recover the prior answer and
// KEEP the citations, appended below it.
// - a bookkeeping closer ("Citations are logged. Short version: …"): the
// model acknowledged the citation round and compressed the answer it had
// already written into a one-liner (mort run b3cb9ee9 — a 2,089-char answer
// shrank to 151 chars at delivery). The compression is strictly poorer than
// the front-loaded answer, so recover the prior turn and DISCARD the
// closer — but only when the prior turn clearly dwarfs it, because unlike a
// back-reference this closer DOES carry answer content (see modeSummary).
//
// A citations addendum is tested first and wins over the back-reference test (a
// short terminal can be both), so its links are never discarded. When the
// terminal text stands on its own it is returned unchanged; when it is
// degenerate but nothing better can be recovered, it is returned as-is (a bare
// sources list still beats nothing).
// A citations addendum is tested first and wins over the other two (a short
// terminal can match more than one shape), so its links are never discarded;
// the summary closer is tested before the plain weak-final test so the
// stricter recovery bar applies when both match. When the terminal text stands
// on its own it is returned unchanged; when it is degenerate but nothing
// better can be recovered, it is returned as-is (a compressed answer still
// beats nothing).
//
// msgs must already include the terminal assistant message as its last element
// (the loop appends it before calling this); terminal is that message's text.
func finalOutput(msgs []llm.Message, terminal string) string {
citations := isCitationsOnly(terminal)
if !citations && !isWeakFinal(terminal) {
mode := modeBackRef
switch {
case isCitationsOnly(terminal):
mode = modeCitations
case isSummaryCloser(terminal):
mode = modeSummary
case !isWeakFinal(terminal):
return terminal
}
rec, ok := lastSubstantiveAssistantText(msgs, terminal, citations)
rec, ok := lastSubstantiveAssistantText(msgs, terminal, mode)
if !ok {
return terminal
}
if citations {
if mode == modeCitations {
// Preserve the citations addendum below the recovered answer, unless the
// recovered turn already carries it (guards against a duplicate sources
// block when the front-loaded turn included its own citations). The
@@ -65,11 +79,47 @@ func stripURLAngles(s string) string {
return strings.NewReplacer("<", "", ">", "").Replace(s)
}
// recoveryMode selects the bar a prior assistant turn must clear to replace
// the terminal turn (see isSubstantiveAnswer) and what finalOutput does with
// the terminal once recovery succeeds.
type recoveryMode int
const (
// modeBackRef: the terminal is empty or a pure back-reference — worthless
// on its own, so any real prior answer replaces it and it is discarded.
modeBackRef recoveryMode = iota
// modeCitations: the terminal is a sources-only addendum — not a rival
// answer, so the dwarf ratio is skipped and the addendum is kept, appended
// below the recovered answer.
modeCitations
// modeSummary: the terminal acknowledges the citation round and may carry
// a short compression of the front-loaded answer. Unlike a back-reference
// it DOES contain answer content, so it is only replaced when a prior turn
// clearly dwarfs it — the ratio is mandatory at every length, and the
// closer is discarded (its content is a strict subset of what it replaced).
modeSummary
)
// backRefRe matches a terminal turn that merely points back to an earlier
// message instead of stating the answer ("(Already answered above.)",
// "see above", "as I said", ...).
var backRefRe = regexp.MustCompile(`(?i)(already answered|see above|as (i )?(said|mentioned|stated|noted)|answered (that )?above|per my (previous|earlier))`)
// summaryCloserRe matches a terminal turn that OPENS with a bookkeeping
// acknowledgment of the citation round — "Citations are logged.", "Sources
// cited.", "Logged the citations." — the shape a model produces when it
// front-loaded its answer into an earlier cite-call turn and closes by
// acknowledging the tool results, often followed by a "Short version: …"
// compression of the answer it already wrote. The ack clause must end at a
// sentence terminator ([.!]) DIRECTLY after the verb: "The citations are
// recorded in the court transcript…" is a real answer about citations, not
// bookkeeping, and must never match. A compression marker without the ack
// ("Short version: no.") is deliberately out of scope — a user who asked for
// brevity would be answered with exactly that shape, and misclassifying it
// would hijack a legitimate answer; an unmatched closer merely keeps today's
// behavior (fail closed).
var summaryCloserRe = regexp.MustCompile(`(?i)^[\s>#*_-]*((done|all set|ok(ay)?)[\s,.!:—-]+)?(((all|the)\s+)?(citations?|sources?|references?|claims?)\s+((are|were|have\s+been|all)\s+)*(logged|recorded|cited|saved|noted|captured|filed)|logged\s+((all|the)\s+)*(citations?|sources?|references?))[.!]`)
// preambleRe matches intent-announcing prefixes ("Let me search...", "I'll
// check...") so a preamble is never mistaken for the answer during recovery.
var preambleRe = regexp.MustCompile(`(?i)^(let me|let'?s|i'?ll|i will|first[, ]|sure[,. ]|okay[,. ]|on it|checking)`)
@@ -114,6 +164,11 @@ const (
// be at most len/N of the whole, so a prose answer that merely opens with
// "Source:" and cites a URL mid-sentence is not mistaken for a bare list.
citationDominatedDivisor = 3
// summaryCloserMaxChars bounds a summary closer: room for the ack sentence
// plus a couple of compression sentences (the b3cb9ee9 closer was 151).
// Beyond this the "short version" is substantial enough that replacing it
// risks losing content the front-loaded turn never had.
summaryCloserMaxChars = 300
)
// isWeakFinal reports whether a terminal turn's text fails to stand on its own
@@ -152,11 +207,26 @@ func isCitationsOnly(s string) bool {
return len(residue) <= len(t)/citationDominatedDivisor
}
// isSummaryCloser reports whether a terminal turn is a bookkeeping closer: it
// opens with a complete "citations are logged"-style ack sentence (see
// summaryCloserRe) and is short enough that whatever follows the ack can only
// be a compression of an earlier, fuller answer. Whether that fuller answer
// actually exists is modeSummary's job — the dwarf ratio in
// isSubstantiveAnswer keeps a matching closer in place when nothing earlier
// clearly outweighs it.
func isSummaryCloser(s string) bool {
t := strings.TrimSpace(s)
if t == "" || len(t) > summaryCloserMaxChars {
return false
}
return summaryCloserRe.MatchString(t)
}
// lastSubstantiveAssistantText scans msgs newest→oldest (skipping the terminal
// turn and empty tool-only turns) for the most recent assistant turn whose text
// reads like a real answer. citations selects the recovery bar (see
// reads like a real answer. mode selects the recovery bar (see
// isSubstantiveAnswer). Returns ("", false) when nothing qualifies.
func lastSubstantiveAssistantText(msgs []llm.Message, terminal string, citations bool) (string, bool) {
func lastSubstantiveAssistantText(msgs []llm.Message, terminal string, mode recoveryMode) (string, bool) {
tt := strings.TrimSpace(terminal)
for i := len(msgs) - 1; i >= 0; i-- {
m := msgs[i]
@@ -167,7 +237,7 @@ func lastSubstantiveAssistantText(msgs []llm.Message, terminal string, citations
if txt == "" || txt == tt {
continue // the terminal turn itself, or an empty tool-only turn
}
if isSubstantiveAnswer(txt, tt, citations) {
if isSubstantiveAnswer(txt, tt, mode) {
return txt, true
}
}
@@ -177,20 +247,28 @@ func lastSubstantiveAssistantText(msgs []llm.Message, terminal string, citations
// isSubstantiveAnswer reports whether txt (a prior assistant turn) reads like a
// real answer rather than a preamble, relative to the terminal text.
//
// A sufficiently long turn (>= recoverMinChars) is accepted unconditionally: a
// multi-hundred-char turn is an answer even when it opens conversationally
// ("Sure, here's…", "Let me explain: …"), so the preamble filter is NOT applied
// to it — applying it there would drop a legitimate long front-loaded answer.
// Only in the borderline band does a turn have to clear a floor, not read like a
// short planning preamble ("Let me look that up…"), and — unless the terminal is
// a citations addendum (not a rival answer, so its length is irrelevant) — also
// clearly dwarf the terminal.
func isSubstantiveAnswer(txt, terminal string, citations bool) bool {
// modeSummary demands the dwarf ratio FIRST, at every length: a summary closer
// carries a real (compressed) answer, so replacing it is only justified when
// the prior turn is clearly the fuller original it was compressed from.
//
// A sufficiently long turn (>= recoverMinChars) is otherwise accepted
// unconditionally: a multi-hundred-char turn is an answer even when it opens
// conversationally ("Sure, here's…", "Let me explain: …"), so the preamble
// filter is NOT applied to it — applying it there would drop a legitimate long
// front-loaded answer. Only in the borderline band does a turn have to clear a
// floor, not read like a short planning preamble ("Let me look that up…"),
// and — for modeBackRef only — also clearly dwarf the terminal (a citations
// addendum is not a rival answer, so its length is irrelevant; a summary
// closer already proved the ratio above).
func isSubstantiveAnswer(txt, terminal string, mode recoveryMode) bool {
if mode == modeSummary && len(txt) < recoverRatio*len(terminal) {
return false
}
if len(txt) >= recoverMinChars {
return true
}
if len(txt) < recoverFloorChars || preambleRe.MatchString(txt) {
return false
}
return citations || len(txt) >= recoverRatio*len(terminal)
return mode != modeBackRef || len(txt) >= recoverRatio*len(terminal)
}