fix(agent): recover the front-loaded answer over a summary closer
A third degenerate terminal shape from the glm-5.2 cite pattern: the model
front-loads its full answer into the cite-call turn, then closes with a
bookkeeping ack plus a one-line compression ("Citations are logged. Short
version: ..."). mort run b3cb9ee9 delivered 151 chars of a 2,089-char
answer this way — the closer was neither a back-reference (over the 120
cap, no back-ref phrase) nor a citations addendum (no label-colon, no
links), so finalOutput let it stand.
isSummaryCloser keys on the ack sentence alone (the verb must end the
sentence, so prose about citations never matches; a compression marker
without the ack is deliberately out of scope), and the new modeSummary
recovery bar makes the 3x dwarf ratio mandatory at every length: unlike a
back-reference this closer carries real answer content, so it is only
displaced by the clearly-fuller original it compressed.
The citations/back-ref bool becomes a three-way recoveryMode; existing
behavior for both old modes is unchanged.
Co-Authored-By: Claude Fable 5 <[email protected]>
This commit is contained in:
+102
-24
@@ -14,8 +14,8 @@ import (
|
||||
// their answer to the final, tool-free turn. But some models — notably several
|
||||
// open-weight ones — "front-load" their full answer into an earlier turn that
|
||||
// ALSO calls a tool (e.g. answer text alongside a citation call), then close
|
||||
// with a degenerate terminal turn that is not itself the answer. Two shapes are
|
||||
// recovered from the transcript (zero extra model calls):
|
||||
// with a degenerate terminal turn that is not itself the answer. Three shapes
|
||||
// are recovered from the transcript (zero extra model calls):
|
||||
//
|
||||
// - a trivial back-reference ("(Already answered above.)", "see above", …):
|
||||
// the real answer sits earlier, so recover it and DISCARD the worthless
|
||||
@@ -25,25 +25,39 @@ import (
|
||||
// glm-5.2 "cite" pattern behind mort issue #1418). The citations are real,
|
||||
// useful content — unlike a back-reference — so recover the prior answer and
|
||||
// KEEP the citations, appended below it.
|
||||
// - a bookkeeping closer ("Citations are logged. Short version: …"): the
|
||||
// model acknowledged the citation round and compressed the answer it had
|
||||
// already written into a one-liner (mort run b3cb9ee9 — a 2,089-char answer
|
||||
// shrank to 151 chars at delivery). The compression is strictly poorer than
|
||||
// the front-loaded answer, so recover the prior turn and DISCARD the
|
||||
// closer — but only when the prior turn clearly dwarfs it, because unlike a
|
||||
// back-reference this closer DOES carry answer content (see modeSummary).
|
||||
//
|
||||
// A citations addendum is tested first and wins over the back-reference test (a
|
||||
// short terminal can be both), so its links are never discarded. When the
|
||||
// terminal text stands on its own it is returned unchanged; when it is
|
||||
// degenerate but nothing better can be recovered, it is returned as-is (a bare
|
||||
// sources list still beats nothing).
|
||||
// A citations addendum is tested first and wins over the other two (a short
|
||||
// terminal can match more than one shape), so its links are never discarded;
|
||||
// the summary closer is tested before the plain weak-final test so the
|
||||
// stricter recovery bar applies when both match. When the terminal text stands
|
||||
// on its own it is returned unchanged; when it is degenerate but nothing
|
||||
// better can be recovered, it is returned as-is (a compressed answer still
|
||||
// beats nothing).
|
||||
//
|
||||
// msgs must already include the terminal assistant message as its last element
|
||||
// (the loop appends it before calling this); terminal is that message's text.
|
||||
func finalOutput(msgs []llm.Message, terminal string) string {
|
||||
citations := isCitationsOnly(terminal)
|
||||
if !citations && !isWeakFinal(terminal) {
|
||||
mode := modeBackRef
|
||||
switch {
|
||||
case isCitationsOnly(terminal):
|
||||
mode = modeCitations
|
||||
case isSummaryCloser(terminal):
|
||||
mode = modeSummary
|
||||
case !isWeakFinal(terminal):
|
||||
return terminal
|
||||
}
|
||||
rec, ok := lastSubstantiveAssistantText(msgs, terminal, citations)
|
||||
rec, ok := lastSubstantiveAssistantText(msgs, terminal, mode)
|
||||
if !ok {
|
||||
return terminal
|
||||
}
|
||||
if citations {
|
||||
if mode == modeCitations {
|
||||
// Preserve the citations addendum below the recovered answer, unless the
|
||||
// recovered turn already carries it (guards against a duplicate sources
|
||||
// block when the front-loaded turn included its own citations). The
|
||||
@@ -65,11 +79,47 @@ func stripURLAngles(s string) string {
|
||||
return strings.NewReplacer("<", "", ">", "").Replace(s)
|
||||
}
|
||||
|
||||
// recoveryMode selects the bar a prior assistant turn must clear to replace
|
||||
// the terminal turn (see isSubstantiveAnswer) and what finalOutput does with
|
||||
// the terminal once recovery succeeds.
|
||||
type recoveryMode int
|
||||
|
||||
const (
|
||||
// modeBackRef: the terminal is empty or a pure back-reference — worthless
|
||||
// on its own, so any real prior answer replaces it and it is discarded.
|
||||
modeBackRef recoveryMode = iota
|
||||
// modeCitations: the terminal is a sources-only addendum — not a rival
|
||||
// answer, so the dwarf ratio is skipped and the addendum is kept, appended
|
||||
// below the recovered answer.
|
||||
modeCitations
|
||||
// modeSummary: the terminal acknowledges the citation round and may carry
|
||||
// a short compression of the front-loaded answer. Unlike a back-reference
|
||||
// it DOES contain answer content, so it is only replaced when a prior turn
|
||||
// clearly dwarfs it — the ratio is mandatory at every length, and the
|
||||
// closer is discarded (its content is a strict subset of what it replaced).
|
||||
modeSummary
|
||||
)
|
||||
|
||||
// backRefRe matches a terminal turn that merely points back to an earlier
|
||||
// message instead of stating the answer ("(Already answered above.)",
|
||||
// "see above", "as I said", ...).
|
||||
var backRefRe = regexp.MustCompile(`(?i)(already answered|see above|as (i )?(said|mentioned|stated|noted)|answered (that )?above|per my (previous|earlier))`)
|
||||
|
||||
// summaryCloserRe matches a terminal turn that OPENS with a bookkeeping
|
||||
// acknowledgment of the citation round — "Citations are logged.", "Sources
|
||||
// cited.", "Logged the citations." — the shape a model produces when it
|
||||
// front-loaded its answer into an earlier cite-call turn and closes by
|
||||
// acknowledging the tool results, often followed by a "Short version: …"
|
||||
// compression of the answer it already wrote. The ack clause must end at a
|
||||
// sentence terminator ([.!]) DIRECTLY after the verb: "The citations are
|
||||
// recorded in the court transcript…" is a real answer about citations, not
|
||||
// bookkeeping, and must never match. A compression marker without the ack
|
||||
// ("Short version: no.") is deliberately out of scope — a user who asked for
|
||||
// brevity would be answered with exactly that shape, and misclassifying it
|
||||
// would hijack a legitimate answer; an unmatched closer merely keeps today's
|
||||
// behavior (fail closed).
|
||||
var summaryCloserRe = regexp.MustCompile(`(?i)^[\s>#*_-]*((done|all set|ok(ay)?)[\s,.!:—-]+)?(((all|the)\s+)?(citations?|sources?|references?|claims?)\s+((are|were|have\s+been|all)\s+)*(logged|recorded|cited|saved|noted|captured|filed)|logged\s+((all|the)\s+)*(citations?|sources?|references?))[.!]`)
|
||||
|
||||
// preambleRe matches intent-announcing prefixes ("Let me search...", "I'll
|
||||
// check...") so a preamble is never mistaken for the answer during recovery.
|
||||
var preambleRe = regexp.MustCompile(`(?i)^(let me|let'?s|i'?ll|i will|first[, ]|sure[,. ]|okay[,. ]|on it|checking)`)
|
||||
@@ -114,6 +164,11 @@ const (
|
||||
// be at most len/N of the whole, so a prose answer that merely opens with
|
||||
// "Source:" and cites a URL mid-sentence is not mistaken for a bare list.
|
||||
citationDominatedDivisor = 3
|
||||
// summaryCloserMaxChars bounds a summary closer: room for the ack sentence
|
||||
// plus a couple of compression sentences (the b3cb9ee9 closer was 151).
|
||||
// Beyond this the "short version" is substantial enough that replacing it
|
||||
// risks losing content the front-loaded turn never had.
|
||||
summaryCloserMaxChars = 300
|
||||
)
|
||||
|
||||
// isWeakFinal reports whether a terminal turn's text fails to stand on its own
|
||||
@@ -152,11 +207,26 @@ func isCitationsOnly(s string) bool {
|
||||
return len(residue) <= len(t)/citationDominatedDivisor
|
||||
}
|
||||
|
||||
// isSummaryCloser reports whether a terminal turn is a bookkeeping closer: it
|
||||
// opens with a complete "citations are logged"-style ack sentence (see
|
||||
// summaryCloserRe) and is short enough that whatever follows the ack can only
|
||||
// be a compression of an earlier, fuller answer. Whether that fuller answer
|
||||
// actually exists is modeSummary's job — the dwarf ratio in
|
||||
// isSubstantiveAnswer keeps a matching closer in place when nothing earlier
|
||||
// clearly outweighs it.
|
||||
func isSummaryCloser(s string) bool {
|
||||
t := strings.TrimSpace(s)
|
||||
if t == "" || len(t) > summaryCloserMaxChars {
|
||||
return false
|
||||
}
|
||||
return summaryCloserRe.MatchString(t)
|
||||
}
|
||||
|
||||
// lastSubstantiveAssistantText scans msgs newest→oldest (skipping the terminal
|
||||
// turn and empty tool-only turns) for the most recent assistant turn whose text
|
||||
// reads like a real answer. citations selects the recovery bar (see
|
||||
// reads like a real answer. mode selects the recovery bar (see
|
||||
// isSubstantiveAnswer). Returns ("", false) when nothing qualifies.
|
||||
func lastSubstantiveAssistantText(msgs []llm.Message, terminal string, citations bool) (string, bool) {
|
||||
func lastSubstantiveAssistantText(msgs []llm.Message, terminal string, mode recoveryMode) (string, bool) {
|
||||
tt := strings.TrimSpace(terminal)
|
||||
for i := len(msgs) - 1; i >= 0; i-- {
|
||||
m := msgs[i]
|
||||
@@ -167,7 +237,7 @@ func lastSubstantiveAssistantText(msgs []llm.Message, terminal string, citations
|
||||
if txt == "" || txt == tt {
|
||||
continue // the terminal turn itself, or an empty tool-only turn
|
||||
}
|
||||
if isSubstantiveAnswer(txt, tt, citations) {
|
||||
if isSubstantiveAnswer(txt, tt, mode) {
|
||||
return txt, true
|
||||
}
|
||||
}
|
||||
@@ -177,20 +247,28 @@ func lastSubstantiveAssistantText(msgs []llm.Message, terminal string, citations
|
||||
// isSubstantiveAnswer reports whether txt (a prior assistant turn) reads like a
|
||||
// real answer rather than a preamble, relative to the terminal text.
|
||||
//
|
||||
// A sufficiently long turn (>= recoverMinChars) is accepted unconditionally: a
|
||||
// multi-hundred-char turn is an answer even when it opens conversationally
|
||||
// ("Sure, here's…", "Let me explain: …"), so the preamble filter is NOT applied
|
||||
// to it — applying it there would drop a legitimate long front-loaded answer.
|
||||
// Only in the borderline band does a turn have to clear a floor, not read like a
|
||||
// short planning preamble ("Let me look that up…"), and — unless the terminal is
|
||||
// a citations addendum (not a rival answer, so its length is irrelevant) — also
|
||||
// clearly dwarf the terminal.
|
||||
func isSubstantiveAnswer(txt, terminal string, citations bool) bool {
|
||||
// modeSummary demands the dwarf ratio FIRST, at every length: a summary closer
|
||||
// carries a real (compressed) answer, so replacing it is only justified when
|
||||
// the prior turn is clearly the fuller original it was compressed from.
|
||||
//
|
||||
// A sufficiently long turn (>= recoverMinChars) is otherwise accepted
|
||||
// unconditionally: a multi-hundred-char turn is an answer even when it opens
|
||||
// conversationally ("Sure, here's…", "Let me explain: …"), so the preamble
|
||||
// filter is NOT applied to it — applying it there would drop a legitimate long
|
||||
// front-loaded answer. Only in the borderline band does a turn have to clear a
|
||||
// floor, not read like a short planning preamble ("Let me look that up…"),
|
||||
// and — for modeBackRef only — also clearly dwarf the terminal (a citations
|
||||
// addendum is not a rival answer, so its length is irrelevant; a summary
|
||||
// closer already proved the ratio above).
|
||||
func isSubstantiveAnswer(txt, terminal string, mode recoveryMode) bool {
|
||||
if mode == modeSummary && len(txt) < recoverRatio*len(terminal) {
|
||||
return false
|
||||
}
|
||||
if len(txt) >= recoverMinChars {
|
||||
return true
|
||||
}
|
||||
if len(txt) < recoverFloorChars || preambleRe.MatchString(txt) {
|
||||
return false
|
||||
}
|
||||
return citations || len(txt) >= recoverRatio*len(terminal)
|
||||
return mode != modeBackRef || len(txt) >= recoverRatio*len(terminal)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user