fix(agent): gadfly round 2 — a pointer is not a compression
Four findings, all real.
**A deictic closer now needs a compression marker too** (opus,
correctness). The pointsAbove branch of isSummaryCloser required only a
back-reference, so a terminal like "Given the analysis above, I
recommend option B because X" — a pointer followed by a CONCLUSION the
earlier turn never contained — would be discarded in favour of that
turn, throwing away the answer. The ack shape does not have this problem
because "Citations are logged." carries nothing; a bare pointer does not
carry that guarantee.
Both halves are now required: the pointer says the full answer is
elsewhere, and compressionMarkerRe ("Short version:", "TL;DR", "In
short") is the model saying what sits beside it is a condensation rather
than new reasoning. #1611's closer has both. A marker without a pointer
stays out of scope for the reason already documented — a user who asked
for brevity gets exactly that shape. Unmatched closers keep today's
behaviour, so the narrowing fails closed.
**CRLF** (opus, error-handling): "above\r\n" was not clause-final, so a
CRLF transcript quietly lost every line-final deictic. CR joins LF in the
terminator set.
**A comment wrapped mid-phrase** so that "// -style" read as a list
marker (sonnet) — reflowed as part of rewriting that doc block.
**Process provenance in a test comment** (sonnet): "(gadfly, 3 models)"
is an execution log, not an invariant. The lesson survives, the
attribution does not.
Break-check is ten mutations now — dropping CR and dropping the
compression requirement are each killed by their own named cases — with
the control surviving.
This commit is contained in:
+37
-7
@@ -136,8 +136,9 @@ var backRefRe = regexp.MustCompile(`(?i)(already answered|see above|as (i )?(sai
|
|||||||
// hyphenated compound — "above-average", "above-board", "above-ground" —
|
// hyphenated compound — "above-average", "above-board", "above-ground" —
|
||||||
// read as a clause-final deictic and put a legitimate short answer at risk of
|
// read as a clause-final deictic and put a legitimate short answer at risk of
|
||||||
// being discarded. The em dash stays: a model writes "…above — see the
|
// being discarded. The em dash stays: a model writes "…above — see the
|
||||||
// links", never "above-" as a separator.
|
// links", never "above-" as a separator. CR is listed alongside LF so a
|
||||||
var aboveRefRe = regexp.MustCompile(`(?i)\babove\b[ \t]*([.,;:!?)\]"'’”—]|\n|$)`)
|
// CRLF transcript does not quietly lose every line-final "above".
|
||||||
|
var aboveRefRe = regexp.MustCompile(`(?i)\babove\b[ \t]*([.,;:!?)\]"'’”—]|\r|\n|$)`)
|
||||||
|
|
||||||
// pointsAbove reports whether a deictic "above" appears in the terminal's
|
// pointsAbove reports whether a deictic "above" appears in the terminal's
|
||||||
// OPENING. The offset bound is what makes a bare "above" safe to key on: with
|
// OPENING. The offset bound is what makes a bare "above" safe to key on: with
|
||||||
@@ -157,6 +158,23 @@ func pointsAbove(t string) bool {
|
|||||||
return loc != nil && loc[0] <= backRefHeadChars
|
return loc != nil && loc[0] <= backRefHeadChars
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// compressionMarkerRe matches a model announcing that what follows is the
|
||||||
|
// short form of something longer ("Short version: …", "TL;DR: …", "In short,
|
||||||
|
// …"). It is the second half of the deictic summary-closer test: the pointer
|
||||||
|
// says the full answer is elsewhere, and this says the text beside it is a
|
||||||
|
// condensation rather than new reasoning.
|
||||||
|
//
|
||||||
|
// Both halves are required, because a deictic pointer alone does not mean the
|
||||||
|
// terminal is disposable. "Given the analysis above, I recommend option B
|
||||||
|
// because X" opens with a pointer and then states a CONCLUSION the earlier
|
||||||
|
// turn never contained — discarding it in favour of that turn would throw away
|
||||||
|
// the answer. A compression marker is the model telling us the opposite.
|
||||||
|
//
|
||||||
|
// A marker WITHOUT a pointer stays out of scope, as summaryCloserRe's own
|
||||||
|
// comment explains: a user who asked for brevity is answered with exactly that
|
||||||
|
// shape.
|
||||||
|
var compressionMarkerRe = regexp.MustCompile(`(?i)\b(short version|shorter version|short answer|tl;?dr|in short|in brief|in summary|in sum|bottom line|net[- ]net|the gist)\b`)
|
||||||
|
|
||||||
// summaryCloserRe matches a terminal turn that OPENS with a bookkeeping
|
// summaryCloserRe matches a terminal turn that OPENS with a bookkeeping
|
||||||
// acknowledgment of the citation round — "Citations are logged.", "Sources
|
// acknowledgment of the citation round — "Citations are logged.", "Sources
|
||||||
// cited.", "Logged the citations." — the shape a model produces when it
|
// cited.", "Logged the citations." — the shape a model produces when it
|
||||||
@@ -296,10 +314,21 @@ func isCitationsOnly(s string) bool {
|
|||||||
|
|
||||||
// isSummaryCloser reports whether a terminal turn defers to an earlier answer
|
// isSummaryCloser reports whether a terminal turn defers to an earlier answer
|
||||||
// and is short enough that whatever follows the deferral can only be a
|
// and is short enough that whatever follows the deferral can only be a
|
||||||
// compression of it. Two openers qualify: a complete "citations are logged"
|
// compression of it. Two openers qualify:
|
||||||
// -style ack sentence (summaryCloserRe), and a deictic back-reference in the
|
//
|
||||||
// terminal's opening (pointsAbove — mort issue #1611's "Done — that's the full
|
// - a complete bookkeeping ack sentence — "Citations are logged." — which is
|
||||||
// chain above. Short version: …", which the ack shape alone did not cover).
|
// summaryCloserRe, and carries no answer content of its own;
|
||||||
|
// - a deictic back-reference in the opening PLUS a compression marker —
|
||||||
|
// mort issue #1611's "Done — that's the full chain above. Short version:
|
||||||
|
// …" — which the ack shape alone did not cover.
|
||||||
|
//
|
||||||
|
// The second opener needs both halves. A pointer on its own does not make a
|
||||||
|
// terminal disposable: "Given the analysis above, I recommend option B because
|
||||||
|
// X" points backwards and then states a conclusion the earlier turn never
|
||||||
|
// contained, and replacing it with that turn would discard the answer. The
|
||||||
|
// compression marker is the model saying the opposite — that what follows is
|
||||||
|
// the short form of something it already wrote.
|
||||||
|
//
|
||||||
// Whether the fuller answer actually exists is modeSummary's job — the dwarf
|
// Whether the fuller answer actually exists is modeSummary's job — the dwarf
|
||||||
// ratio in isSubstantiveAnswer keeps a matching closer in place when nothing
|
// ratio in isSubstantiveAnswer keeps a matching closer in place when nothing
|
||||||
// earlier clearly outweighs it.
|
// earlier clearly outweighs it.
|
||||||
@@ -313,7 +342,8 @@ func isSummaryCloser(s string) bool {
|
|||||||
if t == "" || len(t) > summaryCloserMaxChars {
|
if t == "" || len(t) > summaryCloserMaxChars {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
return summaryCloserRe.MatchString(t) || pointsAbove(t)
|
return summaryCloserRe.MatchString(t) ||
|
||||||
|
(pointsAbove(t) && compressionMarkerRe.MatchString(t))
|
||||||
}
|
}
|
||||||
|
|
||||||
// lastSubstantiveAssistantText scans msgs newest→oldest (skipping the terminal
|
// lastSubstantiveAssistantText scans msgs newest→oldest (skipping the terminal
|
||||||
|
|||||||
+31
-2
@@ -70,6 +70,8 @@ func TestPointsAbove(t *testing.T) {
|
|||||||
{"comma", "As shown above, the answer is 60 minutes.", true},
|
{"comma", "As shown above, the answer is 60 minutes.", true},
|
||||||
{"end-of-string", "The full breakdown is above", true},
|
{"end-of-string", "The full breakdown is above", true},
|
||||||
{"line-final", "Everything is above\n\nShort version: yes.", true},
|
{"line-final", "Everything is above\n\nShort version: yes.", true},
|
||||||
|
{"line-final-crlf", "Everything is above\r\nShort version: yes.", true},
|
||||||
|
{"crlf-at-end", "The full breakdown is above\r\n", true},
|
||||||
{"closing-paren", "(the detail is above).", true},
|
{"closing-paren", "(the detail is above).", true},
|
||||||
{"semicolon", "It's above; the short answer is no.", true},
|
{"semicolon", "It's above; the short answer is no.", true},
|
||||||
|
|
||||||
@@ -80,8 +82,9 @@ func TestPointsAbove(t *testing.T) {
|
|||||||
{"above-average", "Turnout was above average in three counties.", false},
|
{"above-average", "Turnout was above average in three counties.", false},
|
||||||
|
|
||||||
// Hyphenated compounds. \b holds between "above" and "-", so a literal
|
// Hyphenated compounds. \b holds between "above" and "-", so a literal
|
||||||
// '-' in the terminator class made all of these read as deictic — the
|
// '-' in the terminator class makes all of these read as deictic. The
|
||||||
// space-separated cases above did NOT cover it (gadfly, 3 models).
|
// space-separated cases above do NOT cover this: they are a different
|
||||||
|
// character, and one passed while the other was broken.
|
||||||
{"hyphen-above-average", "Turnout was above-average in three counties.", false},
|
{"hyphen-above-average", "Turnout was above-average in three counties.", false},
|
||||||
{"hyphen-above-board", "The deal was above-board from the start.", false},
|
{"hyphen-above-board", "The deal was above-board from the start.", false},
|
||||||
{"hyphen-above-ground", "Run the above-ground cable along the fence.", false},
|
{"hyphen-above-ground", "Run the above-ground cable along the fence.", false},
|
||||||
@@ -196,6 +199,13 @@ func TestIsSummaryCloser(t *testing.T) {
|
|||||||
{"1611-verbatim", closer1611, true},
|
{"1611-verbatim", closer1611, true},
|
||||||
{"above-pointer-plus-tldr", "That's the whole picture above. In short: the merger fell through.", true},
|
{"above-pointer-plus-tldr", "That's the whole picture above. In short: the merger fell through.", true},
|
||||||
{"prepositional-above-is-not-a-closer", "Anything above 100 degrees boils off, which is why the sample evaporated.", false},
|
{"prepositional-above-is-not-a-closer", "Anything above 100 degrees boils off, which is why the sample evaporated.", false},
|
||||||
|
|
||||||
|
// A deictic pointer WITHOUT a compression marker is not a summary
|
||||||
|
// closer: these state a conclusion the earlier turn never contained,
|
||||||
|
// so replacing them with that turn would discard the answer.
|
||||||
|
{"pointer-then-a-recommendation", "Given the analysis above, I recommend option B: it is the only one that survives a regional outage.", false},
|
||||||
|
{"pointer-then-a-decision", "Based on everything above, we should ship Tuesday and hold the migration until the following sprint.", false},
|
||||||
|
{"pointer-then-a-new-caveat", "That is the chain above. One thing it misses: the Senate vote is scheduled before any of this takes effect.", false},
|
||||||
{"1611-over-cap", closer1611 + " " + strings.Repeat("Plenty more detail worth keeping here. ", 4), false}, // >300
|
{"1611-over-cap", closer1611 + " " + strings.Repeat("Plenty more detail worth keeping here. ", 4), false}, // >300
|
||||||
}
|
}
|
||||||
for _, c := range cases {
|
for _, c := range cases {
|
||||||
@@ -250,6 +260,10 @@ func TestFinalOutput(t *testing.T) {
|
|||||||
// modeBackRef bar (>=200 bytes, no ratio) accepts it. A shorter closer
|
// modeBackRef bar (>=200 bytes, no ratio) accepts it. A shorter closer
|
||||||
// would pass under either bar and prove nothing.
|
// would pass under either bar and prove nothing.
|
||||||
shortAbovePointer := "Done. The whole chain is above, so there is no point repeating all of that detail down here again."
|
shortAbovePointer := "Done. The whole chain is above, so there is no point repeating all of that detail down here again."
|
||||||
|
// A deictic pointer followed by a NEW conclusion (no compression marker):
|
||||||
|
// >120 bytes so isWeakFinal cannot claim it, and it must not be treated as
|
||||||
|
// a summary closer either.
|
||||||
|
pointerThenConclusion := "Given the analysis above, I recommend option B: it is the only one that survives a regional outage without a manual failover step."
|
||||||
// A terminal using "above" as a PREPOSITION — not a back-reference, so it
|
// A terminal using "above" as a PREPOSITION — not a back-reference, so it
|
||||||
// must survive verbatim next to a dwarfing prior turn.
|
// must survive verbatim next to a dwarfing prior turn.
|
||||||
prepositionalTerminal := "Anything above 100 degrees boils off, which is exactly why the sample evaporated overnight in the unsealed tray."
|
prepositionalTerminal := "Anything above 100 degrees boils off, which is exactly why the sample evaporated overnight in the unsealed tray."
|
||||||
@@ -556,6 +570,21 @@ func TestFinalOutput(t *testing.T) {
|
|||||||
terminal: shortAbovePointer,
|
terminal: shortAbovePointer,
|
||||||
want: longAnswer,
|
want: longAnswer,
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
// A pointer plus a NEW conclusion is not a compression, so it must
|
||||||
|
// survive verbatim even though a much longer prior turn exists —
|
||||||
|
// otherwise the recommendation is thrown away in favour of the
|
||||||
|
// analysis it was drawn from.
|
||||||
|
name: "above-pointer closer with a new conclusion is not discarded",
|
||||||
|
msgs: []llm.Message{
|
||||||
|
llm.UserText("which option?"),
|
||||||
|
asst(hugeAnswer, cite...),
|
||||||
|
llm.ToolResultsMessage(llm.ToolResult{ID: "c1", Name: "cite", Content: "ok"}),
|
||||||
|
asst(pointerThenConclusion),
|
||||||
|
},
|
||||||
|
terminal: pointerThenConclusion,
|
||||||
|
want: pointerThenConclusion,
|
||||||
|
},
|
||||||
{
|
{
|
||||||
// A closer matching BOTH the ack shape and a back-reference
|
// A closer matching BOTH the ack shape and a back-reference
|
||||||
// carries no answer content, so the back-ref test must win and the
|
// carries no answer content, so the back-ref test must win and the
|
||||||
|
|||||||
Reference in New Issue
Block a user