Author SHA1 Message Date
steveandClaude Fable 5 8eb0265e75 fix(reusable): thread the dispatch pr_number as a workflow_call input (hotfix lineage)
Mirrors gadfly#24 on main. Env/inputs only; image sha-ed9e946 unchanged.

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-07-15 23:34:37 -04:00
steveandClaude Fable 5 3664ce858d chore(reusable): replace the retired ragnaros endpoint with netherstorm (hotfix lineage)
Mirrors gadfly#23 on main. Env-only; image sha-ed9e946 unchanged.

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-07-15 23:32:30 -04:00
steveandClaude Fable 5 f542d4e1cd chore(reusable): forward GADFLY_ENDPOINT_NETHERSTORM (hotfix lineage)
Mirrors gadfly#22 on main: the var was added after the forwarding list,
so netherstorm reviewers failed with 'unknown provider' — the env never
crossed the workflow_call boundary. Env-only; image sha-ed9e946 unchanged.

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-07-15 23:30:50 -04:00
steveandClaude Fable 5 b6a33dc218 chore(reusable): bump image pin to sha-ed9e946 (Gitea 1.27 workflow_call hotfix)
Same binary lineage as sha-3095ebf (5007597 + entrypoint fix only) — this
is the conservative hotfix pin for consumers; the executus re-platform on
main rolls out separately.

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-07-15 23:01:30 -04:00
steveandClaude Fable 5 ed9e946cd1 fix: handle Gitea 1.27's workflow_call event name in the trigger gate
Gitea 1.27 (breaking change go-gitea#37478) runs a called (reusable)
workflow with github.event_name = 'workflow_call' instead of propagating
the caller's event. Every consumer stub forwards EVENT_NAME from
github.event_name, so since the server upgrade every review arrived as an
unhandled event and self-skipped in one second while reporting success —
mort PRs #1445-#1447 all went unreviewed.

Reclassify workflow_call from the forwarded payload: a non-empty
COMMENT_BODY can only come from issue_comment (trigger-phrase + actor
gates still apply); otherwise a PR number means a pull_request-shaped
trigger. Neither → the existing unhandled-event skip. Pre-1.27 servers
are unaffected.

Co-Authored-By: Claude Fable 5 <[email protected]>
2026-07-15 23:00:43 -04:00
steveandClaude Opus 4.8 5007597cf9 chore(reusable): bump image pin to sha-3095ebf (inline PR review live)
Phase 3: gadfly's own multi-model reviews now also post a COMMENT-state PR
review with inline comments anchored to changed lines. External consumers
re-pin separately.

[skip ci]

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-28 22:00:39 -04:00
steve 3095ebff23 feat: inline COMMENT-state PR review (findings anchored to changed lines) (#18)
Build & push image / build-and-push (push) Successful in 8s
Co-authored-by: Steve Dudenhoeffer <[email protected]>
Co-committed-by: Steve Dudenhoeffer <[email protected]>
2026-06-29 01:59:36 +00:00
steveandClaude Opus 4.8 8f5adc91b2 chore(reusable): bump image pin to sha-88f74aa (consensus consolidation live)
Phase 2: gadfly's own multi-model reviews now post ONE cross-model consensus
comment instead of N per-model comments. External consumers re-pin separately.

[skip ci]

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-28 18:57:38 -04:00
steve 88f74aa768 feat: cross-model consensus consolidation (one ranked comment, not N walls) (#17)
Build & push image / build-and-push (push) Successful in 9s
Co-authored-by: Steve Dudenhoeffer <[email protected]>
Co-committed-by: Steve Dudenhoeffer <[email protected]>
2026-06-28 22:56:15 +00:00
steveandClaude Opus 4.8 84b891b1ba chore(reusable): bump image pin to sha-5397160 (structured findings contract)
Makes the Phase 1 gadfly-findings contract live for gadfly's own dogfood
reviews (the local-ref reusable). External consumers re-pin separately.

[skip ci]

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-28 18:25:16 -04:00
steve 53971603d3 feat: structured findings contract (machine-readable gadfly-findings block) (#16)
Build & push image / build-and-push (push) Successful in 5s
Co-authored-by: Steve Dudenhoeffer <[email protected]>
Co-committed-by: Steve Dudenhoeffer <[email protected]>
2026-06-28 22:23:02 +00:00
steve f49699fc12 Merge pull request 'docs: correct examples/reusable.yml pin guidance (prefer @sha; runners cache @v1)' (#15) from test/trigger-check into main
Reviewed-on: #15
2026-06-28 22:09:07 +00:00
Steve DudenhoefferandClaude Opus 4.8 6e87a3e73f docs: correct examples/reusable.yml pin guidance (runners cache @v1; prefer @sha)
Adversarial Review (Gadfly) / review (pull_request) Successful in 3m4s
The @v1 comment claimed it auto-updates on releases, but long-lived act_runners
cache the reusable by ref so a moved tag isn't re-fetched. Recommend an
immutable @<sha>; routine tuning rides owner variables.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-06-28 02:10:35 -04:00
steve 7bc3c982fa feat(reusable): runtime-variable swarm config (cache-immune, no more re-pinning to retune) (#14)
Build & push image / build-and-push (push) Successful in 5s
2026-06-28 06:00:18 +00:00
17 changed files with 1777 additions and 49 deletions
+20 -9
View File
@@ -45,7 +45,7 @@ on:
# Owner-set user-scope variables (see README "Central config via variables"): # Owner-set user-scope variables (see README "Central config via variables"):
# GADFLY_DEFAULT_MODELS, GADFLY_DEFAULT_SPECIALISTS, # GADFLY_DEFAULT_MODELS, GADFLY_DEFAULT_SPECIALISTS,
# GADFLY_DEFAULT_PROVIDER_CONCURRENCY, GADFLY_DEFAULT_PROVIDER_LENS_CONCURRENCY, # GADFLY_DEFAULT_PROVIDER_CONCURRENCY, GADFLY_DEFAULT_PROVIDER_LENS_CONCURRENCY,
# GADFLY_ENDPOINT_RAGNAROS (the 4090 Ti endpoint). # GADFLY_ENDPOINT_NETHERSTORM (the local GPU box endpoint).
# An unset variable + no input → the image default (one model, default suite), # An unset variable + no input → the image default (one model, default suite),
# so a public consumer with neither still gets a sane minimal review. # so a public consumer with neither still gets a sane minimal review.
inputs: inputs:
@@ -59,7 +59,14 @@ on:
max_steps: { type: string, default: "14" } # GADFLY_MAX_STEPS max_steps: { type: string, default: "14" } # GADFLY_MAX_STEPS
worker_model: { type: string, default: "" } # GADFLY_WORKER_MODEL worker_model: { type: string, default: "" } # GADFLY_WORKER_MODEL
allowed_users: { type: string, default: "" } # GADFLY_ALLOWED_USERS (consumer-specific; set in your stub) allowed_users: { type: string, default: "" } # GADFLY_ALLOWED_USERS (consumer-specific; set in your stub)
# Gitea >= 1.27 does not propagate the CALLER's workflow_dispatch inputs into a
# called workflow's github.event, so a manual "review PR #N" dispatch arrived
# here with an empty PR and died at the entrypoint's "PR required" check. The
# caller stub must thread it explicitly: `pr_number: ${{ github.event.inputs.pr_number }}`.
pr_number: { type: string, default: "" }
trigger_phrase: { type: string, default: "" } # GADFLY_TRIGGER_PHRASE trigger_phrase: { type: string, default: "" } # GADFLY_TRIGGER_PHRASE
consolidate: { type: string, default: "" } # GADFLY_CONSOLIDATE — "" => auto (one consensus comment for >=2 models); "0" => one comment per model
inline_review: { type: string, default: "" } # GADFLY_INLINE_REVIEW — "" => on (post a COMMENT-state PR review with inline comments on changed lines); "0" => off
# Job wall-clock cap. 90 as a default: the 5-lens suite across a slow lane # Job wall-clock cap. 90 as a default: the 5-lens suite across a slow lane
# (claude-code with extended thinking) over two passes can run long. # (claude-code with extended thinking) over two passes can run long.
timeout_minutes: { type: number, default: 90 } timeout_minutes: { type: number, default: 90 }
@@ -92,7 +99,7 @@ jobs:
runs-on: ubuntu-latest runs-on: ubuntu-latest
timeout-minutes: ${{ inputs.timeout_minutes }} timeout-minutes: ${{ inputs.timeout_minutes }}
steps: steps:
- uses: docker://gitea.stevedudenhoeffer.com/steve/gadfly:sha-c342bdb - uses: docker://gitea.stevedudenhoeffer.com/steve/gadfly:sha-ed9e946
env: env:
# --- event context (from the CALLER's github.*) ------------------- # --- event context (from the CALLER's github.*) -------------------
GITEA_API: ${{ github.server_url }}/api/v1/repos/${{ github.repository }} GITEA_API: ${{ github.server_url }}/api/v1/repos/${{ github.repository }}
@@ -102,7 +109,7 @@ jobs:
# forwarding, since the auto token isn't a forwarded workflow_call secret. # forwarding, since the auto token isn't a forwarded workflow_call secret.
GITEA_TOKEN: ${{ github.token }} GITEA_TOKEN: ${{ github.token }}
EVENT_NAME: ${{ github.event_name }} EVENT_NAME: ${{ github.event_name }}
PR: ${{ github.event.pull_request.number || github.event.issue.number || github.event.inputs.pr_number }} PR: ${{ github.event.pull_request.number || github.event.issue.number || github.event.inputs.pr_number || inputs.pr_number }}
PR_BRANCH: ${{ github.head_ref }} PR_BRANCH: ${{ github.head_ref }}
IS_DRAFT: ${{ github.event.pull_request.draft }} IS_DRAFT: ${{ github.event.pull_request.draft }}
COMMENT_BODY: ${{ github.event.comment.body }} COMMENT_BODY: ${{ github.event.comment.body }}
@@ -122,12 +129,14 @@ jobs:
# token, keep that one a secret instead. # token, keep that one a secret instead.
GADFLY_ENDPOINT_M1: ${{ vars.GADFLY_ENDPOINT_M1 }} GADFLY_ENDPOINT_M1: ${{ vars.GADFLY_ENDPOINT_M1 }}
GADFLY_ENDPOINT_M5: ${{ vars.GADFLY_ENDPOINT_M5 }} GADFLY_ENDPOINT_M5: ${{ vars.GADFLY_ENDPOINT_M5 }}
# ragnaros = the 4090 Ti via its llama-swap proxy. Defined in the user # netherstorm = the local GPU box via its llama-swap proxy. Defined in the
# var GADFLY_ENDPOINT_RAGNAROS (format "<provider>|<base-url>[|<key>]") # user var GADFLY_ENDPOINT_NETHERSTORM (format "<provider>|<base-url>[|<key>]",
# so the URL can change without editing this file; the matching model is # e.g. "llamaswap|https://llama-swap.netherstorm...") so the URL can change
# ragnaros/qwen3.6-27b in GADFLY_DEFAULT_MODELS. NB: use the un-hyphenated # without editing this file; the matching model is netherstorm/qwen3.6-27b in
# `llamaswap` provider spelling in the var — the pinned image needs it. # GADFLY_DEFAULT_MODELS. NB: use the un-hyphenated `llamaswap` provider
GADFLY_ENDPOINT_RAGNAROS: ${{ vars.GADFLY_ENDPOINT_RAGNAROS }} # spelling in the var — the pinned image needs it. Without this line the var
# is silently dropped at the workflow_call boundary ('unknown provider').
GADFLY_ENDPOINT_NETHERSTORM: ${{ vars.GADFLY_ENDPOINT_NETHERSTORM }}
# --- findings telemetry (optional) -------------------------------- # --- findings telemetry (optional) --------------------------------
GADFLY_FINDINGS_URL: ${{ secrets.GADFLY_FINDINGS_URL }} GADFLY_FINDINGS_URL: ${{ secrets.GADFLY_FINDINGS_URL }}
GADFLY_FINDINGS_TOKEN: ${{ secrets.GADFLY_FINDINGS_TOKEN }} GADFLY_FINDINGS_TOKEN: ${{ secrets.GADFLY_FINDINGS_TOKEN }}
@@ -143,3 +152,5 @@ jobs:
GADFLY_WORKER_MODEL: ${{ inputs.worker_model }} GADFLY_WORKER_MODEL: ${{ inputs.worker_model }}
GADFLY_ALLOWED_USERS: ${{ inputs.allowed_users }} GADFLY_ALLOWED_USERS: ${{ inputs.allowed_users }}
GADFLY_TRIGGER_PHRASE: ${{ inputs.trigger_phrase }} GADFLY_TRIGGER_PHRASE: ${{ inputs.trigger_phrase }}
GADFLY_CONSOLIDATE: ${{ inputs.consolidate }}
GADFLY_INLINE_REVIEW: ${{ inputs.inline_review }}
+39
View File
@@ -277,6 +277,43 @@ every `GADFLY_STATUS_POLL_SECS` (default 12s) until the swarm finishes. It's adv
best-effort — the per-model findings comments are unaffected — and entirely separate from those. best-effort — the per-model findings comments are unaffected — and entirely separate from those.
Turn it off with `GADFLY_STATUS_BOARD=0`. Turn it off with `GADFLY_STATUS_BOARD=0`.
### Consensus consolidation
With **two or more models**, posting one comment each means a reader faces N walls of prose that
mostly agree. Instead Gadfly consolidates: every model writes its findings to a shared file, and
after the whole swarm finishes a single pass clusters those findings by location, counts **how
many models independently flagged each one**, and posts **one consensus comment**:
```
## 🪰 Gadfly review — consensus across 7 models
**Verdict: Blocking issues found** · 9 findings (3 with multi-model agreement)
| | Finding | Where | Models | Lens |
|--|--|--|--|--|
| 🔴 | Auth bypass: token not verified | `auth/login.go:42` | 6/7 | security |
| 🟠 | Unbounded retry loop | `sync/worker.go:88` | 3/7 | error-handling |
<details><summary>4 single-model findings (lower confidence)</summary> … </details>
<details><summary>Per-model detail</summary> … each model's full review, folded … </details>
```
Cross-model agreement is the strongest real-vs-false-positive signal available, so findings are
ranked by it (a lone low-severity finding folds away; a lone *critical* still surfaces). The
per-model comments are suppressed in this mode — each model's full review is preserved, folded,
inside the consensus comment — and nothing is lost: if consolidation can't run, Gadfly falls
back to posting the per-model comments. Controlled by `GADFLY_CONSOLIDATE`: `auto` (default — on
for ≥2 models), `1` (force on), `0` (force off, one comment per model). Single-model runs are
unaffected.
**Inline PR review.** Alongside the consensus comment, Gadfly also posts a single Gitea **pull
review** (state `COMMENT` — advisory, **never** request-changes or approve, so it can't block a
merge) whose inline comments anchor each consensus finding to the exact changed line it's about.
Only findings that land on a line in the diff are anchored (Gitea rejects comments off the diff);
the rest stay in the consensus comment. A re-run replaces the previous review instead of stacking.
It's the "reviewer integrated with Gitea" without the blocking — turn it off with
`GADFLY_INLINE_REVIEW=0`.
### Triggers ### Triggers
1. A **new/reopened/ready** non-draft PR — automatic. 1. A **new/reopened/ready** non-draft PR — automatic.
@@ -362,6 +399,8 @@ The reviewer binary reads these (the stub/entrypoint set sane defaults):
| `GADFLY_MAX_DIFF_CHARS` | 60000 | diff chars embedded in the prompt (full diff via `get_diff`) | | `GADFLY_MAX_DIFF_CHARS` | 60000 | diff chars embedded in the prompt (full diff via `get_diff`) |
| `GADFLY_STATUS_BOARD` | on | set `0` to disable the live status-board comment | | `GADFLY_STATUS_BOARD` | on | set `0` to disable the live status-board comment |
| `GADFLY_STATUS_POLL_SECS` | 12 | how often the status board re-renders/upserts | | `GADFLY_STATUS_POLL_SECS` | 12 | how often the status board re-renders/upserts |
| `GADFLY_CONSOLIDATE` | `auto` | cross-model consensus comment: `auto` (on for ≥2 models), `1` (force on), `0` (off — one comment per model) |
| `GADFLY_INLINE_REVIEW` | on | when consolidating, also post a `COMMENT`-state PR review with inline comments on changed lines; `0` disables |
| `GADFLY_TRIGGER_PHRASE` | `@gadfly review` | comment phrase that re-triggers | | `GADFLY_TRIGGER_PHRASE` | `@gadfly review` | comment phrase that re-triggers |
| `GADFLY_ALLOWED_USERS` | *(collaborators)* | comma-separated allow-list for comment triggers | | `GADFLY_ALLOWED_USERS` | *(collaborators)* | comma-separated allow-list for comment triggers |
| `GADFLY_FINDINGS_URL` | — | gadfly-reports store base URL; set to enable findings telemetry (off when empty) | | `GADFLY_FINDINGS_URL` | — | gadfly-reports store base URL; set to enable findings telemetry (off when empty) |
+477
View File
@@ -0,0 +1,477 @@
package main
// Cross-model consensus consolidation. The swarm runs each model independently
// (entrypoint.sh fans them out across provider lanes); historically each model
// posted its OWN comment, so a reader faced N walls of prose that mostly agreed.
//
// Instead, every model writes its findings to a shared directory
// (GADFLY_FINDINGS_OUT, one JSON file per model), and after the whole swarm
// finishes a single consolidation pass (GADFLY_CONSOLIDATE_DIR) clusters those
// findings by location, counts how many models independently flagged each one,
// and renders ONE comment: an agreement-ranked table up top (cross-model
// agreement is the strongest real-vs-false-positive signal we have), with each
// model's full review folded below for drill-down.
//
// This file owns: the per-model artifact (modelFindings), writing it
// (writeFindingsOut), reading the directory back, clustering, and rendering the
// consensus markdown (renderConsensus). It depends only on the stdlib.
import (
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
)
// consensusMarker tags the single consolidated comment so entrypoint.sh can
// upsert it in place across re-runs (mirrors run.sh's per-model marker).
const consensusMarker = "<!-- gadfly-consensus -->"
// modelFindings is the per-model artifact written to GADFLY_FINDINGS_OUT. It
// carries enough to rebuild the consolidated comment without re-running models:
// the structured findings (for clustering) plus the full rendered review (for
// the folded per-model drill-down).
type modelFindings struct {
Model string `json:"model"`
Provider string `json:"provider"`
Verdict string `json:"verdict"` // worst lens verdict, as a label
Errored bool `json:"errored"` // the model produced no usable review (every lens failed, or the run crashed)
Markdown string `json:"markdown"` // full rendered per-model review (findings block already stripped)
Findings []outFinding `json:"findings"`
}
// outFinding is one finding in the per-model artifact (a flattened `finding`
// plus its lens).
type outFinding struct {
Lens string `json:"lens"`
File string `json:"file"`
Line int `json:"line"`
Severity string `json:"severity"`
Confidence string `json:"confidence"`
Title string `json:"title"`
Detail string `json:"detail"`
}
// collectFindings returns a lens result's findings with severity always filled
// in: per-finding when the structured block supplied it, else derived from the
// lens verdict (so heuristic-scraped findings still carry a canonical word). A
// clean or errored lens yields nothing. Shared by the telemetry emit and the
// per-model findings file so both agree on what a finding is.
func collectFindings(r specialistResult) []finding {
if r.errored || r.verdict == verdictClean {
return nil
}
fs := extractStructuredFindingsOrScrape(r)
lensSev := r.verdict.severity()
for i := range fs {
if fs[i].severity == "" {
fs[i].severity = lensSev
}
}
return fs
}
// writeFindingsOut writes this model's findings + rendered review to
// GADFLY_FINDINGS_OUT for the later consolidation pass. No-op unless the env is
// set. Best-effort: any error is logged to stderr and never affects the review
// (it runs after the markdown is already on stdout).
func writeFindingsOut(results []specialistResult) {
path := strings.TrimSpace(os.Getenv("GADFLY_FINDINGS_OUT"))
if path == "" {
return
}
mf := modelFindings{
Model: strings.TrimSpace(os.Getenv("GADFLY_MODEL")),
Provider: modelProvider(),
Verdict: worstVerdict(results).label(),
Errored: allErrored(results),
Markdown: renderConsolidated(results),
}
for _, r := range results {
for _, f := range collectFindings(r) {
mf.Findings = append(mf.Findings, outFinding{
Lens: r.spec.Name,
File: f.file,
Line: f.line,
Severity: f.severity,
Confidence: f.confidence,
Title: f.title,
Detail: f.detail,
})
}
}
data, err := json.Marshal(mf)
if err != nil {
fmt.Fprintln(os.Stderr, "gadfly: marshal findings out:", err)
return
}
// Defensive: make sure the parent dir exists (entrypoint creates it, but a
// missing dir would otherwise silently drop this model from the consensus).
if dir := filepath.Dir(path); dir != "" {
_ = os.MkdirAll(dir, 0o755)
}
if err := os.WriteFile(path, data, 0o644); err != nil {
fmt.Fprintln(os.Stderr, "gadfly: write findings out:", err)
}
}
// allErrored reports whether every lens of a review failed (so the model
// produced no usable findings). Such a model is recorded but excluded from the
// consensus agreement denominator — counting it would dilute every ratio with a
// model that never actually reviewed.
func allErrored(results []specialistResult) bool {
if len(results) == 0 {
return true
}
for _, r := range results {
if !r.errored {
return false
}
}
return true
}
// runConsolidate is the consolidation entry point (GADFLY_CONSOLIDATE_DIR set):
// read every per-model artifact in the directory, render the consensus comment
// to stdout. Errors are fatal to THIS process only — entrypoint.sh treats a
// failed consolidation as advisory and falls back to per-model comments.
func runConsolidate() error {
dir := strings.TrimSpace(os.Getenv("GADFLY_CONSOLIDATE_DIR"))
if dir == "" {
return errors.New("GADFLY_CONSOLIDATE_DIR is empty")
}
entries, err := os.ReadDir(dir)
if err != nil {
return fmt.Errorf("read consolidate dir: %w", err)
}
var models []modelFindings
for _, e := range entries {
if e.IsDir() || !strings.HasSuffix(e.Name(), ".json") {
continue
}
data, err := os.ReadFile(filepath.Join(dir, e.Name()))
if err != nil {
fmt.Fprintln(os.Stderr, "gadfly: read", e.Name(), err)
continue
}
var mf modelFindings
if err := json.Unmarshal(data, &mf); err != nil {
fmt.Fprintln(os.Stderr, "gadfly: parse", e.Name(), err)
continue
}
if strings.TrimSpace(mf.Model) == "" {
continue
}
models = append(models, mf)
}
if len(models) == 0 {
return errors.New("no model findings to consolidate")
}
// Cluster once, then render the consensus comment and (best-effort) the inline
// PR review from the same clusters so the two views can't drift.
clusters := clusterFindings(models)
// Lead with the marker so entrypoint.sh can upsert this comment in place
// (same pattern as run.sh's per-model marker); it appends the advisory footer.
fmt.Println(consensusMarker)
fmt.Println(renderConsensus(models, clusters))
// Inline PR review (COMMENT state) anchoring findings to changed lines.
// Best-effort: no-op without diff/API creds, never affects stdout/exit.
postInlineReview(clusters)
return nil
}
// cluster is a group of findings (across models) judged to be the same issue:
// same file, lines within lineTolerance of the cluster's current span. The span
// [line,maxLine] slides as members join, so a chain of nearby findings merges
// instead of splitting once it drifts past the first line.
type cluster struct {
file string
line int // representative (smallest) line
maxLine int // largest line in the cluster — the span's upper edge
severity string
title string
detail string // highest-severity report's detail; rendered in the inline review comment
models map[string]bool
lenses map[string]bool
}
// findingRef is one model's finding (carrying which model reported it), used
// while grouping findings into clusters.
type findingRef struct {
f outFinding
model string
}
// lineTolerance: a finding in the same file within this many lines of a
// cluster's current span is treated as the same issue (models often cite a line
// or two apart).
const lineTolerance = 3
// renderConsensus builds the single consolidated comment body from every model's
// findings. It does NOT emit the marker or advisory footer — entrypoint.sh wraps
// it (mirroring run.sh's per-model framing).
func renderConsensus(models []modelFindings, clusters []cluster) string {
// effective = models that actually produced a review. Errored models are
// shown (folded, below) but excluded from the agreement denominator so a
// failed model doesn't dilute every ratio.
effective := 0
for _, m := range models {
if !m.Errored {
effective++
}
}
errored := len(models) - effective
worst := verdictClean
for _, m := range models {
if v := parseVerdict(m.Verdict); v > worst {
worst = v
}
}
// Partition in one pass: "headline" findings (multi-model agreement, OR a
// lone CRITICAL) vs folded "single-model" lower-confidence findings. Also
// count multi-model agreements for the summary line.
var headline, folded []cluster
agreed := 0
for _, c := range clusters {
if len(c.models) >= 2 {
agreed++
}
if len(c.models) >= 2 || sevRank(c.severity) >= sevRank("critical") {
headline = append(headline, c)
} else {
folded = append(folded, c)
}
}
var b strings.Builder
fmt.Fprintf(&b, "## 🪰 Gadfly review — consensus across %d model%s", effective, plural(effective))
if errored > 0 {
fmt.Fprintf(&b, " (%d failed)", errored)
}
b.WriteString("\n\n")
fmt.Fprintf(&b, "**Verdict: %s** · %d finding%s (%d with multi-model agreement)\n",
worst.label(), len(clusters), plural(len(clusters)), agreed)
if len(headline) > 0 {
b.WriteString("\n| | Finding | Where | Models | Lens |\n|--|--|--|--|--|\n")
for _, c := range headline {
fmt.Fprintf(&b, "| %s | %s | `%s` | %d/%d | %s |\n",
sevIcon(c.severity), mdCell(c.title), mdCell(location(c.file, c.line)),
len(c.models), effective, mdCell(lensList(c.lenses)))
}
} else if len(clusters) == 0 {
b.WriteString("\nNo material issues found by consensus.\n")
}
// else: only single-model findings — they're shown folded below, so don't
// claim "no material issues" (there are some, just none with consensus).
if len(folded) > 0 {
fmt.Fprintf(&b, "\n<details><summary>%d single-model finding%s (lower confidence)</summary>\n\n",
len(folded), plural(len(folded)))
b.WriteString("| | Finding | Where | Model | Lens |\n|--|--|--|--|--|\n")
for _, c := range folded {
fmt.Fprintf(&b, "| %s | %s | `%s` | %s | %s |\n",
sevIcon(c.severity), mdCell(c.title), mdCell(location(c.file, c.line)),
mdCell(oneModel(c.models)), mdCell(lensList(c.lenses)))
}
b.WriteString("\n</details>\n")
}
// Per-model full reviews, folded for drill-down (nothing is lost).
b.WriteString("\n<details><summary>Per-model detail</summary>\n")
for _, m := range models {
body := strings.TrimSpace(m.Markdown)
if body == "" {
body = "_(no output)_"
}
verdict := m.Verdict
if m.Errored {
verdict = "⚠️ reviewer failed"
}
fmt.Fprintf(&b, "\n<details><summary><b>%s</b> (%s) — %s</summary>\n\n%s\n\n</details>\n",
mdCell(m.Model), mdCell(m.Provider), verdict, body)
}
b.WriteString("\n</details>")
return b.String()
}
// clusterFindings groups every model's findings into cross-model clusters,
// sorted by agreement (desc), then severity (desc), then location.
func clusterFindings(models []modelFindings) []cluster {
// Group by file, then greedily merge by line proximity.
byFile := map[string][]findingRef{}
for _, m := range models {
for _, f := range m.Findings {
if strings.TrimSpace(f.File) == "" {
continue
}
byFile[f.File] = append(byFile[f.File], findingRef{f, m.Model})
}
}
var clusters []cluster
for file, items := range byFile {
sort.SliceStable(items, func(i, j int) bool { return items[i].f.Line < items[j].f.Line })
// Cluster within THIS file only (clusters never span files), so the inner
// scan is over same-file clusters, not every cluster seen so far.
var fileClusters []cluster
for _, it := range items {
placed := false
for ci := range fileClusters {
c := &fileClusters[ci]
// Join if the line falls within the cluster's span, widened by the
// tolerance on both edges — so the window slides as the span grows.
if it.f.Line >= c.line-lineTolerance && it.f.Line <= c.maxLine+lineTolerance {
mergeIntoCluster(c, it.f, it.model)
placed = true
break
}
}
if !placed {
c := cluster{
file: file,
line: it.f.Line,
maxLine: it.f.Line,
severity: it.f.Severity,
title: it.f.Title,
detail: it.f.Detail,
models: map[string]bool{},
lenses: map[string]bool{},
}
mergeIntoCluster(&c, it.f, it.model)
fileClusters = append(fileClusters, c)
}
}
clusters = append(clusters, fileClusters...)
}
sort.SliceStable(clusters, func(i, j int) bool {
if len(clusters[i].models) != len(clusters[j].models) {
return len(clusters[i].models) > len(clusters[j].models)
}
if sevRank(clusters[i].severity) != sevRank(clusters[j].severity) {
return sevRank(clusters[i].severity) > sevRank(clusters[j].severity)
}
if clusters[i].file != clusters[j].file {
return clusters[i].file < clusters[j].file
}
return clusters[i].line < clusters[j].line
})
return clusters
}
// mergeIntoCluster folds one finding into a cluster: union the model/lens sets,
// widen the [line,maxLine] span, and keep the highest-severity report's title.
func mergeIntoCluster(c *cluster, f outFinding, model string) {
if model != "" {
c.models[model] = true
}
if f.Lens != "" {
c.lenses[f.Lens] = true
}
if f.Line > 0 && (c.line == 0 || f.Line < c.line) {
c.line = f.Line
}
if f.Line > c.maxLine {
c.maxLine = f.Line
}
// Backfill an empty title/detail from any report, regardless of severity, so a
// higher-severity-but-terse finding doesn't leave the cluster without context.
if strings.TrimSpace(c.title) == "" && strings.TrimSpace(f.Title) != "" {
c.title = f.Title
}
if strings.TrimSpace(c.detail) == "" && strings.TrimSpace(f.Detail) != "" {
c.detail = f.Detail
}
// A strictly-higher-severity report takes over the title/detail.
if sevRank(f.Severity) > sevRank(c.severity) {
c.severity = f.Severity
if strings.TrimSpace(f.Title) != "" {
c.title = f.Title
}
if strings.TrimSpace(f.Detail) != "" {
c.detail = f.Detail
}
}
}
// sevRank orders the canonical severity words for sorting/comparison.
func sevRank(s string) int {
switch strings.ToLower(strings.TrimSpace(s)) {
case "critical":
return 5
case "high":
return 4
case "medium":
return 3
case "small":
return 2
case "trivial":
return 1
default:
return 0
}
}
// sevIcon is the at-a-glance severity badge for the consensus table.
func sevIcon(s string) string {
switch strings.ToLower(strings.TrimSpace(s)) {
case "critical", "high":
return "🔴"
case "medium":
return "🟠"
case "small":
return "🟡"
default:
return "⚪"
}
}
func location(file string, line int) string {
if line > 0 {
return fmt.Sprintf("%s:%d", file, line)
}
return file
}
func lensList(lenses map[string]bool) string {
out := make([]string, 0, len(lenses))
for l := range lenses {
out = append(out, l)
}
sort.Strings(out)
return strings.Join(out, ", ")
}
func oneModel(models map[string]bool) string {
for m := range models {
return m
}
return ""
}
// mdCell makes a string safe for a one-line markdown table cell: collapse
// newlines, escape pipes (which delimit columns), and neutralize backticks
// (a stray one would break an inline-code span — a backslash can't escape it
// inside code, so replace with an apostrophe). Inputs are model-influenced, so
// this keeps a malformed file path or title from breaking the table.
func mdCell(s string) string {
s = strings.ReplaceAll(s, "\n", " ")
s = strings.ReplaceAll(s, "|", "\\|")
s = strings.ReplaceAll(s, "`", "'")
return strings.TrimSpace(s)
}
func plural(n int) string {
if n == 1 {
return ""
}
return "s"
}
+217
View File
@@ -0,0 +1,217 @@
package main
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
)
func TestClusterFindingsAgreementAndTolerance(t *testing.T) {
models := []modelFindings{
{Model: "m1", Verdict: "Blocking issues found", Findings: []outFinding{
{Lens: "security", File: "a.go", Line: 10, Severity: "high", Title: "auth bypass"},
{Lens: "perf", File: "b.go", Line: 5, Severity: "trivial", Title: "tiny nit"},
}},
{Model: "m2", Verdict: "Minor issues", Findings: []outFinding{
{Lens: "security", File: "a.go", Line: 11, Severity: "critical", Title: "auth bypass (crit)"}, // within tolerance of a.go:10
}},
{Model: "m3", Verdict: "Minor issues", Findings: []outFinding{
{Lens: "correctness", File: "a.go", Line: 10, Severity: "medium", Title: "auth bypass"},
}},
}
clusters := clusterFindings(models)
if len(clusters) != 2 {
t.Fatalf("want 2 clusters (a.go:10±, b.go:5), got %d: %+v", len(clusters), clusters)
}
// First cluster (highest agreement) is the a.go auth one: 3 models, severity
// escalated to critical, representative line the smallest (10).
c := clusters[0]
if len(c.models) != 3 {
t.Errorf("want 3 models on the top cluster, got %d", len(c.models))
}
if c.severity != "critical" {
t.Errorf("want escalated severity critical, got %q", c.severity)
}
if c.line != 10 {
t.Errorf("want representative line 10, got %d", c.line)
}
if !c.lenses["security"] || !c.lenses["correctness"] {
t.Errorf("want union of lenses, got %v", c.lenses)
}
}
func TestRenderConsensusFoldsSingleModelNits(t *testing.T) {
models := []modelFindings{
{Model: "m1", Provider: "p", Verdict: "Blocking issues found", Markdown: "m1 detail", Findings: []outFinding{
{Lens: "security", File: "a.go", Line: 10, Severity: "high", Title: "auth bypass"},
{Lens: "perf", File: "b.go", Line: 5, Severity: "trivial", Title: "tiny nit"},
}},
{Model: "m2", Provider: "p", Verdict: "Minor issues", Markdown: "m2 detail", Findings: []outFinding{
{Lens: "security", File: "a.go", Line: 10, Severity: "high", Title: "auth bypass"},
}},
}
out := renderConsensus(models, clusterFindings(models))
// Headline table: the agreed finding with a 2/2 badge.
if !strings.Contains(out, "2/2") {
t.Errorf("expected a 2/2 agreement badge in headline:\n%s", out)
}
if !strings.Contains(out, "auth bypass") || !strings.Contains(out, "a.go:10") {
t.Errorf("headline missing the consensus finding:\n%s", out)
}
// The lone trivial finding is folded, not in the headline table.
if !strings.Contains(out, "single-model finding") {
t.Errorf("expected a folded single-model section:\n%s", out)
}
// Per-model detail is preserved (folded).
if !strings.Contains(out, "m1 detail") || !strings.Contains(out, "m2 detail") {
t.Errorf("per-model detail not preserved:\n%s", out)
}
}
func TestRenderConsensusHighSeverityLoneFindingStaysHeadline(t *testing.T) {
// A single model, single critical finding must still surface in the headline
// (not be folded as "low confidence").
models := []modelFindings{
{Model: "solo", Verdict: "Blocking issues found", Markdown: "x", Findings: []outFinding{
{Lens: "security", File: "a.go", Line: 1, Severity: "critical", Title: "rce"},
}},
}
out := renderConsensus(models, clusterFindings(models))
headline := out
if i := strings.Index(out, "single-model finding"); i >= 0 {
headline = out[:i]
}
if !strings.Contains(headline, "rce") {
t.Errorf("lone critical should be in the headline, not folded:\n%s", out)
}
}
func TestClusterSlidingWindowMergesChain(t *testing.T) {
// Findings at 10, 13, 16 (each 3 apart) from three models must merge into ONE
// cluster — the window slides with the span instead of anchoring at line 10.
models := []modelFindings{
{Model: "m1", Findings: []outFinding{{Lens: "x", File: "a.go", Line: 10, Severity: "medium", Title: "t"}}},
{Model: "m2", Findings: []outFinding{{Lens: "x", File: "a.go", Line: 13, Severity: "medium", Title: "t"}}},
{Model: "m3", Findings: []outFinding{{Lens: "x", File: "a.go", Line: 16, Severity: "medium", Title: "t"}}},
}
clusters := clusterFindings(models)
if len(clusters) != 1 {
t.Fatalf("chain 10/13/16 should merge into 1 cluster, got %d", len(clusters))
}
if len(clusters[0].models) != 3 {
t.Errorf("want 3 models in the merged cluster, got %d", len(clusters[0].models))
}
}
func TestRenderConsensusExcludesErroredFromDenominator(t *testing.T) {
models := []modelFindings{
{Model: "m1", Verdict: "Minor issues", Markdown: "a", Findings: []outFinding{
{Lens: "security", File: "a.go", Line: 9, Severity: "medium", Title: "leak"}}},
{Model: "m2", Verdict: "Minor issues", Markdown: "b", Findings: []outFinding{
{Lens: "security", File: "a.go", Line: 9, Severity: "medium", Title: "leak"}}},
{Model: "broken", Verdict: "reviewer failed", Errored: true, Markdown: "boom"},
}
out := renderConsensus(models, clusterFindings(models))
// Denominator is the 2 effective models, not 3; the failure is noted.
if !strings.Contains(out, "2/2") {
t.Errorf("errored model must be excluded from the denominator (want 2/2):\n%s", out)
}
if !strings.Contains(out, "1 failed") {
t.Errorf("expected a '1 failed' note:\n%s", out)
}
if !strings.Contains(out, "reviewer failed") {
t.Errorf("errored model should still appear (folded) as failed:\n%s", out)
}
}
func TestRenderConsensusLoneHighFolds(t *testing.T) {
// A single-model HIGH (not critical) folds — only consensus or a lone CRITICAL
// earns the headline, so a lone Blocking-lens finding doesn't reintroduce noise.
models := []modelFindings{
{Model: "solo", Verdict: "Blocking issues found", Markdown: "x", Findings: []outFinding{
{Lens: "security", File: "a.go", Line: 1, Severity: "high", Title: "maybe-bug"}}},
}
out := renderConsensus(models, clusterFindings(models))
head := out
if i := strings.Index(out, "single-model finding"); i >= 0 {
head = out[:i]
}
if strings.Contains(head, "maybe-bug") {
t.Errorf("a lone HIGH should fold, not headline:\n%s", out)
}
}
func TestWriteAndConsolidateRoundTrip(t *testing.T) {
dir := t.TempDir()
// Two model artifacts on disk.
write := func(name string, mf modelFindings) {
data, _ := json.Marshal(mf)
if err := os.WriteFile(filepath.Join(dir, name), data, 0o644); err != nil {
t.Fatal(err)
}
}
write("m1.json", modelFindings{Model: "m1", Provider: "ollama", Verdict: "Minor issues", Markdown: "md1",
Findings: []outFinding{{Lens: "security", File: "x.go", Line: 3, Severity: "medium", Title: "leak"}}})
write("m2.json", modelFindings{Model: "m2", Provider: "ollama", Verdict: "Minor issues", Markdown: "md2",
Findings: []outFinding{{Lens: "security", File: "x.go", Line: 3, Severity: "high", Title: "leak"}}})
// A junk file must be skipped, not crash consolidation.
if err := os.WriteFile(filepath.Join(dir, "notes.txt"), []byte("ignore me"), 0o644); err != nil {
t.Fatal(err)
}
t.Setenv("GADFLY_CONSOLIDATE_DIR", dir)
// runConsolidate prints to stdout; capture it.
out := captureStdout(t, func() {
if err := runConsolidate(); err != nil {
t.Fatalf("runConsolidate: %v", err)
}
})
if !strings.HasPrefix(strings.TrimSpace(out), consensusMarker) {
t.Errorf("consolidated output must lead with the marker:\n%s", out)
}
if !strings.Contains(out, "2/2") || !strings.Contains(out, "x.go:3") {
t.Errorf("expected the agreed x.go:3 finding at 2/2:\n%s", out)
}
}
func TestRunConsolidateEmptyDirErrors(t *testing.T) {
t.Setenv("GADFLY_CONSOLIDATE_DIR", t.TempDir())
if err := runConsolidate(); err == nil {
t.Error("want an error for an empty consolidate dir (entrypoint falls back)")
}
}
// captureStdout redirects os.Stdout for the duration of fn and returns what was
// written.
func captureStdout(t *testing.T, fn func()) string {
t.Helper()
orig := os.Stdout
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
os.Stdout = w
defer func() { os.Stdout = orig }() // restore even if fn panics
done := make(chan string)
go func() {
var sb strings.Builder
buf := make([]byte, 4096)
for {
n, err := r.Read(buf)
if n > 0 {
sb.Write(buf[:n])
}
if err != nil {
break
}
}
r.Close()
done <- sb.String()
}()
fn()
w.Close()
return <-done
}
+19 -1
View File
@@ -28,6 +28,22 @@ func (v verdict) label() string {
} }
} }
// severity maps a verdict to a canonical severity word (the same vocabulary as
// the structured findings: critical/high/medium/small/trivial). Used as the
// raw_severity for heuristic-scraped findings, which carry no per-finding
// severity of their own — so the telemetry store sees one consistent vocabulary
// instead of a mix of canonical words and full verdict phrases.
func (v verdict) severity() string {
switch v {
case verdictBlocking:
return "high"
case verdictMinor:
return "small"
default:
return "trivial"
}
}
// parseVerdict extracts a specialist's self-reported verdict from its output. // parseVerdict extracts a specialist's self-reported verdict from its output.
// The base prompt tells each lens to lead with one of the three phrases. // The base prompt tells each lens to lead with one of the three phrases.
func parseVerdict(out string) verdict { func parseVerdict(out string) verdict {
@@ -100,7 +116,9 @@ func renderConsolidated(results []specialistResult) string {
headline, len(results), strings.Join(specialistNames(results), ", ")) headline, len(results), strings.Join(specialistNames(results), ", "))
for _, r := range results { for _, r := range results {
body := strings.TrimSpace(r.out) // Strip the machine-readable ```gadfly-findings block — it's for tooling
// (telemetry/consolidation), not for human readers of the comment.
body := strings.TrimSpace(stripFindingsBlock(r.out))
if body == "" { if body == "" {
body = "_(no output)_" body = "_(no output)_"
} }
+20 -19
View File
@@ -10,11 +10,14 @@ package main
// the review markdown consumed by run.sh) and never changes the exit code. // the review markdown consumed by run.sh) and never changes the exit code.
// - It depends only on the Go stdlib (net/http). // - It depends only on the Go stdlib (net/http).
// //
// Findings are extracted heuristically from each lens's markdown: a "path:line" // Findings come PRIMARILY from each lens's machine-readable ```gadfly-findings
// reference (e.g. run/executor.go:166) anchors a finding, whose title is the // block (findings.go: exact file/line + per-finding severity/confidence). When a
// nearest preceding markdown heading / numbered item / bold lead-in (else the // model emits no parseable block, emit falls back to a heuristic prose scrape: a
// first sentence of the finding's paragraph). This is best-effort signal for the // "path:line" reference (e.g. run/executor.go:166) anchors a finding, whose title
// store to aggregate — not a structured contract the reviewer guarantees. // is the nearest preceding markdown heading / numbered item / bold lead-in (else
// the first sentence of the finding's paragraph). The scrape is best-effort
// signal for the store to aggregate, not a structured contract the reviewer
// guarantees.
import ( import (
"bytes" "bytes"
@@ -50,11 +53,16 @@ var pathLineRe = regexp.MustCompile(`([A-Za-z0-9_./-]+\.[A-Za-z0-9]+):(\d+)`)
var numberedRe = regexp.MustCompile(`^\d+[.)]\s+(.+)$`) var numberedRe = regexp.MustCompile(`^\d+[.)]\s+(.+)$`)
// finding is one extracted issue: where it points and a derived human title. // finding is one extracted issue: where it points and a derived human title.
// severity/confidence are populated from the structured ```gadfly-findings block
// (extractStructuredFindings); they are empty for findings recovered by the
// heuristic prose scrape (parseFindings), which has no per-finding signal.
type finding struct { type finding struct {
file string file string
line int line int
title string title string
detail string detail string
severity string // per-finding: critical/high/medium/small/trivial ("" if heuristic)
confidence string // per-finding: high/medium/low ("" if heuristic)
} }
// runPayload is the POST /runs body. Field names match the gadfly-reports API EXACTLY. // runPayload is the POST /runs body. Field names match the gadfly-reports API EXACTLY.
@@ -86,6 +94,7 @@ type reportPayload struct {
Provider string `json:"provider"` Provider string `json:"provider"`
RunID string `json:"run_id"` RunID string `json:"run_id"`
RawSeverity string `json:"raw_severity"` RawSeverity string `json:"raw_severity"`
Confidence string `json:"confidence"` // per-finding high/medium/low ("" if heuristic)
Detail string `json:"detail"` Detail string `json:"detail"`
} }
@@ -116,21 +125,12 @@ func emit(results []specialistResult, elapsed time.Duration) {
// InputTokens/OutputTokens/CostUSD stay nil -> JSON null (not metered). // InputTokens/OutputTokens/CostUSD stay nil -> JSON null (not metered).
} }
// collectFindings (consensus.go) skips clean/errored lenses and fills in each
// finding's severity (per-finding when structured, else derived from the lens
// verdict), so every reportPayload carries a canonical raw_severity.
var reports []reportPayload var reports []reportPayload
for _, r := range results { for _, r := range results {
if r.errored { for _, f := range collectFindings(r) {
continue // a failed lens contributes no findings
}
// A lens that reports "No material issues found" has nothing to flag —
// its path:line references are verification notes ("verified X at
// file:line is safe"), not problems. Extracting them pollutes the
// findings store with false positives and unfairly penalizes thorough
// reviewers that do clean passes, so a clean lens emits no findings.
if r.verdict == verdictClean {
continue
}
sev := r.verdict.label()
for _, f := range parseFindings(r.spec, r.out) {
reports = append(reports, reportPayload{ reports = append(reports, reportPayload{
Repo: repo, Repo: repo,
PR: pr, PR: pr,
@@ -141,7 +141,8 @@ func emit(results []specialistResult, elapsed time.Duration) {
Model: model, Model: model,
Provider: provider, Provider: provider,
RunID: runID, RunID: runID,
RawSeverity: sev, RawSeverity: f.severity,
Confidence: f.confidence,
Detail: f.detail, Detail: f.detail,
}) })
} }
+6 -3
View File
@@ -157,7 +157,7 @@ func TestEmit_PostsRunsAndReports(t *testing.T) {
if len(reportBody) != 2 { if len(reportBody) != 2 {
t.Fatalf("/reports array length = %d, want 2", len(reportBody)) t.Fatalf("/reports array length = %d, want 2", len(reportBody))
} }
for _, k := range []string{"repo", "pr", "lens", "file", "line", "title", "model", "provider", "run_id", "raw_severity", "detail"} { for _, k := range []string{"repo", "pr", "lens", "file", "line", "title", "model", "provider", "run_id", "raw_severity", "confidence", "detail"} {
if _, ok := reportBody[0][k]; !ok { if _, ok := reportBody[0][k]; !ok {
t.Errorf("/reports[0] missing field %q (got keys %v)", k, keysOf(reportBody[0])) t.Errorf("/reports[0] missing field %q (got keys %v)", k, keysOf(reportBody[0]))
} }
@@ -171,8 +171,11 @@ func TestEmit_PostsRunsAndReports(t *testing.T) {
if reportBody[0]["line"] != float64(166) { if reportBody[0]["line"] != float64(166) {
t.Errorf("reports[0].line = %v, want 166", reportBody[0]["line"]) t.Errorf("reports[0].line = %v, want 166", reportBody[0]["line"])
} }
if reportBody[0]["raw_severity"] != "Blocking issues found" { // No structured block in sampleLensMarkdown, so this is a heuristic-scraped
t.Errorf("reports[0].raw_severity = %v, want 'Blocking issues found'", reportBody[0]["raw_severity"]) // finding: raw_severity is the canonical word derived from the lens verdict
// (Blocking -> "high"), not the full verdict phrase.
if reportBody[0]["raw_severity"] != "high" {
t.Errorf("reports[0].raw_severity = %v, want 'high'", reportBody[0]["raw_severity"])
} }
if reportBody[0]["run_id"] != "owner/repo#7:ollama-cloud/qwen3" { if reportBody[0]["run_id"] != "owner/repo#7:ollama-cloud/qwen3" {
t.Errorf("reports[0].run_id = %v, want owner/repo#7:ollama-cloud/qwen3", reportBody[0]["run_id"]) t.Errorf("reports[0].run_id = %v, want owner/repo#7:ollama-cloud/qwen3", reportBody[0]["run_id"])
+236
View File
@@ -0,0 +1,236 @@
package main
// Structured findings: the machine-readable contract between a lens's output and
// the telemetry/consolidation pipeline. Each lens is asked (see
// scripts/system-prompt.txt) to append a fenced ```gadfly-findings code block
// holding a JSON array of its findings. Parsing that exact block is far more
// reliable than scraping prose with a path:line regex (emit.go's heuristic),
// and it carries PER-FINDING severity + confidence the prose verdict can't.
//
// Everything here degrades gracefully: a missing, unterminated, or malformed
// block makes extractStructuredFindings return ok=false (and yield no findings),
// so the caller falls back to the heuristic scrape — a weak model that ignores
// the contract still contributes findings, exactly as before.
import (
"encoding/json"
"strconv"
"strings"
)
// structuredFinding mirrors one element of the ```gadfly-findings JSON array.
// Line is json.Number so we tolerate both 123 and "123" from less-precise models.
type structuredFinding struct {
File string `json:"file"`
Line json.Number `json:"line"`
Severity string `json:"severity"`
Confidence string `json:"confidence"`
Title string `json:"title"`
Detail string `json:"detail"` // optional; the prose paragraph is used when absent
}
// findingsFence is the info-string that tags the machine-readable block.
const findingsFence = "gadfly-findings"
// extractStructuredFindings parses the ```gadfly-findings JSON block out of a
// lens's markdown. It returns the findings and ok=true when a TERMINATED block is
// present AND parses as a JSON array (an empty array is valid "nothing found" —
// ok is still true). A missing, unterminated, or unparseable block returns
// ok=false so the caller falls back to the heuristic scrape.
//
// Findings are deduped by file:line (keeping the first, matching parseFindings),
// findings with no usable file are dropped, and each title/detail is backfilled
// from the prose when the JSON omits it (best of both: exact location + the human
// context the model already wrote).
func extractStructuredFindings(out string) ([]finding, bool) {
lines := strings.Split(out, "\n")
start, end, ok := findingsSpan(lines)
if !ok {
return nil, false
}
var raw []structuredFinding
if err := json.Unmarshal([]byte(strings.Join(lines[start+1:end], "\n")), &raw); err != nil {
return nil, false
}
findings := make([]finding, 0, len(raw))
seen := map[string]bool{}
var prose map[string]string // built lazily — only if a finding lacks its own detail
for _, sf := range raw {
file := strings.TrimSpace(sf.File)
if file == "" {
continue
}
ln := 0
if n, err := strconv.Atoi(strings.TrimSpace(sf.Line.String())); err == nil && n > 0 {
ln = n
}
key := file + ":" + strconv.Itoa(ln)
if ln > 0 { // only dedupe concrete locations; unknown-line findings are kept
if seen[key] {
continue
}
seen[key] = true
}
detail := strings.TrimSpace(sf.Detail)
if detail == "" && ln > 0 {
if prose == nil {
prose = proseParagraphs(out)
}
detail = prose[key]
}
title := strings.TrimSpace(sf.Title)
if title == "" { // never empty: fall back to the prose detail, then the location
if detail != "" {
title = truncate(detail, 120)
} else if ln > 0 {
title = key
} else {
title = file
}
}
findings = append(findings, finding{
file: file,
line: ln,
title: title,
detail: truncate(detail, 500),
severity: normalizeSeverity(sf.Severity),
confidence: normalizeConfidence(sf.Confidence),
})
if len(findings) >= maxFindingsPerLens {
break
}
}
return findings, true
}
// extractStructuredFindingsOrScrape returns a lens's findings, preferring the
// structured ```gadfly-findings block and falling back to the heuristic prose
// scrape when the block is absent, unterminated/malformed, OR parsed to zero
// usable findings (e.g. an empty [] emitted alongside real prose findings).
// Factored out of emit() so the fallback rule is unit-testable.
func extractStructuredFindingsOrScrape(r specialistResult) []finding {
if fs, _ := extractStructuredFindings(r.out); len(fs) > 0 {
return fs
}
return parseFindings(r.spec, r.out)
}
// stripFindingsBlock removes every TERMINATED ```gadfly-findings block from out so
// the machine-readable JSON never shows in the rendered comment. An UNTERMINATED
// fence is left in place — treating it as a block would swallow the rest of the
// comment (e.g. when a model's output was truncated mid-block). Trailing
// whitespace is trimmed.
func stripFindingsBlock(out string) string {
lines := strings.Split(out, "\n")
for {
start, end, ok := findingsSpan(lines)
if !ok {
break
}
lines = append(lines[:start:start], lines[end+1:]...)
}
return strings.TrimRight(strings.Join(lines, "\n"), "\n")
}
// findingsSpan returns the [start,end] inclusive line indices of the first
// TERMINATED ```gadfly-findings block in lines (start = the opening fence, end =
// the closing fence), or ok=false when there is none or it is unterminated.
// extract and strip share it so they always agree on what is (and isn't) a block.
func findingsSpan(lines []string) (start, end int, ok bool) {
start = -1
for i, ln := range lines {
if isFindingsOpen(ln) {
start = i
break
}
}
if start < 0 {
return 0, 0, false
}
for j := start + 1; j < len(lines); j++ {
if isFenceClose(lines[j]) {
return start, j, true
}
}
return 0, 0, false // unterminated
}
// fenceInfo returns the info-string (text after the backticks) of a code-fence
// line and whether the line opens/closes a fence at all. A bare ``` yields ("",
// true); ```gadfly-findings yields ("gadfly-findings", true).
func fenceInfo(line string) (string, bool) {
t := strings.TrimSpace(line)
if !strings.HasPrefix(t, "```") {
return "", false
}
return strings.TrimSpace(strings.TrimLeft(t, "`")), true
}
// isFindingsOpen reports whether line opens a ```gadfly-findings block, matching
// the info-string EXACTLY (not as a substring) so a fence like ```not-findings
// can't masquerade as ours.
func isFindingsOpen(line string) bool {
info, ok := fenceInfo(line)
return ok && info == findingsFence
}
// isFenceClose reports whether line is a bare closing fence (``` with no info).
func isFenceClose(line string) bool {
info, ok := fenceInfo(line)
return ok && info == ""
}
// proseParagraphs maps "file:line" -> the prose paragraph that first references
// it, so a structured finding without its own detail can borrow the human
// context the model already wrote. Built from the markdown OUTSIDE the findings
// block (the block is JSON, not prose).
func proseParagraphs(out string) map[string]string {
prose := stripFindingsBlock(out)
lines := strings.Split(prose, "\n")
m := map[string]string{}
for _, loc := range pathLineRe.FindAllStringSubmatchIndex(prose, -1) {
key := prose[loc[2]:loc[3]] + ":" + prose[loc[4]:loc[5]]
if _, dup := m[key]; dup {
continue
}
li := strings.Count(prose[:loc[0]], "\n")
m[key] = paragraphAt(lines, li)
}
return m
}
// normalizeSeverity maps a model's severity word onto the canonical set
// (critical/high/medium/small/trivial), accepting common synonyms. An
// unrecognized value is returned lowercased so the store still sees the raw word.
func normalizeSeverity(s string) string {
switch t := strings.ToLower(strings.TrimSpace(s)); t {
case "critical", "crit", "blocker", "blocking":
return "critical"
case "high", "major", "severe":
return "high"
case "medium", "moderate":
return "medium"
case "small", "low", "minor":
return "small"
case "trivial", "nit", "nitpick", "info", "informational", "style", "cosmetic":
return "trivial"
default:
return t
}
}
// normalizeConfidence maps a model's confidence word onto high/medium/low,
// accepting common synonyms; an unrecognized value is returned lowercased.
func normalizeConfidence(s string) string {
switch t := strings.ToLower(strings.TrimSpace(s)); t {
case "high", "certain", "confirmed", "verified":
return "high"
case "medium", "med", "moderate":
return "medium"
case "low", "unsure", "tentative", "unverified", "speculative":
return "low"
default:
return t
}
}
+146
View File
@@ -0,0 +1,146 @@
package main
import (
"strings"
"testing"
)
const sampleReview = "**Blocking issues found**\n\n" +
"- **Unauthenticated endpoint** — `model.go:184` leaks PR content to a third party.\n" +
"- A nit about naming in `util.go:12`.\n\n" +
"```gadfly-findings\n" +
"[\n" +
" {\"file\": \"model.go\", \"line\": 184, \"severity\": \"high\", \"confidence\": \"high\", \"title\": \"Unauthenticated endpoint\"},\n" +
" {\"file\": \"util.go\", \"line\": 12, \"severity\": \"nit\", \"confidence\": \"medium\", \"title\": \"naming\"}\n" +
"]\n" +
"```\n"
func TestExtractStructuredFindings(t *testing.T) {
fs, ok := extractStructuredFindings(sampleReview)
if !ok {
t.Fatal("expected ok=true for a well-formed block")
}
if len(fs) != 2 {
t.Fatalf("want 2 findings, got %d", len(fs))
}
if fs[0].file != "model.go" || fs[0].line != 184 || fs[0].severity != "high" || fs[0].confidence != "high" {
t.Errorf("finding[0] mismatch: %+v", fs[0])
}
// "nit" must normalize to the canonical "trivial".
if fs[1].severity != "trivial" {
t.Errorf("want severity normalized to trivial, got %q", fs[1].severity)
}
// Detail is borrowed from the prose paragraph referencing the same file:line.
if fs[0].detail == "" {
t.Error("expected detail borrowed from prose, got empty")
}
}
func TestExtractStructuredFindingsEmptyArray(t *testing.T) {
out := "No material issues found\n\n```gadfly-findings\n[]\n```\n"
fs, ok := extractStructuredFindings(out)
if !ok {
t.Fatal("an empty array is a valid block; want ok=true")
}
if len(fs) != 0 {
t.Fatalf("want 0 findings, got %d", len(fs))
}
}
func TestExtractStructuredFindingsFallback(t *testing.T) {
// No block at all -> ok=false so the caller uses the heuristic scrape.
if _, ok := extractStructuredFindings("Minor issues\n\n- something at `x.go:1`\n"); ok {
t.Error("want ok=false when there is no block")
}
// Malformed JSON -> ok=false (graceful fallback).
bad := "Minor issues\n\n```gadfly-findings\n{not json}\n```\n"
if _, ok := extractStructuredFindings(bad); ok {
t.Error("want ok=false for malformed JSON")
}
}
func TestExtractStructuredFindingsStringLine(t *testing.T) {
// Tolerate a quoted line number from a less-precise model.
out := "Minor issues\n\n```gadfly-findings\n[{\"file\":\"a.go\",\"line\":\"42\",\"severity\":\"medium\",\"title\":\"x\"}]\n```\n"
fs, ok := extractStructuredFindings(out)
if !ok || len(fs) != 1 || fs[0].line != 42 {
t.Fatalf("want one finding at line 42, got ok=%v %+v", ok, fs)
}
}
func TestStripFindingsBlock(t *testing.T) {
stripped := stripFindingsBlock(sampleReview)
if strings.Contains(stripped, findingsFence) {
t.Errorf("block not stripped: %q", stripped)
}
// The prose findings must survive.
if !strings.Contains(stripped, "Unauthenticated endpoint") || !strings.Contains(stripped, "util.go:12") {
t.Errorf("prose lost during strip: %q", stripped)
}
}
func TestStripFindingsBlockUnterminated(t *testing.T) {
// A truncated, unterminated block must NOT swallow the prose before it.
out := "Minor issues\n\n- real finding at `x.go:1`\n\n```gadfly-findings\n[{\"file\":\"x.go\""
got := stripFindingsBlock(out)
if !strings.Contains(got, "real finding at `x.go:1`") {
t.Errorf("unterminated block swallowed the prose: %q", got)
}
}
func TestStripFindingsBlockNoBlock(t *testing.T) {
in := "Minor issues\n\n- finding at `x.go:9`"
if out := stripFindingsBlock(in); out != in {
t.Errorf("strip changed block-free text: %q != %q", out, in)
}
}
func TestNormalizeSeverity(t *testing.T) {
cases := map[string]string{
"Critical": "critical", "blocker": "critical",
"major": "high", "HIGH": "high",
"moderate": "medium",
"minor": "small", "low": "small", // "minor" and "low" both map to small (consistently)
"nit": "trivial", "Style": "trivial",
"weird": "weird", // unknown passes through, lowercased
}
for in, want := range cases {
if got := normalizeSeverity(in); got != want {
t.Errorf("normalizeSeverity(%q) = %q, want %q", in, got, want)
}
}
}
func TestNormalizeConfidence(t *testing.T) {
cases := map[string]string{
"High": "high", "confirmed": "high",
"MEDIUM": "medium", "moderate": "medium",
"low": "low", "unverified": "low",
"hunch": "hunch", // unknown passes through, lowercased
}
for in, want := range cases {
if got := normalizeConfidence(in); got != want {
t.Errorf("normalizeConfidence(%q) = %q, want %q", in, got, want)
}
}
}
func TestExtractStructuredFindingsFallbackOnEmpty(t *testing.T) {
// A non-clean lens that emitted an empty [] but listed prose findings must
// fall through to the heuristic scrape, not silently drop everything.
out := "Minor issues\n\n- bug at `pkg/a.go:7`\n\n```gadfly-findings\n[]\n```\n"
r := specialistResult{spec: Specialist{Name: "correctness"}, out: out, verdict: verdictMinor}
fs := extractStructuredFindingsOrScrape(r)
if len(fs) == 0 {
t.Fatal("empty [] must fall back to the heuristic scrape, got no findings")
}
if fs[0].file != "pkg/a.go" || fs[0].line != 7 {
t.Errorf("heuristic fallback wrong: %+v", fs[0])
}
}
func TestVerdictSeverity(t *testing.T) {
if verdictBlocking.severity() != "high" || verdictMinor.severity() != "small" || verdictUnknown.severity() != "trivial" {
t.Error("verdict.severity mapping changed unexpectedly")
}
}
+13
View File
@@ -123,6 +123,14 @@ func main() {
func run() error { func run() error {
start := time.Now() start := time.Now()
// Consolidation mode: not a review at all — read the per-model findings the
// swarm wrote and print the single cross-model consensus comment. entrypoint.sh
// runs this once, after every model has finished.
if strings.TrimSpace(os.Getenv("GADFLY_CONSOLIDATE_DIR")) != "" {
return runConsolidate()
}
repoDir := os.Getenv("GADFLY_REPO_DIR") repoDir := os.Getenv("GADFLY_REPO_DIR")
diffFile := os.Getenv("GADFLY_DIFF_FILE") diffFile := os.Getenv("GADFLY_DIFF_FILE")
systemFile := os.Getenv("GADFLY_SYSTEM_FILE") systemFile := os.Getenv("GADFLY_SYSTEM_FILE")
@@ -214,6 +222,11 @@ func run() error {
// Optional, best-effort telemetry. OFF unless GADFLY_FINDINGS_URL is set; // Optional, best-effort telemetry. OFF unless GADFLY_FINDINGS_URL is set;
// any failure is logged to stderr and never affects stdout or the exit code. // any failure is logged to stderr and never affects stdout or the exit code.
emit(results, time.Since(start)) emit(results, time.Since(start))
// Optional per-model findings artifact for the cross-model consolidation
// pass. No-op unless GADFLY_FINDINGS_OUT is set (entrypoint sets it for a
// multi-model swarm). Best-effort, never affects stdout or the exit code.
writeFindingsOut(results)
return nil return nil
} }
+5
View File
@@ -49,6 +49,11 @@ Output rules:
- Do NOT invent new findings; this is a verification gate, not a fresh review. - Do NOT invent new findings; this is a verification gate, not a fresh review.
- Do NOT include meta-commentary about the verification process or which - Do NOT include meta-commentary about the verification process or which
findings you dropped — output only the final, corrected review markdown. findings you dropped — output only the final, corrected review markdown.
- The draft ends with a fenced ` + "`gadfly-findings`" + ` JSON block. Regenerate it
so it lists ONLY the findings that SURVIVED your verification, in the same schema
({"file","line","severity","confidence","title"}; severity one of
critical/high/medium/small/trivial, confidence one of high/medium/low). If every
finding was dropped, emit an empty array ` + "`[]`" + `. Keep the block last.
- When done investigating, STOP calling tools and reply with the review.` - When done investigating, STOP calling tools and reply with the review.`
// recheckEnabled reports whether the verification pass should run. On unless // recheckEnabled reports whether the verification pass should run. On unless
+296
View File
@@ -0,0 +1,296 @@
package main
// Inline PR review. After the consensus comment is rendered, Gadfly also posts a
// single Gitea pull review (state COMMENT — advisory, NEVER request-changes or
// approve) whose inline comments anchor consensus findings to the exact changed
// lines. The issue comment stays the ranked overview; the review puts each
// finding next to the code it's about — the "reviewer integrated with Gitea" the
// project wanted, without ever blocking a merge.
//
// All of this is best-effort: disabled by GADFLY_INLINE_REVIEW=0 or when the diff
// / API creds aren't available, only anchors findings that land on a line present
// in the diff (Gitea rejects comments off the diff), and any error is logged to
// stderr without touching the consensus comment (already on stdout) or the exit
// code.
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"regexp"
"strconv"
"strings"
"time"
)
const (
// inlineReviewMarker tags our review body so a re-run can delete the previous
// one instead of stacking duplicate inline comments.
inlineReviewMarker = "<!-- gadfly-inline-review -->"
// maxInlineComments caps how many inline comments one review carries, so a
// huge diff can't produce a wall of annotations. Clusters are pre-sorted by
// agreement×severity, so the cap keeps the most important.
maxInlineComments = 25
inlineReviewHTTPTimeout = 20 * time.Second
)
// reviewComment is one inline comment in a Gitea pull review. Field names match
// the Gitea API EXACTLY (new_position = line in the new/head file).
type reviewComment struct {
Path string `json:"path"`
Body string `json:"body"`
NewPosition int `json:"new_position"`
}
// createReview is the POST /pulls/{n}/reviews body. event is ALWAYS "COMMENT".
type createReview struct {
Body string `json:"body"`
Event string `json:"event"`
Comments []reviewComment `json:"comments"`
}
// postInlineReview posts one COMMENT-state pull review with inline comments for
// consensus findings on changed lines. No-op + best-effort (see file comment).
func postInlineReview(clusters []cluster) {
if strings.EqualFold(strings.TrimSpace(os.Getenv("GADFLY_INLINE_REVIEW")), "0") {
return
}
api := strings.TrimRight(strings.TrimSpace(os.Getenv("GITEA_API")), "/")
token := strings.TrimSpace(os.Getenv("GITEA_TOKEN"))
pr := strings.TrimSpace(os.Getenv("GADFLY_PR"))
diffPath := strings.TrimSpace(os.Getenv("GADFLY_DIFF_FILE"))
if api == "" || token == "" || pr == "" || diffPath == "" {
return
}
diff, err := os.ReadFile(diffPath)
if err != nil {
fmt.Fprintln(os.Stderr, "gadfly: inline review: read diff:", err)
return
}
comments := inlineComments(clusters, parseDiffNewLines(string(diff)))
if len(comments) == 0 {
return // nothing anchors to a changed line; the consensus comment covers it
}
client := &http.Client{Timeout: inlineReviewHTTPTimeout}
base := fmt.Sprintf("%s/pulls/%s/reviews", api, pr)
deletePriorReviews(client, base, token) // avoid stacking on re-runs
body := fmt.Sprintf("%s\n🪰 **Gadfly consensus review** — %d inline finding%s on changed lines. See the consensus comment for the full ranked summary.\n\n<sub>Advisory only — does not block merge.</sub>",
inlineReviewMarker, len(comments), plural(len(comments)))
if err := giteaSend(client, http.MethodPost, base, token, createReview{Body: body, Event: "COMMENT", Comments: comments}); err != nil {
fmt.Fprintln(os.Stderr, "gadfly: inline review post:", err)
}
}
// inlineComments builds inline comments for the clusters that anchor to a line
// present in the diff, in priority order (clusters are pre-sorted), capped.
func inlineComments(clusters []cluster, addable map[string]map[int]bool) []reviewComment {
var out []reviewComment
for _, c := range clusters {
path := normPath(c.file)
anchor := anchorLine(addable[path], c.line, c.maxLine)
if anchor == 0 {
continue
}
out = append(out, reviewComment{Path: path, NewPosition: anchor, Body: inlineBody(c)})
if len(out) >= maxInlineComments {
break
}
}
return out
}
// anchorLine returns the first line in [lo,hi] that is an added line in the diff,
// or 0 if none. Scanning the cluster's whole span (not just its representative
// line) anchors a finding whose min line is just outside the diff but whose span
// still overlaps a changed line.
func anchorLine(added map[int]bool, lo, hi int) int {
if added == nil || lo <= 0 {
return 0
}
if hi < lo { // single-line cluster (maxLine unset)
hi = lo
}
for ln := lo; ln <= hi; ln++ {
if added[ln] {
return ln
}
}
return 0
}
// inlineBody renders one inline comment: severity + title, who flagged it, detail.
func inlineBody(c cluster) string {
var b strings.Builder
fmt.Fprintf(&b, "%s **%s**", sevIcon(c.severity), strings.TrimSpace(c.title))
fmt.Fprintf(&b, "\n\n_%s · flagged by %d model%s_", lensList(c.lenses), len(c.models), plural(len(c.models)))
if d := strings.TrimSpace(c.detail); d != "" {
fmt.Fprintf(&b, "\n\n%s", d)
}
b.WriteString("\n\n<sub>🪰 Gadfly · advisory</sub>")
return b.String()
}
// parseDiffNewLines returns, per file, the set of NEW-file line numbers that were
// ADDED in the unified diff — the safest lines for an inline comment to anchor to
// (Gitea reliably accepts comments on added lines). Context lines are walked to
// keep the line counter correct but are NOT recorded: anchoring only to added
// lines avoids the all-or-nothing review POST being rejected for an off-change
// anchor. Hunk lengths from the @@ header bound each hunk, so a content line that
// happens to start with "+++ " or "@@" is still read as content, not a header.
func parseDiffNewLines(diff string) map[string]map[int]bool {
out := map[string]map[int]bool{}
var file string
var newLine, oldRem, newRem int
inHunk := false
for _, line := range strings.Split(diff, "\n") {
if inHunk && (newRem > 0 || oldRem > 0) {
switch {
case strings.HasPrefix(line, "+"):
record(out, file, newLine) // added line — anchorable
newLine++
newRem--
case strings.HasPrefix(line, "-"):
oldRem--
case strings.HasPrefix(line, "\\"): // "\ No newline at end of file"
default: // context line (leading space, or an empty line): advance, don't record
newLine++
newRem--
oldRem--
}
if newRem <= 0 && oldRem <= 0 {
inHunk = false
}
continue
}
switch {
case strings.HasPrefix(line, "+++ "):
file = normPath(strings.TrimPrefix(line, "+++ "))
if file == "/dev/null" {
file = ""
}
case strings.HasPrefix(line, "@@"):
if m := hunkRe.FindStringSubmatch(line); m != nil && file != "" {
newLine, _ = strconv.Atoi(m[3])
oldRem = atoiOr(m[2], 1)
newRem = atoiOr(m[4], 1)
inHunk = newRem > 0 || oldRem > 0
}
}
}
return out
}
// hunkRe captures a unified-diff hunk header's old/new start+length:
// @@ -<oldStart>[,<oldLen>] +<newStart>[,<newLen>] @@
var hunkRe = regexp.MustCompile(`^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@`)
// normPath trims a unified-diff path down to a repo-relative one: strip a single
// leading "a/" or "b/" prefix (and any "./"), and surrounding whitespace. Applied
// to BOTH the diff paths and a finding's file so they match even when a model
// writes "./pkg/x.go" or the diff carries the "b/" prefix.
func normPath(p string) string {
p = strings.TrimSpace(p)
p = strings.TrimPrefix(p, "./")
if strings.HasPrefix(p, "a/") || strings.HasPrefix(p, "b/") {
p = p[2:]
}
return p
}
func record(out map[string]map[int]bool, file string, line int) {
if file == "" {
return
}
if out[file] == nil {
out[file] = map[int]bool{}
}
out[file][line] = true
}
// atoiOr parses s, returning def when s is empty or unparseable. Used for the
// optional hunk-length fields (absent => length 1).
func atoiOr(s string, def int) int {
if s == "" {
return def
}
if n, err := strconv.Atoi(s); err == nil {
return n
}
return def
}
// deletePriorReviews removes our previous inline reviews (matched by the body
// marker) so a re-run replaces rather than stacks. Best-effort and quiet.
func deletePriorReviews(client *http.Client, base, token string) {
const perPage = 50
for page := 1; page <= 10; page++ { // bound the scan, but page past 50 so a stale marked review isn't missed
req, err := http.NewRequest(http.MethodGet, fmt.Sprintf("%s?limit=%d&page=%d", base, perPage, page), nil)
if err != nil {
return
}
req.Header.Set("Authorization", "token "+token)
resp, err := client.Do(req)
if err != nil {
return
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
io.Copy(io.Discard, resp.Body)
resp.Body.Close()
return
}
var reviews []struct {
ID int `json:"id"`
Body string `json:"body"`
}
_ = json.NewDecoder(resp.Body).Decode(&reviews)
resp.Body.Close()
for _, r := range reviews {
if !strings.Contains(r.Body, inlineReviewMarker) {
continue
}
dreq, err := http.NewRequest(http.MethodDelete, fmt.Sprintf("%s/%d", base, r.ID), nil)
if err != nil {
continue
}
dreq.Header.Set("Authorization", "token "+token)
if dresp, err := client.Do(dreq); err == nil {
io.Copy(io.Discard, dresp.Body)
dresp.Body.Close()
}
}
if len(reviews) < perPage {
return // last page
}
}
}
// giteaSend marshals payload and sends it with Gitea's "token" auth scheme,
// treating a non-2xx response as an error (with a snippet of the body).
func giteaSend(client *http.Client, method, url, token string, payload any) error {
body, err := json.Marshal(payload)
if err != nil {
return err
}
req, err := http.NewRequest(method, url, bytes.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "token "+token)
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
snippet, _ := io.ReadAll(io.LimitReader(resp.Body, 2048))
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("%s %s: status %d: %s", method, url, resp.StatusCode, strings.TrimSpace(string(snippet)))
}
return nil
}
+107
View File
@@ -0,0 +1,107 @@
package main
import (
"strings"
"testing"
)
func TestParseDiffNewLines(t *testing.T) {
diff := "diff --git a/a.go b/a.go\n" +
"index 111..222 100644\n" +
"--- a/a.go\n" +
"+++ b/a.go\n" +
"@@ -1,3 +1,4 @@\n" +
" line1\n" +
"-old2\n" +
"+new2\n" +
"+new3\n" +
" line4\n"
got := parseDiffNewLines(diff)
// Only ADDED lines anchor: new2 (line 2) and new3 (line 3). Context lines 1
// and 4 are walked for counting but not recorded.
want := map[int]bool{2: true, 3: true}
if len(got["a.go"]) != len(want) {
t.Fatalf("a.go lines = %v, want %v (added-only)", got["a.go"], want)
}
for ln := range want {
if !got["a.go"][ln] {
t.Errorf("expected a.go line %d anchorable", ln)
}
}
if got["a.go"][1] || got["a.go"][4] {
t.Error("context lines 1/4 should not be anchorable (added-only)")
}
}
func TestParseDiffNewLinesContentLooksLikeHeader(t *testing.T) {
// An added line whose CONTENT is "++ weird" appears as "+++ weird" in the
// diff. Hunk-length tracking must read it as content, not a file header.
diff := "--- a/x\n+++ b/x\n@@ -1,1 +1,2 @@\n ctx\n+++ weird\n"
got := parseDiffNewLines(diff)
if got["x"][1] { // context line, not recorded (added-only)
t.Errorf("line 1 is context, should not anchor: %v", got["x"])
}
if !got["x"][2] { // the "+++ weird" added line
t.Errorf("want added line 2 anchorable, got %v", got["x"])
}
}
func TestAnchorLineScansSpan(t *testing.T) {
// A cluster spanning 10..14 whose min line (10) isn't in the diff but whose
// span includes added line 14 must anchor to 14, not be dropped.
added := map[string]map[int]bool{"a.go": {14: true}}
clusters := []cluster{{file: "a.go", line: 10, maxLine: 14, severity: "high", title: "t", models: set("m1"), lenses: set("x")}}
cs := inlineComments(clusters, added)
if len(cs) != 1 || cs[0].NewPosition != 14 {
t.Fatalf("want anchor at 14 via span scan, got %+v", cs)
}
}
func TestParseDiffNewLinesMultiFile(t *testing.T) {
diff := "diff --git a/one.go b/one.go\n--- a/one.go\n+++ b/one.go\n@@ -0,0 +1,2 @@\n+a\n+b\n" +
"diff --git a/two.go b/two.go\n--- a/two.go\n+++ b/two.go\n@@ -5,0 +6,1 @@\n+c\n"
got := parseDiffNewLines(diff)
if !got["one.go"][1] || !got["one.go"][2] {
t.Errorf("one.go: want 1,2 got %v", got["one.go"])
}
if !got["two.go"][6] {
t.Errorf("two.go: want 6 got %v", got["two.go"])
}
}
func TestInlineCommentsNormalizesPaths(t *testing.T) {
// Diff path "b/pkg/x.go" -> "pkg/x.go"; a finding written as "./pkg/x.go" must
// still anchor (both normalize to the same repo-relative path).
addable := parseDiffNewLines("--- a/pkg/x.go\n+++ b/pkg/x.go\n@@ -0,0 +1,1 @@\n+bug\n")
clusters := []cluster{{file: "./pkg/x.go", line: 1, severity: "high", title: "t", models: set("m1"), lenses: set("x")}}
if cs := inlineComments(clusters, addable); len(cs) != 1 || cs[0].Path != "pkg/x.go" {
t.Errorf("path normalization failed to anchor: %+v", cs)
}
}
func TestInlineCommentsFiltersToDiffLines(t *testing.T) {
addable := map[string]map[int]bool{"a.go": {10: true, 11: true}}
clusters := []cluster{
{file: "a.go", line: 10, severity: "high", title: "anchored", models: set("m1", "m2"), lenses: set("security"), detail: "d"},
{file: "a.go", line: 99, severity: "high", title: "off-diff line", models: set("m1"), lenses: set("security")},
{file: "b.go", line: 1, severity: "high", title: "off-diff file", models: set("m1"), lenses: set("security")},
}
cs := inlineComments(clusters, addable)
if len(cs) != 1 {
t.Fatalf("want 1 anchorable inline comment, got %d", len(cs))
}
if cs[0].Path != "a.go" || cs[0].NewPosition != 10 {
t.Errorf("wrong anchor: %+v", cs[0])
}
if !strings.Contains(cs[0].Body, "anchored") || !strings.Contains(cs[0].Body, "2 model") {
t.Errorf("inline body missing title/agreement: %q", cs[0].Body)
}
}
func set(xs ...string) map[string]bool {
m := map[string]bool{}
for _, x := range xs {
m[x] = true
}
return m
}
+115 -2
View File
@@ -45,6 +45,10 @@
# GADFLY_FINDINGS_URL optional gadfly-reports store base URL; set to POST the run + # GADFLY_FINDINGS_URL optional gadfly-reports store base URL; set to POST the run +
# findings for model-quality tracking (off when empty) # findings for model-quality tracking (off when empty)
# GADFLY_FINDINGS_TOKEN optional bearer token for the gadfly-reports store # GADFLY_FINDINGS_TOKEN optional bearer token for the gadfly-reports store
# GADFLY_CONSOLIDATE cross-model consensus comment: "auto" (default; on for >=2
# models), "1" force on, "0" force off (one comment per model)
# GADFLY_INLINE_REVIEW when consolidating, also post a COMMENT-state PR review with
# inline comments on changed lines (default on; "0" disables)
set -uo pipefail set -uo pipefail
# One model by default: the specialist suite already provides breadth, so a # One model by default: the specialist suite already provides breadth, so a
@@ -64,6 +68,29 @@ die() { log "ERROR: $*"; exit 1; }
API() { curl -fsS --connect-timeout 20 --max-time 30 -H "Authorization: token ${GITEA_TOKEN}" "$@"; } API() { curl -fsS --connect-timeout 20 --max-time 30 -H "Authorization: token ${GITEA_TOKEN}" "$@"; }
# upsert_comment_body MARKER BODY — create or update (by leading MARKER) a single
# PR comment. Mirrors run.sh's per-model upsert; used for the consensus comment
# and the per-model fallback when consolidation is on.
upsert_comment_body() {
local marker="$1" body="$2" post_body existing_id="" page=1 cmts
post_body="$(jq -n --arg b "$body" '{body:$b}')"
while [ "$page" -le 10 ]; do
cmts="$(API "${GITEA_API}/issues/${PR}/comments?limit=50&page=${page}" || echo '[]')"
[ "$(echo "$cmts" | jq 'length')" = "0" ] && break
existing_id="$(echo "$cmts" | jq -r --arg m "$marker" \
'.[] | select(.body != null and (.body | startswith($m))) | .id' | head -n1)"
[ -n "$existing_id" ] && break
page=$((page+1))
done
if [ -n "$existing_id" ]; then
curl -sS --connect-timeout 20 --max-time 30 -X PATCH -H "Authorization: token ${GITEA_TOKEN}" \
-H "Content-Type: application/json" "${GITEA_API}/issues/comments/${existing_id}" -d "$post_body" >/dev/null
else
curl -sS --connect-timeout 20 --max-time 30 -X POST -H "Authorization: token ${GITEA_TOKEN}" \
-H "Content-Type: application/json" "${GITEA_API}/issues/${PR}/comments" -d "$post_body" >/dev/null
fi
}
# --- is the commenter allowed to trigger a re-review? ---------------------- # --- is the commenter allowed to trigger a re-review? ----------------------
actor_allowed() { actor_allowed() {
local actor="$1" local actor="$1"
@@ -83,6 +110,26 @@ actor_allowed() {
} }
# --- trigger gating -------------------------------------------------------- # --- trigger gating --------------------------------------------------------
# Gitea >= 1.27 (breaking change go-gitea#37478, "improve support for reusable
# workflows") runs a CALLED workflow with github.event_name = 'workflow_call'
# instead of propagating the caller's event, so every consumer stub's trigger
# arrived here as an unhandled event and the review silently self-skipped
# (observed 2026-07-14: mort PRs #1445-#1447 got 1-second "success" runs).
# Reclassify from the forwarded event payload: a comment body can only come
# from issue_comment (its trigger-phrase + actor gates still apply below);
# otherwise a PR number means a pull_request-shaped trigger (this also covers
# dispatch-through-reusable, whose pr_number input populates PR — the draft
# check degrades safely since IS_DRAFT defaults false). Neither → fall through
# to the unhandled-event skip. Pre-1.27 servers still send the caller's event
# name and never enter this branch.
if [ "${EVENT_NAME:-}" = "workflow_call" ]; then
if [ -n "${COMMENT_BODY:-}" ]; then
EVENT_NAME="issue_comment"
elif [ -n "${PR:-}" ]; then
EVENT_NAME="pull_request"
fi
log "caller event arrived as 'workflow_call'; reclassified to '${EVENT_NAME}'"
fi
case "$EVENT_NAME" in case "$EVENT_NAME" in
workflow_dispatch) workflow_dispatch)
log "manual dispatch for PR #${PR}" ;; log "manual dispatch for PR #${PR}" ;;
@@ -176,10 +223,11 @@ provider_cap() { # provider -> concurrency (override map "p=N,...", else default
} }
review_one() { review_one() {
local sf="" local sf="" ff=""
[ "${GADFLY_STATUS_BOARD:-1}" != "0" ] && sf="$(status_file_for "$1")" [ "${GADFLY_STATUS_BOARD:-1}" != "0" ] && sf="$(status_file_for "$1")"
[ "$CONSOLIDATE" = "1" ] && ff="$(findings_file_for "$1")"
PROVIDER=ollama MODEL="$1" GADFLY_BIN="/usr/local/bin/gadfly" GADFLY_REPO_DIR="$REPO_DIR" \ PROVIDER=ollama MODEL="$1" GADFLY_BIN="/usr/local/bin/gadfly" GADFLY_REPO_DIR="$REPO_DIR" \
GADFLY_STATUS_FILE="$sf" \ GADFLY_STATUS_FILE="$sf" GADFLY_FINDINGS_OUT="$ff" GADFLY_CONSOLIDATE="$CONSOLIDATE" \
bash "${SCRIPTS_DIR}/run.sh" || log "model $1 failed (continuing)" bash "${SCRIPTS_DIR}/run.sh" || log "model $1 failed (continuing)"
# If the binary never wrote real status (run.sh skipped it: empty diff, no key, # If the binary never wrote real status (run.sh skipped it: empty diff, no key,
# binary missing), the pre-seed stays {started:0, done:false} and the board # binary missing), the pre-seed stays {started:0, done:false} and the board
@@ -196,6 +244,34 @@ IFS=',' read -ra _raw <<< "$MODELS" || true
MODEL_LIST=() MODEL_LIST=()
for raw in "${_raw[@]}"; do m="$(echo "$raw" | tr -d '[:space:]')"; [ -n "$m" ] && MODEL_LIST+=("$m"); done for raw in "${_raw[@]}"; do m="$(echo "$raw" | tr -d '[:space:]')"; [ -n "$m" ] && MODEL_LIST+=("$m"); done
# --- cross-model consolidation decision ------------------------------------
# With >=2 models, post ONE consensus comment (findings clustered + ranked by
# cross-model agreement) instead of N per-model walls of prose. Each model writes
# its findings to FINDINGS_DIR; a final pass (the binary in GADFLY_CONSOLIDATE_DIR
# mode) renders the consensus comment. GADFLY_CONSOLIDATE: "auto" (default; on for
# >=2 models), "1" force on, "0" force off (keep per-model comments).
FINDINGS_DIR="${WORKDIR}/findings"
CONSOLIDATE=0
case "${GADFLY_CONSOLIDATE:-auto}" in
1) CONSOLIDATE=1 ;;
0) CONSOLIDATE=0 ;;
*) [ "${#MODEL_LIST[@]}" -ge 2 ] && CONSOLIDATE=1 ;;
esac
# A model spec can contain '/' and ':' (e.g. claude-code/opus, qwen3:14b), so
# sanitize to a flat filename — but append a checksum of the raw spec so two
# specs that sanitize the same (foo:bar vs foo/bar -> foo_bar) don't collide onto
# one file and silently drop a model from the consensus.
findings_file_for() {
local safe sum
safe="$(echo "$1" | tr -c '[:alnum:]._-' '_')"
sum="$(printf '%s' "$1" | cksum | cut -d' ' -f1)"
echo "${FINDINGS_DIR}/${safe}-${sum}.json"
}
if [ "$CONSOLIDATE" = "1" ]; then
rm -rf "$FINDINGS_DIR"; mkdir -p "$FINDINGS_DIR"
log "consolidation ON: ${#MODEL_LIST[@]} models -> one consensus comment"
fi
# Distinct providers, in first-seen order (no associative arrays — portable). # Distinct providers, in first-seen order (no associative arrays — portable).
PROVIDERS="" PROVIDERS=""
for m in "${MODEL_LIST[@]}"; do for m in "${MODEL_LIST[@]}"; do
@@ -257,4 +333,41 @@ if [ -n "$BOARD_PID" ]; then
touch "${STATUS_DIR}/.done" 2>/dev/null || true touch "${STATUS_DIR}/.done" 2>/dev/null || true
wait "$BOARD_PID" 2>/dev/null || true wait "$BOARD_PID" 2>/dev/null || true
fi fi
# --- cross-model consensus comment -----------------------------------------
# Render ONE consensus comment from the per-model findings the swarm wrote. This
# is advisory and best-effort: if the consolidation pass produces nothing, fall
# back to posting each model's review as its own comment (the per-model comments
# were suppressed during the run), so a consolidation hiccup never loses output.
if [ "$CONSOLIDATE" = "1" ]; then
n_files="$(ls -1 "${FINDINGS_DIR}"/*.json 2>/dev/null | wc -l | tr -d '[:space:]')"
log "consolidating findings from ${n_files} model(s)"
# Fetch the PR diff so the consolidator can also post an inline PR review,
# anchoring findings to changed lines (GADFLY_DIFF_FILE). GITEA_API/GITEA_TOKEN/
# GADFLY_PR are already in the binary's environment. Best-effort: an empty diff
# just means no inline review.
DIFF_FILE="${WORKDIR}/pr.diff"
API "${GITEA_API}/pulls/${PR}.diff" > "$DIFF_FILE" 2>/dev/null || true
CONSENSUS="$(GADFLY_CONSOLIDATE_DIR="$FINDINGS_DIR" GADFLY_DIFF_FILE="$DIFF_FILE" \
/usr/local/bin/gadfly 2>"${WORKDIR}/consolidate.err" || true)"
if [ -n "$CONSENSUS" ]; then
BODY="$(printf '%s\n\n<sub>Automated adversarial review by Gadfly — consensus across the model swarm. Advisory only — does not block merge.</sub>' "$CONSENSUS")"
upsert_comment_body "<!-- gadfly-consensus -->" "$BODY"
log "consensus comment posted"
else
log "consolidation produced no output; falling back to per-model comments"
log "$(tail -c 500 "${WORKDIR}/consolidate.err" 2>/dev/null)"
for f in "${FINDINGS_DIR}"/*.json; do
[ -f "$f" ] || continue
m="$(jq -r '.model // ""' "$f" 2>/dev/null)"
[ -z "$m" ] && continue
prov="$(jq -r '.provider // ""' "$f" 2>/dev/null)"
md="$(jq -r '.markdown // ""' "$f" 2>/dev/null)"
marker="<!-- gadfly-review:ollama:${m} -->"
body="$(printf '%s\n### 🪰 Gadfly review — `%s` (%s)\n\n%s\n\n<sub>Automated adversarial review by Gadfly. Advisory only — does not block merge.</sub>' \
"$marker" "$m" "$prov" "$md")"
upsert_comment_body "$marker" "$body"
done
fi
fi
log "done" log "done"
+7 -6
View File
@@ -14,10 +14,11 @@
# Forward ONLY the secrets the reviewer uses (least privilege) — see the # Forward ONLY the secrets the reviewer uses (least privilege) — see the
# `secrets:` block below. GITEA_TOKEN is automatic. `secrets: inherit` also works # `secrets:` block below. GITEA_TOKEN is automatic. `secrets: inherit` also works
# but hands the reusable EVERY secret in your repo (registry/deploy/db creds the # but hands the reusable EVERY secret in your repo (registry/deploy/db creds the
# review never touches), so prefer the explicit form. Pin @<ref>: use the @v1 # review never touches), so prefer the explicit form. Pin to an immutable
# release tag (a curated pointer moved on deliberate releases) for auto-updating # @<sha>: long-lived act_runners CACHE the reusable by ref, so a moved tag (@v1)
# stability, or a full @<sha> for an immutable pin. Avoid @main — it moves on # or @main is often not re-fetched and silently runs a stale copy. Bump the @<sha>
# every push and would change what runs with your forwarded secrets. # to adopt a structural change; routine swarm tuning rides owner variables (see
# the gadfly README "Central config via variables") with no re-pin needed.
# #
# For custom named endpoints (GADFLY_ENDPOINT_<NAME>) or a provider the reusable # For custom named endpoints (GADFLY_ENDPOINT_<NAME>) or a provider the reusable
# doesn't map, use the full stub in adversarial-review.yml instead. # doesn't map, use the full stub in adversarial-review.yml instead.
@@ -49,8 +50,8 @@ jobs:
if: >- if: >-
github.event_name != 'issue_comment' github.event_name != 'issue_comment'
|| (github.event.issue.pull_request && github.actor == 'your-username') || (github.event.issue.pull_request && github.actor == 'your-username')
# @v1 = curated release tag (auto-updates on releases); swap for a full @<sha> # Pin to an immutable @<sha> (runners cache the ref, so @v1/@main can run
# if you want an immutable pin. Don't use @main (moves on every push). # stale). Bump it for structural changes; tune the swarm via owner variables.
uses: steve/gadfly/.gitea/workflows/review-reusable.yml@v1 uses: steve/gadfly/.gitea/workflows/review-reusable.yml@v1
# Forward ONLY what the reviewer needs. Add provider keys you use # Forward ONLY what the reviewer needs. Add provider keys you use
# (ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_API_KEY, GADFLY_API_KEY) and/or # (ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_API_KEY, GADFLY_API_KEY) and/or
+28 -1
View File
@@ -50,6 +50,13 @@ MAX_DIFF_CHARS="${MAX_DIFF_CHARS:-60000}"
MARKER="<!-- gadfly-review:${PROVIDER}:${MODEL} -->" MARKER="<!-- gadfly-review:${PROVIDER}:${MODEL} -->"
say() { echo "[gadfly-review:${PROVIDER}:${MODEL}] $*" >&2; } say() { echo "[gadfly-review:${PROVIDER}:${MODEL}] $*" >&2; }
# When the swarm is consolidating (GADFLY_CONSOLIDATE=1, set by entrypoint.sh for
# a multi-model run), this model does NOT post its own comment — it writes its
# findings to GADFLY_FINDINGS_OUT and a single cross-model consensus comment is
# posted after the whole swarm finishes. Live progress still shows on the status
# board. Default 0 (post a per-model comment, the standalone behavior).
CONSOLIDATE="${GADFLY_CONSOLIDATE:-0}"
# Display the model's ACTUAL backend: the provider segment of the spec # Display the model's ACTUAL backend: the provider segment of the spec
# ("m1pro/qwen3.6:35b-mlx" -> "m1pro"); a bare id uses GADFLY_PROVIDER (default # ("m1pro/qwen3.6:35b-mlx" -> "m1pro"); a bare id uses GADFLY_PROVIDER (default
# ollama-cloud). This is what the comment header shows, not the run.sh lane. # ollama-cloud). This is what the comment header shows, not the run.sh lane.
@@ -126,7 +133,9 @@ USR="$(printf 'PR #%s: %s\n\nDescription:\n%s\n\nUnified diff to review:\n```dif
# --- announce start (placeholder comment) ----------------------------------- # --- announce start (placeholder comment) -----------------------------------
START_TS="$(date +%s)" START_TS="$(date +%s)"
say "starting review with ${MODEL}" say "starting review with ${MODEL}"
upsert_comment "$(printf '%s\n### 🪰 Gadfly review — `%s` (%s)\n\n⏳ Reviewing… this comment will update with findings and run time.' \ # Skip the per-model placeholder when consolidating (the consensus comment is
# posted later; live progress is on the status board).
[ "$CONSOLIDATE" = "1" ] || upsert_comment "$(printf '%s\n### 🪰 Gadfly review — `%s` (%s)\n\n⏳ Reviewing… this comment will update with findings and run time.' \
"$MARKER" "$MODEL" "$MODEL_PROVIDER")" "$MARKER" "$MODEL" "$MODEL_PROVIDER")"
# --- call the model --------------------------------------------------------- # --- call the model ---------------------------------------------------------
@@ -170,6 +179,7 @@ case "$PROVIDER" in
GADFLY_BODY="$BODY" \ GADFLY_BODY="$BODY" \
GADFLY_MAX_DIFF_CHARS="$MAX_DIFF_CHARS" \ GADFLY_MAX_DIFF_CHARS="$MAX_DIFF_CHARS" \
GADFLY_STATUS_FILE="${GADFLY_STATUS_FILE:-}" \ GADFLY_STATUS_FILE="${GADFLY_STATUS_FILE:-}" \
GADFLY_FINDINGS_OUT="${GADFLY_FINDINGS_OUT:-}" \
"$BIN" 2>"$ERR_FILE" "$BIN" 2>"$ERR_FILE"
)" )"
rc=$? rc=$?
@@ -204,7 +214,24 @@ esac
# --- assemble + post final comment (with run time) -------------------------- # --- assemble + post final comment (with run time) --------------------------
ELAPSED="$(( $(date +%s) - START_TS ))" ELAPSED="$(( $(date +%s) - START_TS ))"
DUR="$(fmt_duration "$ELAPSED")" DUR="$(fmt_duration "$ELAPSED")"
# Consolidating: the binary writes its findings file on success. If it failed or
# was skipped (no file, or an empty one), write a stub so this model still shows
# up in the consensus (as failed) and an all-models-fail run still posts a
# comment — never silently drop a model or the whole review.
if [ "$CONSOLIDATE" = "1" ] && [ -n "${GADFLY_FINDINGS_OUT:-}" ] && [ ! -s "${GADFLY_FINDINGS_OUT}" ]; then
jq -n --arg model "$MODEL" --arg provider "$MODEL_PROVIDER" --arg md "$REVIEW" \
'{model:$model, provider:$provider, verdict:"reviewer failed", errored:true, markdown:$md, findings:[]}' \
> "${GADFLY_FINDINGS_OUT}" 2>/dev/null || true
fi
# When consolidating, the binary has written this model's findings to
# GADFLY_FINDINGS_OUT; the consensus comment is posted by entrypoint.sh after the
# whole swarm finishes, so this model posts no comment of its own.
if [ "$CONSOLIDATE" = "1" ]; then
say "done in ${DUR} (consolidated; no per-model comment)"
else
COMMENT="$(printf '%s\n### 🪰 Gadfly review — `%s` (%s)\n\n%s\n\n<sub>Automated adversarial review by Gadfly. Advisory only — does not block merge. · ⏱️ reviewed in %s</sub>' \ COMMENT="$(printf '%s\n### 🪰 Gadfly review — `%s` (%s)\n\n%s\n\n<sub>Automated adversarial review by Gadfly. Advisory only — does not block merge. · ⏱️ reviewed in %s</sub>' \
"$MARKER" "$MODEL" "$MODEL_PROVIDER" "$REVIEW" "$DUR")" "$MARKER" "$MODEL" "$MODEL_PROVIDER" "$REVIEW" "$DUR")"
upsert_comment "$COMMENT" upsert_comment "$COMMENT"
say "done in ${DUR}" say "done in ${DUR}"
fi
+18
View File
@@ -43,3 +43,21 @@ Output rules:
- Only report issues you are reasonably confident are real after checking. If the diff - Only report issues you are reasonably confident are real after checking. If the diff
is clean, say so plainly rather than inventing nits. is clean, say so plainly rather than inventing nits.
- When you are done investigating, STOP calling tools and reply with the final review. - When you are done investigating, STOP calling tools and reply with the final review.
Machine-readable findings — AFTER the prose review, append ONE fenced code block,
tagged `gadfly-findings`, holding a JSON array of the SAME findings you described above
(this block is consumed by tooling and hidden from the rendered comment):
```gadfly-findings
[
{"file": "path/to/file.go", "line": 123, "severity": "high", "confidence": "high", "title": "one-line summary of the issue"}
]
```
- One object per real finding, in the same order as your prose. `file`/`line` must be a
concrete location you verified (the line the issue is at). `severity` is one of
`critical`, `high`, `medium`, `small`, `trivial`. `confidence` is your post-verification
confidence the issue is real: one of `high`, `medium`, `low`.
- Include ONLY genuine problems — never verification notes ("confirmed X is safe at f:line"),
and never an "Outside my lens:" aside. If your lens is clean, emit an empty array `[]`.
- This block is in ADDITION to the prose; do not drop the human-readable findings.