Compare commits

..

42 Commits

Author SHA1 Message Date
steve ac6ce06cdd feat: re-platform agentic review onto executus + large-PR cost controls (#20)
Build & push image / build-and-push (push) Successful in 33s
Makes gadfly a consumer of executus (run.Executor compaction/bounding/budget/critic + fanout) and fixes the large-PR token burn in size-gated layers: paginated get_diff, downshift above GADFLY_HUGE_DIFF_BYTES, and a swarm-wide GADFLY_PR_BUDGET_SECS backstop. Small PRs untouched; advisory-only and the static binary preserved. Dogfood swarm reviewed it (6 models, 21 real findings graded + folded in).

Co-authored-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
Co-committed-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
2026-06-30 15:41:03 +00:00
steve 5007597cf9 chore(reusable): bump image pin to sha-3095ebf (inline PR review live)
Phase 3: gadfly's own multi-model reviews now also post a COMMENT-state PR
review with inline comments anchored to changed lines. External consumers
re-pin separately.

[skip ci]

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 22:00:39 -04:00
steve 3095ebff23 feat: inline COMMENT-state PR review (findings anchored to changed lines) (#18)
Build & push image / build-and-push (push) Successful in 8s
Co-authored-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
Co-committed-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
2026-06-29 01:59:36 +00:00
steve 8f5adc91b2 chore(reusable): bump image pin to sha-88f74aa (consensus consolidation live)
Phase 2: gadfly's own multi-model reviews now post ONE cross-model consensus
comment instead of N per-model comments. External consumers re-pin separately.

[skip ci]

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 18:57:38 -04:00
steve 88f74aa768 feat: cross-model consensus consolidation (one ranked comment, not N walls) (#17)
Build & push image / build-and-push (push) Successful in 9s
Co-authored-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
Co-committed-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
2026-06-28 22:56:15 +00:00
steve 84b891b1ba chore(reusable): bump image pin to sha-5397160 (structured findings contract)
Makes the Phase 1 gadfly-findings contract live for gadfly's own dogfood
reviews (the local-ref reusable). External consumers re-pin separately.

[skip ci]

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 18:25:16 -04:00
steve 53971603d3 feat: structured findings contract (machine-readable gadfly-findings block) (#16)
Build & push image / build-and-push (push) Successful in 5s
Co-authored-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
Co-committed-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
2026-06-28 22:23:02 +00:00
steve f49699fc12 Merge pull request 'docs: correct examples/reusable.yml pin guidance (prefer @sha; runners cache @v1)' (#15) from test/trigger-check into main
Reviewed-on: #15
2026-06-28 22:09:07 +00:00
Steve Dudenhoeffer 6e87a3e73f docs: correct examples/reusable.yml pin guidance (runners cache @v1; prefer @sha)
Adversarial Review (Gadfly) / review (pull_request) Successful in 3m4s
The @v1 comment claimed it auto-updates on releases, but long-lived act_runners
cache the reusable by ref so a moved tag isn't re-fetched. Recommend an
immutable @<sha>; routine tuning rides owner variables.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-28 02:10:35 -04:00
steve 7bc3c982fa feat(reusable): runtime-variable swarm config (cache-immune, no more re-pinning to retune) (#14)
Build & push image / build-and-push (push) Successful in 5s
2026-06-28 06:00:18 +00:00
steve 95a9ec546a feat(reusable): add the 4090 Ti (qwen3.6-27b via llama-swap) to the default swarm (#13)
Build & push image / build-and-push (push) Successful in 7s
2026-06-28 05:01:50 +00:00
steve 8f69e71311 docs: recommend the @v1 release tag for reusable-workflow consumers (#12)
Build & push image / build-and-push (push) Successful in 6s
2026-06-28 04:17:19 +00:00
steve 0d80ae73d8 tune(reusable): claude-code=3 models × 5 lenses (claude was the bottleneck) (#11)
Build & push image / build-and-push (push) Successful in 8s
2026-06-28 04:02:17 +00:00
steve b02b11d691 feat(reusable): ship the curated swarm as the default config consumers inherit (#10)
Build & push image / build-and-push (push) Successful in 8s
2026-06-28 02:23:40 +00:00
steve 20a5c431f2 Merge pull request 'security: scope reusable-workflow secrets (least privilege) over secrets: inherit' (#9) from sec/scope-secrets into main
Build & push image / build-and-push (push) Successful in 6s
Reviewed-on: #9
2026-06-28 01:17:16 +00:00
Steve Dudenhoeffer daff6d08a1 docs: drop stale 'secrets: inherit' mentions (reusable comment + CLAUDE.md)
Build & push image / build-and-push (pull_request) Successful in 6s
Self-review on PR #9 flagged two doc-drift spots left over from the
explicit-secret-forwarding switch. Cosmetic.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-27 21:00:40 -04:00
Steve Dudenhoeffer 18de9b8ebc fix: source GITEA_TOKEN from github.token (auto) under explicit secret forwarding
Build & push image / build-and-push (pull_request) Successful in 7s
Adversarial Review (Gadfly) / review (pull_request) Successful in 8m2s
The first attempt failed at entrypoint.sh:61 'GITEA_TOKEN required' — with
explicit secrets (no `inherit`), secrets.GITEA_TOKEN resolves empty in the
reusable job. github.token comes from the github context (not a forwarded
secret), so it's present regardless. The forwarded provider/findings secrets
arrived correctly; only the auto-token sourcing was wrong.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-27 20:53:00 -04:00
Steve Dudenhoeffer f06fe5ef72 security: scope reusable-workflow secrets (least privilege) over secrets: inherit
Adversarial Review (Gadfly) / review (pull_request) Failing after 2s
Build & push image / build-and-push (pull_request) Successful in 6s
The swarm (reviewing the mort/executus rollout PRs) correctly flagged that
`secrets: inherit` forwards EVERY caller secret to the reusable review
workflow — registry/deploy/db creds the reviewer never touches. Fix:

- review-reusable.yml: declare workflow_call.secrets (all optional) so a
  caller can forward only what the reviewer needs.
- adversarial-review.yml (gadfly's own caller) + examples/reusable.yml:
  replace `secrets: inherit` with an explicit forward of just
  OLLAMA_CLOUD_API_KEY / CLAUDE_CODE_OAUTH_TOKEN / findings tokens.
  GITEA_TOKEN stays automatic.
- Docs (README, examples) updated; also advise pinning consumers to an
  immutable @<sha> instead of @main (supply-chain, the other finding).

gadfly's own review on this PR exercises the explicit-secrets path (local
reusable ref) — validating it on the act_runner before mort/executus adopt it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-27 20:45:18 -04:00
steve 5f86062a5a feat: Phase 4 — reusable "subscribe" workflow (+ dogfood it) (#8)
Build & push image / build-and-push (push) Successful in 9s
Centralizes the consumer stub into a reusable Gitea workflow
(.gitea/workflows/review-reusable.yml, workflow_call + defaulted inputs +
secrets: inherit); gadfly's own dogfood is now a thin caller of it, which
proved end-to-end that github.event context propagates into the reusable
on this act_runner. Adds the slim examples/reusable.yml stub + docs.

Folded in the swarm's findings: timeout_minutes default 30->45, map
GADFLY_API_KEY, explicit permissions block, drop the dead specialist_suite
input, and harden the example's actor gate. ~70 findings graded.

Completes the gadfly-games build (Phases 1-4 + quality fixes).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
Co-committed-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
2026-06-27 23:42:01 +00:00
steve b23eeb8cbf feat: bump majordomo + support llama-swap(s) provider spellings (#7)
Build & push image / build-and-push (push) Successful in 7s
Bump majordomo to the latest build and accept every llama-swap spelling
(llama-swap/llama-swaps + un-hyphenated llamaswap/llamaswaps) in gadfly's
endpoint switches; the LLM_* llama-swap(s):// DSN path already worked via
majordomo.Parse. README + error messages + endpointProvider alias tests.

Swarm review: 8/9 clean; qwen3-coder's "Blocking" was a false positive
(claimed llamaswap was untested — it has dedicated test cases). Folded in
its one fair nit (README now lists the un-hyphenated aliases).

gofmt clean, go vet quiet, go test -race green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
Co-committed-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
2026-06-27 23:18:56 +00:00
steve a4cdc905c9 ci: enable claude-code/opus:max (max-thinking) reviewer (#6)
Build & push image / build-and-push (push) Successful in 6s
Adds claude-code/opus:max to the dogfood swarm and pins to :sha-c342bdb
(which has the :thinking parse). Claude Code lineup is now sonnet + opus +
opus:max. All three ran end-to-end on this PR's own review; 0 findings
(clean PR + the telemetry fix suppressing phantom clean-verification
findings — working as intended).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
Co-committed-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
2026-06-27 22:49:49 +00:00
steve c342bdb905 feat: add claude-code/opus reviewer + max-thinking spec support (#5)
Build & push image / build-and-push (push) Successful in 15s
Adds claude-code/opus to gadfly's dogfood swarm (both sonnet and opus run
end-to-end), bumps the image pin to :sha-80d8f53 so the clean-lens
telemetry fix is live, and adds engine support for a
"claude-code/<model>:max" extended-thinking spec (MAX_THINKING_TOKENS,
best-effort). Validated: only 13 findings on this clean PR vs 43 on the
comparable #4 — the telemetry fix works.

Folded in the swarm's two real findings: a runPass env-injection test and
keeping MAX_THINKING_TOKENS in claudeEnv. Follow-up enables
claude-code/opus:max once this image builds.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
Co-committed-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
2026-06-27 22:39:14 +00:00
steve 80d8f53f63 fix: clean-lens findings + trim the dogfood swarm to strong reviewers (#4)
Build & push image / build-and-push (push) Successful in 9s
emit() now skips findings extraction for a "No material issues found"
lens (its path:line refs are verification notes, not problems), fixing
the FP inflation that penalized thorough clean-pass reviewers. Also trims
the dogfood swarm to the strong reviewers: drops m5/qwen3.6 (last local
lane), gemma4, gpt-oss:120b, and kimi-k2.7-code — leaving 6 cloud +
claude-code/sonnet.

Fittingly, PR #4's own 11-model review produced 43 findings that were ALL
clean-verification bullets (zero real) — a live demonstration of the bug
this fixes. gofmt clean, go vet quiet, go test -race green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
Co-committed-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
2026-06-27 22:14:07 +00:00
steve 82f7ef78d5 feat: claude-code backends + llamaswap provider + dogfood the CC engine (#3)
Build & push image / build-and-push (push) Successful in 10s
Phase 2: bump majordomo to latest and wire its new llamaswap provider
into gadfly's endpoint switches; add claude-code/sonnet to gadfly's own
dogfood swarm (pin :sha-86f12c1, map CLAUDE_CODE_OAUTH_TOKEN) so the
Phase-1 engine runs as a live competitor; document the Ollama-through-CC
ANTHROPIC_BASE_URL proxy path as example-only.

The 11-model swarm (incl. claude-code/sonnet) reviewed it; 52 findings
graded via the MCP. Folded in the two real ones: a llamaswap
endpointProvider test (caught by claude-code/sonnet, citing CLAUDE.md)
and adding "openai-compatible" to the provider error messages (gpt-oss).

gofmt clean, go vet quiet, go build + go test -race green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
Co-committed-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
2026-06-27 21:53:41 +00:00
steve 86f12c126f feat: claude-code reviewer engine (#2)
Build & push image / build-and-push (push) Successful in 28s
Phase 1: a second review engine alongside the majordomo agent loop. For
each lens, shell out to the Claude Code CLI (`claude -p --output-format
json`) inside the checked-out repo so it verifies findings with its own
read tools, then reuse gadfly's verdict-parse + recheck + consolidate +
emit pipeline. Select via GADFLY_MODELS `claude-code`/`claude-code/<model>`;
auth via CLAUDE_CODE_OAUTH_TOKEN (no --bare) else ANTHROPIC_API_KEY;
read-only by default; GADFLY_CLAUDE_* knobs. Dockerfile bundles Node +
@anthropic-ai/claude-code. Also bumped the dogfood pin to the status-board
image (PR #2 was the first dogfood with the live board + full fleet).

Folded in the swarm's own review findings: minimal subprocess env (no
GITEA_TOKEN leak to the CLI), runPass robustness (ctx/empty-result/runErr),
process-group cleanup on timeout, rune-safe error truncation, and
engine-neutral prompts (also de-mort-ified the recheck prompt). 66 findings
graded via the gadfly MCP.

gofmt clean, go vet quiet, go build + go test -race green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
Co-committed-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
2026-06-27 20:40:41 +00:00
steve c3d09d3bd4 feat: live status-board comment + full-fleet dogfood (#1)
Build & push image / build-and-push (push) Successful in 6s
Phase 3: one consolidated, live-updating PR comment aggregating every
model's per-lens progress (queued -> running -> finished + verdict), so
the swarm's progress is visible at a glance and a watcher can tell when
it's done. Opt-in statusWriter in the binary (atomic writes) + a
background status-board.sh renderer wired through entrypoint.sh; default
on, GADFLY_STATUS_BOARD=0 to disable.

Also restores gadfly's dogfood swarm to the full cloud fleet (9 cloud +
M5; M1 dropped as too slow) matching mort, and folds in the 3 real bugs
the swarm found on its own PR (skip-binary stuck-waiting, panic-stuck
lens, busy-loop on bad poll interval). All 36 findings graded via the
gadfly MCP (18 real / 18 false-positive).

gofmt clean, go vet quiet, go build + go test -race green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
Co-committed-by: Steve Dudenhoeffer <steve@stevedudenhoeffer.com>
2026-06-27 19:00:12 +00:00
steve 0ad5b66170 ci: dogfood — gadfly reviews its own PRs (mort's full-fleet setup)
Build & push image / build-and-push (push) Successful in 14s
Adds the adversarial-review workflow to gadfly itself (copied from mort: 3 cloud + m1/m5 via foreman, findings telemetry, sha-d7f364d). Future gadfly PRs get reviewed by the swarm.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-27 13:26:37 -04:00
steve d7f364d803 feat: optional findings telemetry — emit runs+findings to a gadfly-reports store
Build & push image / build-and-push (push) Successful in 8s
After each review the binary POSTs the run + its heuristically-extracted findings to GADFLY_FINDINGS_URL (off unless set). Advisory: any error only goes to stderr — never touches stdout, the exit code, or the review. stdlib net/http only (no new deps). entrypoint.sh derives GADFLY_REPO/GADFLY_PR and passes through GADFLY_FINDINGS_URL/GADFLY_FINDINGS_TOKEN. Also renames store references from the old 'docket' name to 'gadfly-reports'.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-27 09:09:51 -04:00
steve d0de034726 feat: configurable lens fan-out, per-provider like model concurrency
Build & push image / build-and-push (push) Successful in 9s
Specialist lenses ran strictly sequentially within a model. Add a
GADFLY_LENS_CONCURRENCY knob (default 1 = unchanged) that overlaps the
independent per-lens review+recheck passes, so a model posts its
consolidated comment as soon as its lenses finish.

Per-provider configurable, mirroring GADFLY_PROVIDER_CONCURRENCY:
GADFLY_PROVIDER_LENS_CONCURRENCY takes a "provider=N,..." map keyed by
the same provider lanes (modelProvider() mirrors entrypoint's provider_of;
providerOverride() mirrors provider_cap). The override wins for the model's
lane, else the scalar default.

runSpecialists fans out via a bounded worker pool, order-preserving
(results written by index) and keeping each lens's own timeout/recheck.
repoFS is immutable + fresh-toolbox-per-pass, so lenses share no mutable
state (verified under -race). Docs/examples updated; dropped a duplicate
GADFLY_TIMEOUT_SECS README row.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 22:53:27 -04:00
steve 6e3a83c437 feat: add foreman provider type for endpoint overrides
Build & push image / build-and-push (push) Successful in 7s
Accept "foreman" in both resolveModel (GADFLY_BASE_URL) and endpointProvider
(GADFLY_ENDPOINT_*) switches, mapping to majordomo's ollama.Foreman() preset
(handles foreman's non-streaming/long-poll quirks). Unlike the HTTPS-only
LLM_* foreman:// DSN, the base URL is verbatim, so a plaintext http:// foreman
queue works. Tests + README provider table + endpoint-aliases example updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-26 20:13:47 -04:00
Steve Dudenhoeffer a1e9d109e5 security: add job-level if-guard to example stubs (gate comment trigger by actor)
Build & push image / build-and-push (push) Successful in 5s
Per a Gadfly self-review finding (kimi-k2.7-code): an issue_comment can start a
secret-bearing run before the in-container allowed-users check. Add a workflow
if: that only lets trusted actors trigger via comment (PR/dispatch already
trusted); keep GADFLY_ALLOWED_USERS as the belt-and-suspenders layer. README
documents it + the default-branch caveat for comment triggers. (Docs/examples
only — paths-ignored, no image rebuild.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
EOF
2026-06-25 21:49:23 -04:00
Steve Dudenhoeffer b409dff4ed fix: parseVerdict matches leniently + earliest phrase wins
Build & push image / build-and-push (push) Successful in 8s
A section that led with '**Blocking issues**' (no 'found') fell through to
unknown, so the consolidated header wrongly read 'No material issues found'
(seen live on gpt-oss). Now matches 'blocking issue'/'minor issue'/'no material
issue' and picks the earliest-appearing phrase (the lead verdict). + tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 21:28:19 -04:00
Steve Dudenhoeffer 92bf22a1be fix: correctness lens emoji -> 🎯 ( read like 'no issues')
Build & push image / build-and-push (push) Successful in 8s
A green check on a section reporting blocking issues was misleading; 🎯 signals
accuracy/on-target. Section verdict text already conveys pass/fail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 20:50:39 -04:00
Steve Dudenhoeffer 9e582bfaca feat: per-provider concurrency lanes (cloud parallel while local churns)
Build & push image / build-and-push (push) Successful in 7s
entrypoint.sh groups models by provider into lanes that run in PARALLEL; within
a lane at most `cap` models run at once. cap = GADFLY_PROVIDER_CONCURRENCY map
("ollama-cloud=3,m1pro=1") else GADFLY_CONCURRENCY (default 1). So a single
local box stays serial (1 at a time) while cloud models run several at once and
both lanes progress simultaneously. Portable bash (no associative arrays).
Default cap 1 keeps a single-provider pool sequential as before. Pairs with the
per-lens timeout so a slow lane can't starve others. Docs: README Concurrency
section + config table; CLAUDE.md lessons incl. the docker://:latest cache gotcha.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 20:29:08 -04:00
Steve Dudenhoeffer 49f3623204 fix: per-lens timeout, errored-verdict honesty, accurate provider label, tighter lens focus, run timing
Build & push image / build-and-push (push) Successful in 8s
Five fixes, several surfaced by the live bake-off:

- PER-LENS TIMEOUT (critical): GADFLY_TIMEOUT_SECS now applies to EACH specialist
  (own context), not shared across the suite. A slow model (e.g. a 35B local MLX)
  was exhausting the whole 600s budget on lens 1, leaving the rest "step 0:
  context deadline exceeded". Default lowered to 300s (per-lens). cmd/gadfly/main.go.
- ERRORED VERDICT: a lens whose review pass failed no longer counts as "clean".
  Header shows "· ⚠️ N/M lens(es) errored" (or "Review incomplete — all lenses
  errored"); the section reads "⚠️ could not complete". consolidate.go.
- PROVIDER LABEL: the comment header now shows the model's ACTUAL backend from the
  spec ("m1pro/qwen3.6:35b-mlx" -> m1pro), not the global GADFLY_PROVIDER default
  (was wrongly "ollama-cloud" for local models). scripts/run.sh.
- LENS FOCUS: base prompt no longer licenses "report anything serious"; each lens
  stays in its lane, says "nothing in my area" rather than re-reporting another
  lens's bug, with a one-line "Outside my lens:" escape hatch. The re-derive-
  constants discipline is now lane-scoped, not "every lens". system-prompt.txt + specialists.go.
- RUN TIMING: run.sh posts a " Reviewing…" placeholder at model start and updates
  it with "⏱️ reviewed in 1m 23s" on finish, for per-model comparison.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 20:15:40 -04:00
Steve Dudenhoeffer 4b8f9aa39b feat: dynamic auto specialist selection + worker-tier delegation
Build & push image / build-and-push (push) Successful in 33s
Two Phase-2 swarm upgrades:

- auto.go: GADFLY_SPECIALISTS=auto routes the review — a selector model
  (GADFLY_SELECTOR_MODEL, else the review model) reads the changed files + PR
  description and picks the smallest relevant lens set from the catalog, and may
  propose ad-hoc lenses for gaps (e.g. migrations). Structured output via
  majordomo.Generate[T]; capped + de-duped; falls back to the default suite.
- delegate.go: GADFLY_WORKER_MODEL adds a delegate_investigation tool so the
  reviewer offloads mechanical legwork (trace callers, gather usages) to a cheap
  worker sub-agent that returns an evidence-cited digest — the top model reasons
  over summaries, not raw file dumps. Workers get an fs-only toolbox (no
  sub-delegation). Unset = off.

resolveSpecialists now also returns the registry + an auto flag. Docs (README
Specialists + config table, CLAUDE.md, main.go header) + tests updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 19:35:59 -04:00
Steve Dudenhoeffer 7809d1b93d feat: specialist suite — configurable + custom review lenses (one consolidated comment)
Build & push image / build-and-push (push) Successful in 8s
Replace the single generic review with a suite of focused specialists, each its
own review+recheck pass, merged into ONE comment (a collapsible section per lens,
led by the worst verdict; the optional `improvements` lens never escalates it).

- cmd/gadfly/specialists.go: built-in lenses + default suite (security, correctness,
  maintainability, performance, error-handling) + opt-in (tests, docs, conventions,
  improvements). Selection via GADFLY_SPECIALISTS (csv/"all"); custom defs via
  GADFLY_SPECIALIST_<NAME> env and a repo .gadfly.yml (specialists + define).
  Precedence: built-ins < file < env. Unknown names error but don't sink the run.
- cmd/gadfly/consolidate.go: verdict parse + one-comment render.
- main.go: loop specialists; per-lens failure is an inline notice, never fatal.
  Default timeout bumped to 600s (suite runs sequentially).
- base system prompt trimmed to persona+tools+discipline+output; lens-specific
  focus is appended per specialist (semantic re-derivation discipline kept in base).
- entrypoint default models -> single model (suite already gives breadth; cost ~=
  specialists × models × 2). Adds gopkg.in/yaml.v3.
- docs/examples: README "Specialists" section, examples/.gadfly.yml, stub var,
  CLAUDE.md architecture/config. Dynamic `auto` selection is the planned next step.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 19:23:05 -04:00
Steve Dudenhoeffer 676c9d4f07 ci: skip image rebuild on docs/example-only changes (paths-ignore)
Build & push image / build-and-push (push) Successful in 5s
Tag pushes (v*) bypass path filters, so releases always build.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 19:10:24 -04:00
Steve Dudenhoeffer 04cd260ff9 docs: add CLAUDE.md + provider example configs
Build & push image / build-and-push (push) Successful in 6s
- CLAUDE.md: project goals (advisory-only, real-bugs-not-nits, easy-to-enable,
  provider-agnostic, portable), architecture map, build/test/release, and
  maintenance rules — incl. "keep README + examples/ current with any env/flag/
  provider/trigger change" and the advisory-only invariant.
- examples/: local-ollama.yml, openai-compatible.yml, endpoint-aliases.yml +
  an examples/README index; README setup step points at them.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 19:06:08 -04:00
Steve Dudenhoeffer bd76aa8286 feat: env-defined endpoint aliases (http-capable, local Ollama friendly)
Build & push image / build-and-push (push) Successful in 9s
majordomo's built-in LLM_* env DSNs are HTTPS-only (DSN.BaseURL forces https),
so they can't express a plaintext local Ollama. Add Gadfly-native env families
that register named providers/aliases with majordomo before resolution:

  GADFLY_ENDPOINT_<NAME>="<provider>|<base-url>[|<key>]"  # base URL verbatim (http ok)
  GADFLY_ALIAS_<NAME>="<majordomo spec>"                  # plain alias / failover chain

Then reference them as "<name>/<model>" (or the bare alias) in GADFLY_MODEL(S).
<NAME> lowercases to the registry name, matching majordomo's LLM_* convention.
LLM_* DSNs still work (and are documented) for HTTPS endpoints. + unit tests,
README "Endpoint aliases via env vars", stub example.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 19:01:07 -04:00
Steve Dudenhoeffer d9405f4f69 feat: multi-provider model support via majordomo (local Ollama, OpenAI-compatible, etc.)
Build & push image / build-and-push (push) Successful in 18s
Replace the hardcoded ollama.Cloud binding with majordomo's provider registry,
so Gadfly can target any backend majordomo supports without code changes.

- cmd/gadfly/model.go: resolveModel() — GADFLY_PROVIDER (default ollama-cloud)
  prefixes bare model ids; GADFLY_MODEL may be a full provider/model spec, alias,
  or failover chain (verbatim). GADFLY_BASE_URL constructs openai/ollama/anthropic/
  google directly at a custom endpoint (OpenAI-compatible + local/remote Ollama).
  GADFLY_API_KEY else the provider's standard env var. + buildSpec unit tests.
- run.sh: provider-aware key gate (local Ollama needs none); maps OLLAMA_CLOUD_API_KEY
  -> OLLAMA_API_KEY; provider/base-url/key inherited by the binary. Gadfly-branded comment.
- entrypoint.sh: GADFLY_MODELS alias for OLLAMA_REVIEW_MODELS; provider passthrough.
- examples + README: Models & providers section. Upfront: only the Ollama paths
  (local + OpenAI-compatible-against-Ollama) are tested; OpenAI/Anthropic/Google
  are wired via majordomo but UNTESTED (no spend).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 18:58:00 -04:00
Steve Dudenhoeffer 6123604595 ci: auto build & push image on main (:latest) + v* tags
Build & push image / build-and-push (push) Successful in 58s
Mirror mort-ci.yml's build-and-push: BuildKit secrets (REGISTRY_USER/
REGISTRY_PASSWORD) for private majordomo access instead of build-args, and the
LAN --add-host so the builder can reach the registry. push main -> :latest +
:sha-<short>; tag v* -> :<tag> + :latest; other branches -> :branch-<safe>;
PRs build-only (no push). Optional DISCORD_WEBHOOK_URL notifications.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 18:45:48 -04:00
48 changed files with 7349 additions and 339 deletions
+55
View File
@@ -0,0 +1,55 @@
# Gadfly reviewing its OWN PRs — a thin CALLER of the reusable workflow
# (.gitea/workflows/review-reusable.yml), dogfooding the "subscribe" path. The
# reusable holds the image pin, env plumbing, AND the default swarm; this file
# holds only the triggers, the actor gate, secret forwarding, and allow-list.
#
# Advisory only — never blocks a merge. It inherits the default swarm: 3 cloud
# models + Claude Code (sonnet, opus, opus:max), 5-lens suite (claude models run
# one at a time, each with all 5 lenses at once).
name: Adversarial Review (Gadfly)
on:
pull_request:
types: [opened, reopened, ready_for_review]
issue_comment:
types: [created]
workflow_dispatch:
inputs:
pr_number:
description: "PR number to review"
required: true
permissions:
contents: read
issues: write
pull-requests: write
concurrency:
group: gadfly-${{ github.event.issue.number || github.event.pull_request.number || github.event.inputs.pr_number }}
cancel-in-progress: true
jobs:
review:
# Security: only trusted users may trigger a secret-bearing run via a PR
# comment (pull_request + workflow_dispatch are already trusted). Mirrors
# the allowed_users input below, the in-container belt-and-suspenders check.
if: >-
github.event_name != 'issue_comment'
|| (github.event.issue.pull_request
&& (github.actor == 'steve'
|| github.actor == 'fizi'
|| github.actor == 'dazed'))
uses: ./.gitea/workflows/review-reusable.yml
# Least privilege: forward ONLY the secrets this swarm uses (cloud + Claude
# Code + findings telemetry), not `secrets: inherit`. GITEA_TOKEN is auto.
secrets:
OLLAMA_CLOUD_API_KEY: ${{ secrets.OLLAMA_CLOUD_API_KEY }}
CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
GADFLY_FINDINGS_URL: ${{ secrets.GADFLY_FINDINGS_URL }}
GADFLY_FINDINGS_TOKEN: ${{ secrets.GADFLY_FINDINGS_TOKEN }}
with:
# Inherit the default swarm (3 cloud + Claude Code sonnet/opus/opus:max,
# 5-lens suite) from review-reusable.yml. Only the consumer-specific
# allow-list is set here.
allowed_users: "steve,fizi,dazed"
+99 -19
View File
@@ -1,47 +1,127 @@
name: Build & push image
# Builds the Gadfly reviewer container and pushes it to the Gitea container
# registry. Tag a release (v1, v1.2.0, …) to publish that version + :latest.
# registry. Mirrors mort-ci.yml's build-and-push (BuildKit secrets for private
# module access + the LAN --add-host so the builder can reach the registry).
#
# push to main -> :latest + :sha-<short>
# push tag v* -> :<tag> + :latest
# other branch push -> :branch-<safe> + :sha-<short>
# pull_request -> build only (no push), as a sanity check
#
# Required repo secrets:
# REGISTRY_USER / REGISTRY_PASSWORD Gitea creds with registry push + read
# access to the private majordomo module.
# Optional:
# DISCORD_WEBHOOK_URL build notifications (unset => silent).
on:
push:
branches: [main]
tags: ["v*"]
# Docs/example-only changes don't change the image — skip the rebuild.
# (Path filters are not applied to tag pushes, so `v*` releases always build.)
paths-ignore:
- "**.md"
- "examples/**"
- "LICENSE"
- ".gitignore"
- ".dockerignore"
pull_request:
types: [opened, synchronize, reopened]
paths-ignore:
- "**.md"
- "examples/**"
- "LICENSE"
- ".gitignore"
- ".dockerignore"
workflow_dispatch: {}
concurrency:
group: gadfly-image-${{ github.ref }}
cancel-in-progress: true
env:
IMAGE: gitea.stevedudenhoeffer.com/steve/gadfly
IMAGE_NAME: gitea.stevedudenhoeffer.com/steve/gadfly
jobs:
image:
build-and-push:
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v4
- name: Set up Docker Buildx
run: docker buildx create --use --name gadfly-builder --driver docker-container 2>/dev/null || docker buildx use gadfly-builder
- name: Log in to the registry
run: |
echo "${{ secrets.REGISTRY_PASSWORD }}" \
| docker login gitea.stevedudenhoeffer.com -u "${{ secrets.REGISTRY_USER }}" --password-stdin
if: github.event_name != 'pull_request'
env:
REGISTRY_USER: ${{ secrets.REGISTRY_USER }}
REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
run: echo "${REGISTRY_PASSWORD}" | docker login gitea.stevedudenhoeffer.com -u "${REGISTRY_USER}" --password-stdin
- name: Resolve tags
id: tags
- name: Compute tags
id: meta
run: |
if [ "${{ github.ref_type }}" = "tag" ]; then
echo "version=${{ github.ref_name }}" >> "$GITHUB_OUTPUT"
SHA_SHORT=$(echo "${GITHUB_SHA}" | cut -c1-7)
PUSH=true
if [ "${{ github.event_name }}" = "pull_request" ]; then
# Build-only sanity check; nothing published.
TAGS="${IMAGE_NAME}:pr-${{ github.event.pull_request.number }}"
PUSH=false
elif [ "${{ github.ref_type }}" = "tag" ]; then
TAGS="${IMAGE_NAME}:${GITHUB_REF_NAME},${IMAGE_NAME}:latest"
elif [ "${GITHUB_REF_NAME}" = "main" ]; then
TAGS="${IMAGE_NAME}:latest,${IMAGE_NAME}:sha-${SHA_SHORT}"
else
echo "version=dev-$(echo ${{ github.sha }} | cut -c1-8)" >> "$GITHUB_OUTPUT"
BRANCH_SAFE=$(echo "${GITHUB_REF_NAME}" | sed 's/[^a-zA-Z0-9._-]/-/g; s/--*/-/g; s/^-//; s/-$//')
TAGS="${IMAGE_NAME}:branch-${BRANCH_SAFE},${IMAGE_NAME}:sha-${SHA_SHORT}"
fi
echo "tags=${TAGS}" >> "$GITHUB_OUTPUT"
echo "push=${PUSH}" >> "$GITHUB_OUTPUT"
echo "Tags: ${TAGS} (push=${PUSH})"
- name: Build & push
- name: Notify Discord (started)
if: github.event_name != 'pull_request'
env:
WEBHOOK_URL: ${{ secrets.DISCORD_WEBHOOK_URL }}
run: |
docker build \
--build-arg GIT_USER="${{ secrets.REGISTRY_USER }}" \
--build-arg GIT_TOKEN="${{ secrets.REGISTRY_PASSWORD }}" \
-t "${IMAGE}:${{ steps.tags.outputs.version }}" \
-t "${IMAGE}:latest" \
[ -z "$WEBHOOK_URL" ] && exit 0
MSG="🪰 Gadfly image build #${{ github.run_number }} started on \`${{ github.ref_name }}\` (${{ github.sha }})."
curl -sS -H 'Content-Type: application/json' -d "{\"content\": \"$MSG\"}" "$WEBHOOK_URL" || true
- name: Build and push
env:
REGISTRY_USER: ${{ secrets.REGISTRY_USER }}
REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
run: |
TAG_FLAGS=""
IFS=',' read -ra TAG_ARRAY <<< "${{ steps.meta.outputs.tags }}"
for tag in "${TAG_ARRAY[@]}"; do TAG_FLAGS="$TAG_FLAGS --tag $tag"; done
PUSH_FLAG="--push"
[ "${{ steps.meta.outputs.push }}" = "false" ] && PUSH_FLAG="--output=type=cacheonly"
docker buildx build \
$PUSH_FLAG \
--platform linux/amd64 \
$TAG_FLAGS \
--add-host gitea.stevedudenhoeffer.com:192.168.0.134 \
--secret id=REGISTRY_USER,env=REGISTRY_USER \
--secret id=REGISTRY_PASSWORD,env=REGISTRY_PASSWORD \
--file ./Dockerfile \
.
docker push "${IMAGE}:${{ steps.tags.outputs.version }}"
docker push "${IMAGE}:latest"
- name: Notify Discord (result)
if: always() && github.event_name != 'pull_request'
env:
WEBHOOK_URL: ${{ secrets.DISCORD_WEBHOOK_URL }}
run: |
[ -z "$WEBHOOK_URL" ] && exit 0
if [ "${{ job.status }}" = "success" ]; then
MSG="✅ Gadfly image build #${{ github.run_number }} succeeded. Tags: \`${{ steps.meta.outputs.tags }}\`."
else
MSG="❌ Gadfly image build #${{ github.run_number }} failed. Check Actions logs."
fi
curl -sS -H 'Content-Type: application/json' -d "{\"content\": \"$MSG\"}" "$WEBHOOK_URL" || true
+149
View File
@@ -0,0 +1,149 @@
# Gadfly — REUSABLE adversarial-review workflow (Gitea `workflow_call`).
#
# Centralizes the ~90-line consumer stub so a repo can subscribe to Gadfly with
# a tiny caller. A consumer workflow does:
#
# jobs:
# review:
# if: ... # actor gate for the comment trigger
# uses: steve/gadfly/.gitea/workflows/review-reusable.yml@v1
# secrets: # forward ONLY what the reviewer needs
# OLLAMA_CLOUD_API_KEY: ${{ secrets.OLLAMA_CLOUD_API_KEY }}
# CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
# with: { allowed_users: "..." } # config inputs are optional (see below)
#
# The swarm config (models, specialists, concurrency) is resolved at RUNTIME from,
# in order: a consumer's `with:` input → the owner's user-scope GADFLY_DEFAULT_*
# variable → the image's built-in default. Because variables are injected per-run
# (not part of this cached file), the owner retunes the whole fleet by editing ONE
# variable — see the inputs block and README "Central config via variables".
# Secrets are DECLARED below (workflow_call.secrets) so a
# caller forwards only the credentials the reviewer actually uses — least
# privilege — rather than `secrets: inherit`, which leaks every caller secret
# (registry/deploy/db creds) into this workflow. `secrets: inherit` still works
# if you accept that exposure; the explicit form is recommended. GITEA_TOKEN is
# the automatic job token (no need to forward it).
#
# Advisory only — never blocks a merge. The image is pinned to an immutable
# :sha- tag here (act_runner caches :latest); bump it per Gadfly release.
# Consumers should pin `uses: ...@v1` — a curated release tag moved on deliberate
# releases, so central tuning here propagates without per-consumer edits — or a
# full `@<sha>` for an immutable pin. Avoid `@main` (moves on every push).
name: Gadfly review (reusable)
on:
workflow_call:
# The swarm config (models, specialists, concurrency) is resolved at RUNTIME,
# in priority order: a consumer's explicit `with:` input → the owner's
# user/org-level variable (GADFLY_DEFAULT_*) → the image's built-in default.
# Variables are injected per-run by Gitea (not baked into this file), so the
# owner can retune the whole fleet by editing ONE variable — it propagates even
# though long-lived act_runners CACHE this workflow file by ref (a moved tag is
# NOT re-fetched; only a runtime value or a fresh @<sha> bypasses the cache).
#
# Owner-set user-scope variables (see README "Central config via variables"):
# GADFLY_DEFAULT_MODELS, GADFLY_DEFAULT_SPECIALISTS,
# GADFLY_DEFAULT_PROVIDER_CONCURRENCY, GADFLY_DEFAULT_PROVIDER_LENS_CONCURRENCY,
# GADFLY_ENDPOINT_RAGNAROS (the 4090 Ti endpoint).
# An unset variable + no input → the image default (one model, default suite),
# so a public consumer with neither still gets a sane minimal review.
inputs:
models: { type: string, default: "" } # GADFLY_MODELS — empty falls back to user var GADFLY_DEFAULT_MODELS, then the image default
specialists: { type: string, default: "" } # GADFLY_SPECIALISTS — empty falls back to user var GADFLY_DEFAULT_SPECIALISTS
provider: { type: string, default: "" } # GADFLY_PROVIDER
base_url: { type: string, default: "" } # GADFLY_BASE_URL
provider_concurrency: { type: string, default: "" } # GADFLY_PROVIDER_CONCURRENCY — empty falls back to user var GADFLY_DEFAULT_PROVIDER_CONCURRENCY
provider_lens_concurrency: { type: string, default: "" } # GADFLY_PROVIDER_LENS_CONCURRENCY — empty falls back to user var GADFLY_DEFAULT_PROVIDER_LENS_CONCURRENCY
timeout_secs: { type: string, default: "600" } # GADFLY_TIMEOUT_SECS (per lens)
max_steps: { type: string, default: "14" } # GADFLY_MAX_STEPS
worker_model: { type: string, default: "" } # GADFLY_WORKER_MODEL
allowed_users: { type: string, default: "" } # GADFLY_ALLOWED_USERS (consumer-specific; set in your stub)
trigger_phrase: { type: string, default: "" } # GADFLY_TRIGGER_PHRASE
consolidate: { type: string, default: "" } # GADFLY_CONSOLIDATE — "" => auto (one consensus comment for >=2 models); "0" => one comment per model
inline_review: { type: string, default: "" } # GADFLY_INLINE_REVIEW — "" => on (post a COMMENT-state PR review with inline comments on changed lines); "0" => off
# Job wall-clock cap. 90 as a default: the 5-lens suite across a slow lane
# (claude-code with extended thinking) over two passes can run long.
timeout_minutes: { type: number, default: 90 }
# Declared so callers can forward ONLY the secrets the reviewer needs
# (least privilege) instead of `secrets: inherit`, which would hand this
# workflow every secret in the caller's repo (registry/deploy/db creds the
# review never touches). All optional — an unset/unpassed secret resolves to
# empty, harmless for the providers a given consumer doesn't use. GITEA_TOKEN
# is the automatic job token and need not be declared/forwarded. Named
# endpoints (GADFLY_ENDPOINT_*) come from user/org VARS now, not secrets.
secrets:
OLLAMA_CLOUD_API_KEY: { required: false }
OPENAI_API_KEY: { required: false }
ANTHROPIC_API_KEY: { required: false }
GOOGLE_API_KEY: { required: false }
GADFLY_API_KEY: { required: false }
CLAUDE_CODE_OAUTH_TOKEN: { required: false }
GADFLY_FINDINGS_URL: { required: false }
GADFLY_FINDINGS_TOKEN: { required: false }
# The reusable job posts the review comment, so it needs issues/PR write. Gitea
# caps these by the caller's granted permissions; declaring them here is explicit.
permissions:
contents: read
issues: write
pull-requests: write
jobs:
review:
runs-on: ubuntu-latest
timeout-minutes: ${{ inputs.timeout_minutes }}
steps:
- uses: docker://gitea.stevedudenhoeffer.com/steve/gadfly:sha-3095ebf
env:
# --- event context (from the CALLER's github.*) -------------------
GITEA_API: ${{ github.server_url }}/api/v1/repos/${{ github.repository }}
# github.token is the auto job token from the github CONTEXT (not a
# secret), so it's present even without `secrets: inherit`. Using
# secrets.GITEA_TOKEN here would be empty under explicit secret
# forwarding, since the auto token isn't a forwarded workflow_call secret.
GITEA_TOKEN: ${{ github.token }}
EVENT_NAME: ${{ github.event_name }}
PR: ${{ github.event.pull_request.number || github.event.issue.number || github.event.inputs.pr_number }}
PR_BRANCH: ${{ github.head_ref }}
IS_DRAFT: ${{ github.event.pull_request.draft }}
COMMENT_BODY: ${{ github.event.comment.body }}
COMMENT_ID: ${{ github.event.comment.id }}
ACTOR: ${{ github.actor }}
# --- provider auth (forwarded workflow_call secrets; empty if the caller doesn't forward it) -
OLLAMA_CLOUD_API_KEY: ${{ secrets.OLLAMA_CLOUD_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
GADFLY_API_KEY: ${{ secrets.GADFLY_API_KEY }}
CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
# Named LAN endpoints, defined in user/org vars (format
# "<provider>|<base-url>[|<key>]"). Adding a NEW name still needs a line
# here — a reusable workflow can't enumerate arbitrary vars.GADFLY_ENDPOINT_*.
# NB: vars are NOT masked like secrets — if an endpoint embeds an auth
# token, keep that one a secret instead.
GADFLY_ENDPOINT_M1: ${{ vars.GADFLY_ENDPOINT_M1 }}
GADFLY_ENDPOINT_M5: ${{ vars.GADFLY_ENDPOINT_M5 }}
# ragnaros = the 4090 Ti via its llama-swap proxy. Defined in the user
# var GADFLY_ENDPOINT_RAGNAROS (format "<provider>|<base-url>[|<key>]")
# so the URL can change without editing this file; the matching model is
# ragnaros/qwen3.6-27b in GADFLY_DEFAULT_MODELS. NB: use the un-hyphenated
# `llamaswap` provider spelling in the var — the pinned image needs it.
GADFLY_ENDPOINT_RAGNAROS: ${{ vars.GADFLY_ENDPOINT_RAGNAROS }}
# --- findings telemetry (optional) --------------------------------
GADFLY_FINDINGS_URL: ${{ secrets.GADFLY_FINDINGS_URL }}
GADFLY_FINDINGS_TOKEN: ${{ secrets.GADFLY_FINDINGS_TOKEN }}
# --- config (from inputs; empty => image default) -----------------
GADFLY_MODELS: ${{ inputs.models || vars.GADFLY_DEFAULT_MODELS }}
GADFLY_SPECIALISTS: ${{ inputs.specialists || vars.GADFLY_DEFAULT_SPECIALISTS }}
GADFLY_PROVIDER: ${{ inputs.provider }}
GADFLY_BASE_URL: ${{ inputs.base_url }}
GADFLY_PROVIDER_CONCURRENCY: ${{ inputs.provider_concurrency || vars.GADFLY_DEFAULT_PROVIDER_CONCURRENCY }}
GADFLY_PROVIDER_LENS_CONCURRENCY: ${{ inputs.provider_lens_concurrency || vars.GADFLY_DEFAULT_PROVIDER_LENS_CONCURRENCY }}
GADFLY_TIMEOUT_SECS: ${{ inputs.timeout_secs }}
GADFLY_MAX_STEPS: ${{ inputs.max_steps }}
GADFLY_WORKER_MODEL: ${{ inputs.worker_model }}
GADFLY_ALLOWED_USERS: ${{ inputs.allowed_users }}
GADFLY_TRIGGER_PHRASE: ${{ inputs.trigger_phrase }}
GADFLY_CONSOLIDATE: ${{ inputs.consolidate }}
GADFLY_INLINE_REVIEW: ${{ inputs.inline_review }}
+175
View File
@@ -0,0 +1,175 @@
# Gadfly — Developer Guide
Gadfly (🪰) is an **agentic adversarial code reviewer** that runs in Gitea Actions. On a pull
request it reads the *checked-out repository* with read-only tools, hunts for real problems,
verifies each one against the actual code, and posts its findings as a comment. It is
**advisory only** — it never blocks a merge.
> This is a public, **vibe-coded** project (built largely by an AI agent). Keep that framing
> honest in the README; don't oversell it.
## Project goals (keep changes aligned to these)
1. **Find *real* problems, not nits.** The whole point of the agentic tools + two-pass
recheck is to kill diff-only false positives. Anything that raises the false-positive rate
(or removes verification) works against the project.
2. **Advisory, never blocking.** Gadfly must never fail a CI job for review *content*, never
merge, never deploy. Non-zero exit only on usage/config errors; even then run.sh posts a
notice rather than failing. Do not add it to branch-protection required checks.
3. **Easy to turn on for any repo.** Consumers should need only a ~15-line stub workflow + a
couple of secrets/vars. All real logic lives in the image (`entrypoint.sh`), not in the
consumer's YAML (Gitea's act_runner has weak YAML expression support).
4. **Provider-agnostic.** Powered by [majordomo](https://gitea.stevedudenhoeffer.com/steve/majordomo),
so it can target Ollama (local/cloud), OpenAI, Anthropic, Google, or any
OpenAI/Ollama-compatible endpoint. Don't re-hardcode a single provider.
5. **Portable & self-contained.** `cmd/gadfly` depends only on the Go stdlib, majordomo, and
[executus](https://gitea.stevedudenhoeffer.com/steve/executus) (whose *core*`run`/`compact`/
`model`/`fanout`/`tool` — is itself majordomo+stdlib only, so the binary stays static; do NOT
pull executus's `contrib/store` or any battery that drags in a DB driver). No heavyweight deps,
no coupling to any one consumer repo (e.g. mort). Gadfly is executus's canonical *light* consumer.
## Architecture
```
cmd/gadfly/ the reviewer binary — pure producer of review markdown (stdout)
main.go orchestration: fan specialists out (executus/fanout), each a review pass + recheck
engine.go reviewEngine abstraction: executus run.Executor (majordomo agent loop +
compaction/bounding/budget/critic) vs claude-code CLI shell-out
executus.go executus wiring: tool.Registry over the repo tools, the run.Executor build
(compact + model context-limit threshold + per-PR budget + wrap-up critic)
specialists.go specialist lenses: built-ins, default suite, env + .gadfly.yml resolution
auto.go dynamic `auto` selection: a selector model picks lenses per-diff (may invent)
delegate.go worker-tier delegate_investigation tool (cheap sub-agent does legwork)
consolidate.go verdict parsing + one-comment consolidation (a section per specialist)
model.go provider/model + selector + worker resolution (majordomo.Parse) + endpoint aliases
tools.go the 5 read-only repo tools (read_file/list_dir/grep/find_files/get_diff)
recheck.go second-pass verification prompt + verdict recompute
*_test.go sandbox, recheck, wrap-up, spec/endpoint-parse, specialist-resolution tests
scripts/run.sh fetch PR diff+meta, run the binary, upsert ONE labeled PR comment
scripts/status-board.sh render+upsert ONE live status-board comment (per-model/per-lens progress)
scripts/system-prompt.txt the reviewer persona + verification discipline (generic, not repo-specific)
entrypoint.sh container brains: trigger gating, PR clone, model loop (the logic that
used to live in workflow YAML)
Dockerfile multi-stage; private-module creds via BuildKit secrets never reach the final image
.gitea/workflows/build-image.yml push main → :latest; tag v* → :<tag>+:latest; PR → build-only
.gitea/workflows/review-reusable.yml reusable (workflow_call) review job; resolves swarm config at
RUNTIME: consumer `with:` input → owner user-scope var (GADFLY_DEFAULT_MODELS /
_SPECIALISTS / _PROVIDER_CONCURRENCY / _PROVIDER_LENS_CONCURRENCY, +
GADFLY_ENDPOINT_RAGNAROS) → image default. Vars are injected per-run, so editing
one var retunes the whole fleet even though long-lived act_runners CACHE this file
by ref (a moved tag is NOT re-fetched — only a runtime value or a fresh @<sha>
bypasses the cache). Consumers subscribe with an ~8-line caller forwarding only the
secrets the reviewer needs and pinned to an immutable @<sha> (Phase 4);
gadfly's own adversarial-review.yml is a thin caller of it (dogfoods the path).
examples/ copy-paste consumer stub workflows for different providers
```
**Data flow:** consumer stub workflow → container `entrypoint.sh` (gate + clone) →
`scripts/run.sh` (per model) → `cmd/gadfly` binary (agentic review) → markdown → run.sh
upserts a PR comment as `gitea-actions`.
**Two passes:** a *review* pass drafts findings; an adversarial *recheck* pass independently
re-verifies each finding against the code and drops the unconfirmed ones, recomputing the
verdict. Verdict is one of: `No material issues found` / `Minor issues` / `Blocking issues found`.
## Build / test
```sh
go build ./cmd/gadfly # needs read access to the private majordomo + executus modules
go test ./...
gofmt -l cmd/ # must be clean
docker build -t gadfly:dev --secret id=REGISTRY_USER,env=REGISTRY_USER --secret id=REGISTRY_PASSWORD,env=REGISTRY_PASSWORD .
```
Run it locally against a real diff without CI:
```sh
git -C /path/to/repo diff main > /tmp/x.diff
GADFLY_PROVIDER=ollama GADFLY_MODEL=qwen2.5-coder:7b \
GADFLY_REPO_DIR=/path/to/repo GADFLY_DIFF_FILE=/tmp/x.diff \
GADFLY_SYSTEM_FILE=scripts/system-prompt.txt ./gadfly
```
## Release / deploy
- **Push to `main`** → CI builds and pushes `:latest` (+ `:sha-<short>`).
- **Tag `v*`** → publishes `:<tag>` (+ `:latest`). Pin consumers to `:vN` for stability.
- Required CI secrets: `REGISTRY_USER` / `REGISTRY_PASSWORD` (registry push + read access to the
private majordomo module). Optional `DISCORD_WEBHOOK_URL`.
## Configuration
The full env reference lives in the **README** (`Specialists`, `Models & providers`,
`Configuration`). Provider selection: `GADFLY_PROVIDER` (default `ollama-cloud`),
`GADFLY_MODEL`/`GADFLY_MODELS`, `GADFLY_BASE_URL`, `GADFLY_API_KEY`. Named endpoint aliases via
`GADFLY_ENDPOINT_<NAME>` / `GADFLY_ALIAS_<NAME>` (http-capable) and majordomo `LLM_*` DSNs
(HTTPS-only).
**Specialists (the swarm):** the reviewer runs a suite of focused lenses, one consolidated
comment with a section each. Default suite = security/correctness/maintainability/performance/
error-handling; opt-in built-ins = tests/docs/conventions/improvements. Select via
`GADFLY_SPECIALISTS` (csv or `all`); define/override via `GADFLY_SPECIALIST_<NAME>` env or a repo
`.gadfly.yml` (`specialists:` + `define:`). See `cmd/gadfly/specialists.go`. Cost ≈
specialists × models × 2 passes — the **image/entrypoint** default stays minimal (one model) for
that reason; the **reusable workflow** (`review-reusable.yml`) deliberately ships a heavier
opinionated default swarm (3 cloud + Claude Code, 5 lenses) for steve's own fleet, which consumers
inherit or override per-input.
**Dynamic `auto`** (`GADFLY_SPECIALISTS=auto`): a selector (`GADFLY_SELECTOR_MODEL` or the review
model) picks lenses per-diff and may invent ad-hoc ones (`cmd/gadfly/auto.go`). **Worker-tier**
(`GADFLY_WORKER_MODEL`): a `delegate_investigation` tool offloads grep/read legwork to a cheap
sub-agent (`cmd/gadfly/delegate.go`).
**Tested vs untested:** only the Ollama paths (local + OpenAI-compatible pointed at Ollama)
are actually exercised. OpenAI/Anthropic/Google come from majordomo's abstraction and are
**untested** (no spend). Keep the README honest about this; update it if that changes.
## When making changes — maintenance rules
- **Keep the README and `examples/` current.** Any change to env vars, flags, defaults,
triggers, provider support, or the consumer stub MUST be reflected in `README.md` and the
relevant files under `examples/` in the *same* change. The README's `Configuration` table,
the `Models & providers` table, and the example workflows are the contract users rely on —
stale docs are a bug.
- **Preserve the advisory-only invariant** (goal #2). If you touch exit codes or the workflow,
re-confirm a review can never fail/block a consumer's CI.
- **Don't add mort-specific (or any single-consumer) assumptions** to the binary or system
prompt. The system prompt is intentionally generic; repo-specific conventions should be
discovered by the agent at runtime (it can read the repo's own CONTRIBUTING/CLAUDE.md), not
hardcoded here.
- **Keep secrets out of image layers.** Private-module creds flow via BuildKit `--mount=type=secret`
in the build stage only; never bake them into the final image or commit them.
- Add a test when you add logic (see the `*_test.go` patterns). Keep `gofmt` clean and `go vet` quiet.
## Lessons
- majordomo's `LLM_*` env DSNs are **HTTPS-only** (`DSN.BaseURL()` forces `https://`), so they
can't express a plaintext local Ollama. That's why Gadfly adds the http-capable
`GADFLY_ENDPOINT_<NAME>="provider|base-url[|key]"` mechanism (see `cmd/gadfly/model.go`).
- Gitea `vars`/`secrets` are **not** auto-exposed as env in a job — the consumer stub must map
each one explicitly in its `env:` block (dynamic alias names can't be auto-enumerated).
- **`uses: docker://…:latest` is CACHED by act_runner** — a freshly-pushed `:latest` is often
NOT re-pulled, so the job silently runs the previous image. For a run that must use a specific
build (e.g. validating a just-pushed fix), pin the consumer stub to the immutable
`:sha-<short>` tag the build publishes, not `:latest`.
- **Concurrency is per-provider** (`entrypoint.sh`): each provider is a lane, lanes run in
parallel, `cap` (from `GADFLY_PROVIDER_CONCURRENCY` else `GADFLY_CONCURRENCY`, default 1) bounds
models-at-once within a lane. The review timeout (`GADFLY_TIMEOUT_SECS`) is **per-lens**, not
shared across the suite — a slow model can't starve later lenses (the original timeout bug).
- **Large-PR token burn**: the agent loop re-sends the whole transcript every step, so a giant
diff (the old `get_diff` dumped it untruncated, and it was embedded in both the review and
recheck task) was re-transmitted ~steps × lenses × passes × models times — a ~250 K-token PR
could drain a metered usage block in minutes. Fixed in three size-gated layers (small PRs
untouched): paginated `get_diff` + `executus/compact` compaction in the binary; an
`entrypoint.sh` downshift above `GADFLY_HUGE_DIFF_BYTES` (one cheap model, fewer lenses/steps,
no recheck); and a swarm-wide `GADFLY_PR_BUDGET_SECS` wall-clock backstop. Compaction's threshold
is intentionally LOW (`GADFLY_COMPACT_RATIO` 0.45, not executus's 0.7) because the burning
transcript on the embedded path rarely reaches 0.7×context.
- **executus re-platform**: the in-process review path runs through `executus/run`'s `run.Executor`
(compaction, run-bounding, `Ports.Budget`, the wrap-up nudge as `Ports.Critic`), wiring it in
`cmd/gadfly/executus.go`. Gadfly KEEPS its own `model.go` resolution (so `GADFLY_ENDPOINT_<NAME>`
http aliases + the claude-code engine survive) and only hands `run.Executor` the already-resolved
model via a trivial resolver — do NOT route review-model resolution through
`model.ParseModelForContext` (it bypasses gadfly's endpoint aliases). `run.Result` exposes no
transcript, so the old transcript-based forced-finalization fallback is gone; the wrap-up critic
nudge is the remaining "always emit something" mechanism. The claude-code engine still shells out
and is unaffected.
+23 -13
View File
@@ -1,30 +1,40 @@
# syntax=docker/dockerfile:1
#
# Multi-stage so the private-module access token used to fetch the majordomo
# dependency lives ONLY in the build stage and never lands in the final image.
# Multi-stage so the private-module credentials (used to fetch the majordomo
# dependency) live ONLY in the build stage via BuildKit secrets and never land
# in the final image. Mirrors mort's Dockerfile secret idiom.
FROM golang:1.26 AS build
ARG GIT_HOST=gitea.stevedudenhoeffer.com
ARG GIT_USER=
ARG GIT_TOKEN=
ENV CGO_ENABLED=0 \
GOFLAGS=-mod=mod \
GOSUMDB=off
GOSUMDB=off \
GOTOOLCHAIN=auto
ENV GOPRIVATE=${GIT_HOST}/* GONOSUMDB=${GIT_HOST}/*
WORKDIR /src
# Private Go module access (majordomo). Token is confined to this stage.
RUN if [ -n "$GIT_TOKEN" ]; then \
git config --global url."https://${GIT_USER}:${GIT_TOKEN}@${GIT_HOST}/".insteadOf "https://${GIT_HOST}/"; \
fi
COPY go.mod go.sum ./
RUN go mod download
RUN --mount=type=secret,id=REGISTRY_USER \
--mount=type=secret,id=REGISTRY_PASSWORD \
--mount=type=cache,target=/go/pkg/mod \
git config --global url."https://$(cat /run/secrets/REGISTRY_USER):$(cat /run/secrets/REGISTRY_PASSWORD)@${GIT_HOST}/".insteadOf "https://${GIT_HOST}/" \
&& go mod download
COPY . .
RUN go build -trimpath -ldflags="-s -w" -o /out/gadfly ./cmd/gadfly
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
go build -trimpath -ldflags="-s -w" -o /out/gadfly ./cmd/gadfly
FROM alpine:3.20
RUN apk add --no-cache bash git curl jq ca-certificates
# procps provides pkill/pgrep, which entrypoint.sh's per-PR wall-clock backstop
# (GADFLY_PR_BUDGET_SECS) uses to stop the review subtrees — busybox's applets
# are not guaranteed to include them.
RUN apk add --no-cache bash git curl jq ca-certificates nodejs npm procps
# Bundle the Claude Code CLI so the `claude-code` review engine works out of the
# box (GADFLY_MODELS=claude-code or claude-code/<model>). This adds Node + the
# CLI to the image (notably larger); ollama-only users pay the size but nothing
# else. Auth is provided at runtime via CLAUDE_CODE_OAUTH_TOKEN / ANTHROPIC_API_KEY.
RUN npm install -g @anthropic-ai/claude-code && npm cache clean --force
COPY --from=build /out/gadfly /usr/local/bin/gadfly
COPY scripts /app/scripts
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh /app/scripts/run.sh /usr/local/bin/gadfly
RUN chmod +x /entrypoint.sh /app/scripts/run.sh /app/scripts/status-board.sh /usr/local/bin/gadfly
ENTRYPOINT ["/entrypoint.sh"]
+391 -12
View File
@@ -36,11 +36,18 @@ or **Blocking issues found**.
Gadfly ships as a container image, so consuming repos don't build anything — they just run
it. Drop one file in your repo and set a couple of secrets/vars:
1. Copy [`examples/adversarial-review.yml`](examples/adversarial-review.yml) to
`.gitea/workflows/adversarial-review.yml` in your repo.
1. Copy a stub from [`examples/`](examples/) to `.gitea/workflows/adversarial-review.yml` in
your repo. Two flavors: the slim [`reusable.yml`](examples/reusable.yml) — a tiny caller of
Gadfly's **reusable workflow** (`uses: steve/gadfly/.gitea/workflows/review-reusable.yml@…`,
forwarding only the secrets the reviewer needs), whose **default swarm is set centrally via owner
variables** (see [Central config via variables](#central-config-via-variables)) and inherited by omitting `with:` — or the full self-contained
[`adversarial-review.yml`](examples/adversarial-review.yml) (Ollama Cloud default, with inline
notes for every provider / local Ollama / OpenAI-compatible / endpoint aliases). See the
[examples index](examples/README.md).
2. Add repo config:
- **secret** `OLLAMA_CLOUD_API_KEY` — your [Ollama Cloud](https://ollama.com) key (empty
⇒ Gadfly posts a harmless "not configured" notice instead of reviewing).
⇒ Gadfly posts a harmless "not configured" notice instead of reviewing). *Not needed if
you point Gadfly at a different provider — see [Models & providers](#models--providers).*
- **var** `OLLAMA_REVIEW_MODELS` *(optional)* — comma-separated model ids
(default `qwen3-coder:480b-cloud,gpt-oss:120b-cloud`). One comment per model.
- **var** `GADFLY_ALLOWED_USERS` *(optional)* — who may re-trigger via comment; empty ⇒
@@ -49,6 +56,264 @@ it. Drop one file in your repo and set a couple of secrets/vars:
`GITEA_TOKEN` is provided automatically by Actions; comments post as the `gitea-actions`
user, scoped to that repo — no bot account needed.
## Models & providers
Gadfly is built on [majordomo](https://gitea.stevedudenhoeffer.com/steve/majordomo), so the
reviewer model is not hard-wired — it can target anything majordomo supports. Pick a provider
by setting `GADFLY_PROVIDER` (used to prefix bare model ids); point at a custom endpoint with
`GADFLY_BASE_URL`; supply a key with `GADFLY_API_KEY` or the provider's standard env var. A
`GADFLY_MODEL`/`GADFLY_MODELS` value that already contains a `provider/` prefix (or is a
majordomo failover chain / alias) is used verbatim.
| Provider | `GADFLY_PROVIDER` | Key env | Status |
|----------|-------------------|---------|--------|
| **Ollama Cloud** (default) | `ollama-cloud` | `OLLAMA_API_KEY` / `OLLAMA_CLOUD_API_KEY` | ✅ in active use |
| **Local Ollama** | `ollama` | none (`OLLAMA_HOST` or `GADFLY_BASE_URL` for a remote daemon) | ✅ tested |
| **[foreman](https://gitea.stevedudenhoeffer.com/steve/foreman)** (native-Ollama queue daemon) | `foreman` + `GADFLY_BASE_URL`, or a `GADFLY_ENDPOINT_*` / `LLM_*` `foreman://` entry | optional bearer (via the endpoint/DSN) | ✅ native-Ollama path |
| **[llama-swap](https://github.com/mostlygeek/llama-swap)** (model-swapping proxy) | `llama-swap`/`llama-swaps` (un-hyphenated `llamaswap`/`llamaswaps` also accepted) + `GADFLY_BASE_URL` or a `GADFLY_ENDPOINT_*` entry, or an `LLM_*` `llama-swap://` / `llama-swaps://` DSN | optional bearer | ⚠️ wired, **untested** |
| **OpenAI-compatible** (incl. local Ollama's `/v1`) | `openai` + `GADFLY_BASE_URL` | `OPENAI_API_KEY` (any non-empty for Ollama) | ✅ tested against Ollama |
| **OpenAI** | `openai` | `OPENAI_API_KEY` | ⚠️ wired, **untested** |
| **Anthropic** | `anthropic` | `ANTHROPIC_API_KEY` | ⚠️ wired, **untested** |
| **Google (Gemini)** | `google` | `GOOGLE_API_KEY` / `GEMINI_API_KEY` | ⚠️ wired, **untested** |
> ### 🧪 Honest status
> Only the **Ollama** paths above are actually exercised. The OpenAI / Anthropic / Google
> providers come "for free" from majordomo's abstraction and *should* work, but I haven't
> spent money verifying them — treat them as untested. The OpenAI-**compatible** path **is**
> tested, because you can point it at a local Ollama (`GADFLY_BASE_URL=http://localhost:11434/v1`)
> and exercise the exact same code an OpenAI/OpenRouter endpoint would hit, for free. If you
> try a cloud provider and it works (or doesn't), please open an issue.
### Claude Code engine (`claude-code`)
Besides the majordomo model loop, Gadfly can review through the **[Claude Code](https://claude.com/claude-code)
CLI**: for each lens it shells out to `claude -p` *inside the checked-out repo*, so Claude Code
uses its **own** read tools (Read/Grep/Glob) to verify findings against real code, then Gadfly
parses the result and runs the same verdict-parse → recheck → consolidate → emit pipeline. The
CLI is bundled in the image (Node + `@anthropic-ai/claude-code`).
Select it as a model id — bare `claude-code` (CLI default model) or `claude-code/<model>` (the
suffix becomes `--model`, e.g. `claude-code/sonnet`, `claude-code/opus`). An optional
`:<thinking>` suffix forces an extended-thinking budget for that reviewer — `:max` (the high
"ultrathink" tier) or `:<n>` for a specific token budget — so you can run the same model at two
thinking depths as separate reviewers:
```yaml
GADFLY_MODELS: "claude-code/sonnet,claude-code/opus,claude-code/opus:max"
```
The thinking budget is applied via the `MAX_THINKING_TOKENS` env on the CLI subprocess; it's
best-effort (a no-op if the installed CLI build doesn't honor it).
Auth is read from the environment: the default is a **Pro/Max subscription** via
`CLAUDE_CODE_OAUTH_TOKEN` (from `claude setup-token`; no `--bare`), falling back to
`ANTHROPIC_API_KEY`. Don't set both. Tuning knobs (all optional):
| Env | Default | Meaning |
|-----|---------|---------|
| `GADFLY_CLAUDE_MODEL` | *(from the spec suffix)* | overrides the `--model` value |
| `GADFLY_CLAUDE_PERMISSION_MODE` | `plan` | `--permission-mode` (read-only `plan` keeps it from editing) |
| `GADFLY_CLAUDE_ALLOWED_TOOLS` | *(unset)* | `--allowedTools` value, passed verbatim (e.g. `Read,Grep,Glob`) |
| `GADFLY_CLAUDE_EXTRA_ARGS` | *(unset)* | extra CLI args, **whitespace-split** (no shell quoting) and appended after the defaults (e.g. `--max-turns 30`) |
| `GADFLY_CLAUDE_BIN` | `claude` | CLI binary path |
> These are **operator** knobs (workflow env), not PR-author input. Because
> `GADFLY_CLAUDE_EXTRA_ARGS` is appended *after* the defaults, it can override the
> read-only `--permission-mode plan` (e.g. passing `--permission-mode acceptEdits`),
> so keep it read-only unless you mean otherwise. It's whitespace-split, so values
> can't contain spaces — use `GADFLY_CLAUDE_ALLOWED_TOOLS` / `_PERMISSION_MODE` /
> `_MODEL` for those. The subprocess runs with a **minimal environment** (its auth
> token + `PATH`/`HOME`/locale/`GADFLY_CLAUDE_*`), not the runner's full env, so the
> Gitea token and provider keys aren't handed to the CLI.
**Alternate backends (example only, not validated here).** Because the subprocess env forwards
`ANTHROPIC_*` and `CLAUDE_*`, you can point the same engine at a non-Anthropic backend by setting
`ANTHROPIC_BASE_URL` (and `ANTHROPIC_AUTH_TOKEN`/`ANTHROPIC_API_KEY`) to an **Anthropic-API-compatible
proxy** — e.g. [claude-code-router](https://github.com/musistudio/claude-code-router) or LiteLLM in
front of Ollama — to run *Ollama models through Claude Code's harness* and compare it against the
native majordomo loop. Whether tool-use survives a given proxy/backend varies, so this is documented
as an example, not wired or tested here.
> **The Pro/Max path is dogfooded but otherwise lightly tested.** `claude-code/sonnet` now runs on
> gadfly's own PRs (see `.gitea/workflows/adversarial-review.yml`), but treat the engine as new —
> and note that subscription auth in automated CI is a gray area in Anthropic's terms. `auto`
> specialist selection and the `delegate_investigation` worker are majordomo-only and are skipped
> with this engine (Claude Code does its own legwork).
### Endpoint aliases via env vars
For multiple named backends (e.g. a couple of Ollama boxes on your LAN), register them by
name with env vars and then reference `name/model` in `GADFLY_MODEL`/`GADFLY_MODELS`:
```sh
# http-capable (Gadfly-native) — base URL used verbatim, so plaintext LAN works:
GADFLY_ENDPOINT_BIGBOX="ollama|http://192.168.1.50:11434"
GADFLY_ENDPOINT_GPU="openai|http://gpu.lan:8000/v1|sk-local"
GADFLY_ENDPOINT_M1="foreman|http://foreman-m1:8080|tok" # native-Ollama queue daemon
GADFLY_MODELS="bigbox/qwen2.5-coder:7b,gpu/llama3.1,m1/qwen3:14b"
# pure spec alias (a model, or a failover chain):
GADFLY_ALIAS_FAST="bigbox/qwen2.5-coder:7b,ollama-cloud/gpt-oss:120b-cloud"
GADFLY_MODEL="fast"
```
`<NAME>` is lowercased to form the registry name (`GADFLY_ENDPOINT_BIGBOX``bigbox`). This
is the same idea as majordomo's built-in **`LLM_*` env DSNs** (`LLM_BIGBOX=ollama://tok@host`,
`LLM_M1=foreman://tok@host`), which Gadfly also honors — but those are **HTTPS-only**, so for a
plaintext local Ollama or `http://` foreman use `GADFLY_ENDPOINT_*` instead.
> **Gitea Actions note:** repo `vars`/`secrets` aren't auto-exposed as env — add each alias to
> the stub workflow's `env:` block, e.g. `GADFLY_ENDPOINT_BIGBOX: ${{ vars.GADFLY_ENDPOINT_BIGBOX }}`.
## Specialists (the review swarm)
Instead of one generic reviewer, Gadfly runs a **suite of specialists** — each a focused lens
with its own review (+recheck) pass — and merges them into **one comment**, a collapsible
section per lens, led by an overall verdict (the worst across lenses; the optional
`improvements` lens never escalates it).
**Default suite** (when nothing is configured):
`security`, `correctness`, `maintainability` (code cleanliness), `performance`, `error-handling`.
**Also built in** (opt-in by name): `tests`, `docs`, `conventions`, and `improvements`
(strict & quiet — at most 12 high-value, non-blocking suggestions, silent otherwise).
Select which run with **`GADFLY_SPECIALISTS`** (comma-separated names, or `all`):
```yaml
GADFLY_SPECIALISTS: "security,correctness,maintainability,tests"
```
**Define your own** — two ways, which compose (env overrides file overrides built-ins):
```yaml
# 1. env: GADFLY_SPECIALIST_<NAME>="<focus>" (also overrides a built-in by reusing its name)
GADFLY_SPECIALIST_MIGRATIONS: "Review DB migrations for destructive or unindexed changes."
GADFLY_SPECIALISTS: "security,correctness,migrations"
```
```yaml
# 2. a repo .gadfly.yml at the repo root (version-controlled). See examples/.gadfly.yml:
specialists: [security, correctness, maintainability, migrations]
define:
- name: migrations
title: "🗃️ DB migrations"
focus: "Review schema migrations for destructive ops, missing indexes, table locks."
```
**Dynamic selection (`auto`):** set `GADFLY_SPECIALISTS: auto` and a selector model reads the
changed files + PR description and picks only the lenses that materially apply (and may invent
an ad-hoc one — e.g. a "migrations" lens for a schema change). The selector is
`GADFLY_SELECTOR_MODEL` if set (a cheap tier is ideal), else the review model. Capped and
de-duplicated; falls back to the default suite if selection fails.
**Worker-tier delegation:** set `GADFLY_WORKER_MODEL` (a cheap/fast model) to give every
reviewer a `delegate_investigation` tool — it offloads mechanical legwork (trace all callers,
gather every usage, check a pattern across files) to a worker sub-agent that returns a concise,
evidence-cited digest, so the expensive model reasons over summaries instead of raw file dumps.
Unset = no delegation (current behavior).
> **Cost:** each specialist is its own review+recheck, so cost ≈ *specialists × models × 2*.
> The default suite runs on a **single** model. Trim with `GADFLY_SPECIALISTS`, let `auto` pick
> only what a diff needs, and point heavy legwork at a cheap `GADFLY_WORKER_MODEL`.
### Concurrency (per-provider lanes)
With multiple models, each **provider** is its own lane and lanes run in **parallel**, so a fast
cloud provider isn't stuck behind a slow local box. Within a lane, at most `cap` models run at
once — `cap` comes from `GADFLY_PROVIDER_CONCURRENCY` (a `provider=N` map) else `GADFLY_CONCURRENCY`
(default `1`). The timeout is **per-lens** (`GADFLY_TIMEOUT_SECS`), so a slow model on one lens
can't starve the others.
```yaml
# One local box (serial — it serves one model at a time) + 3 cloud reviews at once,
# both lanes running concurrently:
GADFLY_PROVIDER_CONCURRENCY: "ollama-cloud=3,m1pro=1"
GADFLY_MODELS: "m1pro/qwen3:14b,qwen3-coder:480b-cloud,gpt-oss:120b-cloud"
```
A model's provider is the spec's first segment (`m1pro/…``m1pro`), or `GADFLY_PROVIDER`/
`ollama-cloud` for a bare id. Default (`cap 1`) keeps a single-provider pool fully sequential.
**Lens fan-out (within a model).** By default the specialist lenses run **sequentially** inside
each model (`GADFLY_LENS_CONCURRENCY=1`). Raise it to overlap the independent per-lens
review+recheck passes — the model then posts its consolidated comment as soon as its lenses
finish (so with sequential models, results stream in per model and per-model timings stay
clean). Like the model cap, it's **per-provider configurable**: `GADFLY_PROVIDER_LENS_CONCURRENCY`
takes a `provider=N` map keyed by the **same provider lanes** as `GADFLY_PROVIDER_CONCURRENCY`,
falling back to the `GADFLY_LENS_CONCURRENCY` scalar (default `1`). **It multiplies with the
model cap:** total in-flight requests ≈ *models-at-once × lenses-at-once*, so to fan lenses out
without oversubscribing a backend, keep its model cap low and raise its lens cap:
```yaml
# Per provider: cloud runs one model at a time but fans its 3 lenses out (3 concurrent requests);
# the slow local box stays fully serial. Both provider lanes still run in parallel.
GADFLY_PROVIDER_CONCURRENCY: "ollama-cloud=1,m1=1"
GADFLY_PROVIDER_LENS_CONCURRENCY: "ollama-cloud=3,m1=1"
GADFLY_SPECIALISTS: "security,correctness,error-handling"
```
### Live status board
When several models (each with several lenses) review a PR, the individual findings land in
**one comment per model** — but while that's in flight all you'd see is a row of
`⏳ Reviewing…` placeholders. So Gadfly also upserts **one consolidated status-board comment**
that aggregates every model's per-lens progress as it happens:
```
## 🪰 Gadfly — live review status
1/3 reviewers finished · updated 2026-06-27 18:14:56Z
#### `glm-5.2:cloud` · ollama-cloud — ⏳ 2/4 lenses
- ✅ security — No material issues found
- 🔄 correctness — running
- ⏸️ performance — queued
```
Each model process publishes its lenses (queued → running → finished + verdict) to a small
JSON file, and a background renderer in `entrypoint.sh` re-renders + upserts the single comment
every `GADFLY_STATUS_POLL_SECS` (default 12s) until the swarm finishes. It's advisory and
best-effort — the per-model findings comments are unaffected — and entirely separate from those.
Turn it off with `GADFLY_STATUS_BOARD=0`.
### Consensus consolidation
With **two or more models**, posting one comment each means a reader faces N walls of prose that
mostly agree. Instead Gadfly consolidates: every model writes its findings to a shared file, and
after the whole swarm finishes a single pass clusters those findings by location, counts **how
many models independently flagged each one**, and posts **one consensus comment**:
```
## 🪰 Gadfly review — consensus across 7 models
**Verdict: Blocking issues found** · 9 findings (3 with multi-model agreement)
| | Finding | Where | Models | Lens |
|--|--|--|--|--|
| 🔴 | Auth bypass: token not verified | `auth/login.go:42` | 6/7 | security |
| 🟠 | Unbounded retry loop | `sync/worker.go:88` | 3/7 | error-handling |
<details><summary>4 single-model findings (lower confidence)</summary> … </details>
<details><summary>Per-model detail</summary> … each model's full review, folded … </details>
```
Cross-model agreement is the strongest real-vs-false-positive signal available, so findings are
ranked by it (a lone low-severity finding folds away; a lone *critical* still surfaces). The
per-model comments are suppressed in this mode — each model's full review is preserved, folded,
inside the consensus comment — and nothing is lost: if consolidation can't run, Gadfly falls
back to posting the per-model comments. Controlled by `GADFLY_CONSOLIDATE`: `auto` (default — on
for ≥2 models), `1` (force on), `0` (force off, one comment per model). Single-model runs are
unaffected.
**Inline PR review.** Alongside the consensus comment, Gadfly also posts a single Gitea **pull
review** (state `COMMENT` — advisory, **never** request-changes or approve, so it can't block a
merge) whose inline comments anchor each consensus finding to the exact changed line it's about.
Only findings that land on a line in the diff are anchored (Gitea rejects comments off the diff);
the rest stay in the consensus comment. A re-run replaces the previous review instead of stacking.
It's the "reviewer integrated with Gitea" without the blocking — turn it off with
`GADFLY_INLINE_REVIEW=0`.
### Triggers
1. A **new/reopened/ready** non-draft PR — automatic.
@@ -58,20 +323,55 @@ user, scoped to that repo — no bot account needed.
(Pushing new commits does *not* auto-re-review — comment `@gadfly review` after pushing
fixes. This keeps usage down.)
> **Comment trigger needs the workflow on your default branch.** Gitea runs `issue_comment`
> workflows from the **default branch**, so `@gadfly review` only works once this stub is
> merged to `main` (the `pull_request` auto-trigger works from the PR branch immediately).
>
> **Security:** the example stubs gate the comment trigger with a job-level
> `if: github.event_name != 'issue_comment' || github.actor == '<you>'` so an untrusted
> commenter can't start a secret-bearing run — edit it to your maintainers and keep it in
> sync with `GADFLY_ALLOWED_USERS` (the in-container check). `@gadfly review` is plain-text
> matched (configurable via `GADFLY_TRIGGER_PHRASE`), so no bot account is required; comments
> post as `gitea-actions`.
## How it's packaged
```
cmd/gadfly/ the agentic reviewer binary (majordomo + Ollama Cloud); zero deps beyond stdlib + majordomo
scripts/run.sh fetches the PR diff, runs the reviewer, upserts one labeled comment
scripts/status-board.sh renders + upserts the single live status-board comment (per-lens progress)
scripts/system-prompt.txt the reviewer persona + verification discipline
entrypoint.sh the container brains: trigger gating, clone, model loop (logic lives here, not in YAML)
Dockerfile multi-stage; the build-time module token never reaches the final image
.gitea/workflows/build-image.yml tags v* → build & push the image
Dockerfile multi-stage; build-time module creds (BuildKit secrets) never reach the final image
.gitea/workflows/build-image.yml push to main → :latest; tag v* → :<tag> + :latest
examples/ the ~15-line stub a consuming repo drops in
```
The image is published to `gitea.stevedudenhoeffer.com/steve/gadfly`. Push a `v*` tag to
build and publish a new version (and `:latest`).
The image is published to `gitea.stevedudenhoeffer.com/steve/gadfly`. Every push to `main`
rebuilds and republishes `:latest` (plus `:sha-<short>`); pushing a `v*` tag publishes that
pinned version (plus `:latest`). Pin full-stub consumers to a `:vN` image tag for stability, or track
`:latest` to ride main. **Reusable-workflow consumers should pin the workflow ref to an immutable
`review-reusable.yml@<sha>`** — long-lived act_runners *cache the workflow file by ref*, so a moved tag
(`@v1`) or `@main` is often **not** re-fetched and silently runs a stale copy. A fresh `@<sha>` is the
only reliable way to roll out a *structural* change to the reusable.
### Central config via variables
So you don't have to re-pin every consumer just to retune the swarm, the reusable resolves its config
at **runtime**`with:` input → owner **user/org-level variable** → image default — and variables are
injected per-run (not part of the cached file), so changing one variable propagates to every consumer
on its next review **without** a re-pin or a tag move:
| Variable (user/org scope) | Sets |
|---|---|
| `GADFLY_DEFAULT_MODELS` | `GADFLY_MODELS` (csv) |
| `GADFLY_DEFAULT_SPECIALISTS` | the lens suite |
| `GADFLY_DEFAULT_PROVIDER_CONCURRENCY` | models-at-once per provider |
| `GADFLY_DEFAULT_PROVIDER_LENS_CONCURRENCY` | lenses-at-once per provider |
| `GADFLY_ENDPOINT_RAGNAROS` | a named endpoint, e.g. `llamaswap\|https://host` |
Adding a *new* named endpoint still needs a one-line reusable edit (Gitea can't auto-expose arbitrary
`vars.GADFLY_ENDPOINT_*`); the values of already-wired ones are pure variables.
## Configuration (advanced)
@@ -79,20 +379,99 @@ The reviewer binary reads these (the stub/entrypoint set sane defaults):
| Env | Default | Meaning |
|-----|---------|---------|
| `OLLAMA_API_KEY` | — | Ollama Cloud bearer key (required for real reviews) |
| `GADFLY_MODEL` | — | model id |
| `GADFLY_MODEL` | — | model id, or `provider/model` spec, or majordomo alias/chain |
| `GADFLY_PROVIDER` | `ollama-cloud` | provider prefix for a bare model id |
| `GADFLY_BASE_URL` | — | override endpoint (OpenAI/Ollama-compatible servers) |
| `GADFLY_API_KEY` | — | provider key; falls back to the provider's standard env |
| `claude-code` model id | — | route a model through the bundled Claude Code CLI (`claude-code` / `claude-code/<model>`); see [Claude Code engine](#claude-code-engine-claude-code) for its `GADFLY_CLAUDE_*` knobs |
| `GADFLY_SPECIALISTS` | default suite | csv of lenses, `all`, or `auto` (dynamic selection) |
| `GADFLY_SELECTOR_MODEL` | review model | model that picks lenses in `auto` mode |
| `GADFLY_WORKER_MODEL` | — | cheap model for `delegate_investigation`; unset = no delegation |
| `GADFLY_WORKER_MAX_STEPS` | 8 | tool-step cap for a delegated worker run |
| `GADFLY_CONCURRENCY` | 1 | default max models run at once **per provider** |
| `GADFLY_PROVIDER_CONCURRENCY` | — | per-provider overrides, e.g. `ollama-cloud=3,m1pro=1` |
| `GADFLY_LENS_CONCURRENCY` | 1 | specialist lenses run at once **within a model** (× model cap = total in-flight) |
| `GADFLY_PROVIDER_LENS_CONCURRENCY` | — | per-provider lens overrides, same lanes as `GADFLY_PROVIDER_CONCURRENCY`, e.g. `ollama-cloud=3,m1=1` |
| `GADFLY_MAX_STEPS` | 24 | review-pass tool-step cap |
| `GADFLY_TIMEOUT_SECS` | 300 | deadline **per specialist lens** (review+recheck) |
| `GADFLY_RECHECK` | on | set `0`/`false` to skip the recheck pass |
| `GADFLY_RECHECK_MAX_STEPS` | 16 | recheck-pass step cap |
| `GADFLY_TIMEOUT_SECS` | 300 | overall deadline (both passes) |
| `GADFLY_MAX_DIFF_CHARS` | 60000 | diff chars embedded in the prompt (full diff via `get_diff`) |
| `GADFLY_MAX_DIFF_CHARS` | 60000 | diff chars embedded in the **review** prompt (the full diff is reachable via the paginated `get_diff` tool, scoped per file with its `path` arg) |
| `GADFLY_RECHECK_DIFF_CHARS` | 20000 | diff chars embedded in the **recheck** prompt (smaller — the recheck pages `get_diff` for the hunks it verifies) |
| `GADFLY_COMPACT` | on | context compaction (via [executus](https://gitea.stevedudenhoeffer.com/steve/executus)): fold the transcript's runaway middle into a summary as it nears the model's context window, so a big diff + accumulating tool output can't balloon every step. `0` disables |
| `GADFLY_COMPACT_RATIO` | 0.45 | fraction of the model's context window at which compaction fires |
| `GADFLY_COMPACT_MODEL` | worker, else review model | cheap model the compactor uses to summarize the folded middle |
| `GADFLY_COMPACT_KEEP_RECENT` | 8 | most-recent messages kept verbatim during compaction |
| `GADFLY_COMPACT_SUMMARY_WORDS` | 200 | word cap on the compaction summary |
| `GADFLY_MODEL_CONTEXT_TOKENS` | *(auto)* | override the model's context-window size (tokens) for the compaction threshold; set it for self-hosted endpoints executus can't introspect (Ollama Cloud models resolve automatically) |
| `GADFLY_PR_TOKEN_BUDGET` | — | per-model token ceiling for this PR; once spent, remaining lenses/passes are skipped (advisory). 0 = off |
| `GADFLY_PR_TIME_BUDGET_SECS` | — | per-model wall-clock ceiling for this PR (advisory). 0 = off |
| `GADFLY_STATUS_BOARD` | on | set `0` to disable the live status-board comment |
| `GADFLY_STATUS_POLL_SECS` | 12 | how often the status board re-renders/upserts |
| `GADFLY_CONSOLIDATE` | `auto` | cross-model consensus comment: `auto` (on for ≥2 models), `1` (force on), `0` (off — one comment per model) |
| `GADFLY_INLINE_REVIEW` | on | when consolidating, also post a `COMMENT`-state PR review with inline comments on changed lines; `0` disables |
| `GADFLY_TRIGGER_PHRASE` | `@gadfly review` | comment phrase that re-triggers |
| `GADFLY_ALLOWED_USERS` | *(collaborators)* | comma-separated allow-list for comment triggers |
| `GADFLY_FINDINGS_URL` | — | gadfly-reports store base URL; set to enable findings telemetry (off when empty) |
| `GADFLY_FINDINGS_TOKEN` | — | bearer token for the gadfly-reports store (sent as `Authorization: Bearer …`) |
| `GADFLY_REPO` | *(from `GITEA_API`)* | `owner/repo` slug stamped on emitted runs/findings (set by `entrypoint.sh`) |
| `GADFLY_PR` | *(from event)* | PR number stamped on emitted runs/findings (set by `entrypoint.sh`) |
### Large-PR cost controls
A very large diff is the one thing that can blow the budget: every review step
re-sends it, multiplied across models × lenses × passes × steps (a single
~250 K-token PR can otherwise burn a whole metered usage block). Gadfly handles
big PRs in three layers, all **size-gated so small PRs are untouched**:
1. **Paginated `get_diff` + compaction** (reviewer binary, on by default) —
`get_diff` returns a paginated, optionally per-file window instead of the whole
diff, and once a transcript nears the model's context window its middle is
folded into a summary (powered by [executus](https://gitea.stevedudenhoeffer.com/steve/executus)'s
`compact`). Tune with the `GADFLY_COMPACT_*` knobs above.
2. **Downshift** (`entrypoint.sh`) — above `GADFLY_HUGE_DIFF_BYTES` the whole fleet
collapses to a single cheap model + a focused lens subset, fewer steps, and no
recheck. A finished shallow review beats a budget-nuking one, and the posted
comment says so.
3. **Hard backstop** (`entrypoint.sh`) — `GADFLY_PR_BUDGET_SECS` is a wall-clock
ceiling across the *entire* fleet; on expiry the review is stopped and whatever
was found so far is posted. Like everything else, it never fails CI.
| Env | Default | Meaning |
|-----|---------|---------|
| `GADFLY_HUGE_DIFF_BYTES` | 600000 | downshift the fleet when the PR diff exceeds this many bytes (0 = never downshift) |
| `GADFLY_HUGE_DIFF_MODELS` | first model | model(s) to run on a downshifted huge PR |
| `GADFLY_HUGE_DIFF_SPECIALISTS` | `security,correctness,error-handling` | lenses on a downshifted huge PR |
| `GADFLY_HUGE_DIFF_MAX_STEPS` | 12 | review step cap on a huge PR |
| `GADFLY_HUGE_DIFF_RECHECK_MAX_STEPS` | 8 | recheck step cap on a huge PR |
| `GADFLY_HUGE_DIFF_RECHECK` | 0 | run the recheck pass on a huge PR (off by default) |
| `GADFLY_HUGE_DIFF_MAX_DIFF_CHARS` | 20000 | embedded review-diff chars on a huge PR |
| `GADFLY_PR_BUDGET_SECS` | — | swarm-wide wall-clock backstop; stops the whole fleet when reached (0 = off) |
## Findings telemetry (optional)
Gadfly can record what it found so model quality can be tracked over time. It is
**off by default** and purely advisory: set **`GADFLY_FINDINGS_URL`** to a
[gadfly-reports](https://gitea.stevedudenhoeffer.com/steve/gadfly-reports) store base URL and,
after each review, the binary best-effort `POST`s the run (`/runs`) and the
findings it surfaced (`/reports`) to that store. Add **`GADFLY_FINDINGS_TOKEN`**
to send an `Authorization: Bearer …` header. `entrypoint.sh` supplies the run
context (`GADFLY_REPO`, `GADFLY_PR`) automatically.
Findings are extracted heuristically from each lens's markdown — a `path:line`
reference anchors a finding, titled by the nearest preceding heading / numbered
item / bold lead-in. A lens whose verdict is **"No material issues found"**
emits **no** findings: its `path:line` references are verification notes
("verified X is safe"), not problems, so extracting them would record false
positives and unfairly penalize thorough clean-pass reviewers. The emit is
strictly best-effort: a short (~10s) timeout, any error (or a non-2xx response)
is logged to stderr only, and it **never** changes the review output or the exit
code.
## Building locally
```sh
go build ./cmd/gadfly # needs read access to the private majordomo module
go build ./cmd/gadfly # needs read access to the private majordomo + executus modules
go test ./...
```
+148
View File
@@ -0,0 +1,148 @@
package main
import (
"context"
"fmt"
"os"
"sort"
"strings"
"gitea.stevedudenhoeffer.com/steve/majordomo"
llm "gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
// maxAutoSpecialists bounds how many lenses dynamic selection may run, so a
// runaway selector can't blow up cost.
const maxAutoSpecialists = 6
// autoCustom is an ad-hoc specialist the selector may propose when the change
// needs a lens the catalog lacks (e.g. DB migrations, i18n).
type autoCustom struct {
Name string `json:"name" description:"short lowercase id, e.g. migrations"`
Title string `json:"title" description:"human label for the section, e.g. 🗃️ DB migrations"`
Focus string `json:"focus" description:"one or two sentences telling the reviewer what to look for"`
}
// autoSelection is the structured result the selector returns.
type autoSelection struct {
Specialists []string `json:"specialists" description:"names of EXISTING catalog specialists that materially apply to this change"`
Custom []autoCustom `json:"custom" description:"ad-hoc specialists to add ONLY when the catalog lacks a clearly-needed lens; usually empty"`
}
// resolveSelectorModel returns the model that picks specialists in auto mode:
// GADFLY_SELECTOR_MODEL (a cheap tier is ideal) if set, else the review model.
func resolveSelectorModel(fallback llm.Model) (llm.Model, error) {
spec := strings.TrimSpace(os.Getenv("GADFLY_SELECTOR_MODEL"))
if spec == "" {
return fallback, nil
}
provider := strings.TrimSpace(os.Getenv("GADFLY_PROVIDER"))
if provider == "" {
provider = defaultProvider
}
return majordomo.Parse(buildSpec(provider, spec))
}
// autoSelectSpecialists asks the selector model which lenses a given diff needs.
// It picks from the registry and may invent a few ad-hoc lenses, capped and
// de-duplicated. On any failure it returns an error and the caller falls back
// to the default suite.
func autoSelectSpecialists(ctx context.Context, selector llm.Model, title, body, diff string, registry map[string]Specialist) ([]Specialist, error) {
system := "You are the router for an adversarial code review. Given a pull request's " +
"changed files and description plus a catalog of available review specialists (lenses), " +
"choose the SMALLEST set that materially applies to THIS change — usually 2 to 4. Skip " +
"lenses the diff doesn't touch (e.g. no 'performance' for a docs-only change). Strongly " +
"prefer existing catalog lenses; only propose a custom one when the change clearly needs a " +
"lens the catalog lacks (e.g. database migrations, i18n, build/CI). Be conservative."
user := fmt.Sprintf("PR title: %s\n\nPR description:\n%s\n\nChanged files:\n%s\n\nAvailable specialists (name — focus):\n%s\n\nSelect the lenses to run.",
strings.TrimSpace(title), strings.TrimSpace(body), changedFilesList(diff), catalog(registry))
sel, err := majordomo.Generate[autoSelection](ctx, selector, majordomo.Request{
System: system,
Messages: []llm.Message{llm.UserText(user)},
})
if err != nil {
return nil, fmt.Errorf("auto-select: %w", err)
}
var out []Specialist
seen := map[string]bool{}
add := func(sp Specialist) {
if sp.Name == "" || seen[sp.Name] || len(out) >= maxAutoSpecialists {
return
}
seen[sp.Name] = true
out = append(out, sp)
}
for _, name := range sel.Specialists {
if sp, ok := registry[strings.ToLower(strings.TrimSpace(name))]; ok {
add(sp)
}
}
for _, c := range sel.Custom {
name := strings.ToLower(strings.TrimSpace(c.Name))
if name == "" || strings.TrimSpace(c.Focus) == "" {
continue
}
if sp, ok := registry[name]; ok { // prefer the catalog definition if it exists
add(sp)
continue
}
add(Specialist{Name: name, Title: titleOr(c.Title, name), Focus: c.Focus})
}
if len(out) == 0 {
return nil, fmt.Errorf("auto-select returned no usable specialists")
}
return out, nil
}
// catalog renders the registry as a compact name — focus list for the selector.
func catalog(registry map[string]Specialist) string {
names := make([]string, 0, len(registry))
for k := range registry {
names = append(names, k)
}
sort.Strings(names)
var b strings.Builder
for _, n := range names {
fmt.Fprintf(&b, "- %s — %s\n", n, firstSentence(registry[n].Focus))
}
return b.String()
}
// changedFilesList extracts the changed file paths from a unified diff.
func changedFilesList(diff string) string {
var files []string
seen := map[string]bool{}
for _, line := range strings.Split(diff, "\n") {
if !strings.HasPrefix(line, "+++ ") && !strings.HasPrefix(line, "--- ") {
continue
}
p := strings.TrimSpace(line[4:])
if p == "/dev/null" {
continue
}
p = strings.TrimPrefix(strings.TrimPrefix(p, "a/"), "b/")
if p != "" && !seen[p] {
seen[p] = true
files = append(files, p)
}
}
if len(files) == 0 {
return "(could not parse file list; review the whole diff)"
}
return "- " + strings.Join(files, "\n- ")
}
func firstSentence(s string) string {
s = strings.TrimSpace(s)
if i := strings.IndexByte(s, '.'); i > 0 && i < 140 {
return s[:i]
}
if len(s) > 140 {
return s[:140] + "…"
}
return s
}
+477
View File
@@ -0,0 +1,477 @@
package main
// Cross-model consensus consolidation. The swarm runs each model independently
// (entrypoint.sh fans them out across provider lanes); historically each model
// posted its OWN comment, so a reader faced N walls of prose that mostly agreed.
//
// Instead, every model writes its findings to a shared directory
// (GADFLY_FINDINGS_OUT, one JSON file per model), and after the whole swarm
// finishes a single consolidation pass (GADFLY_CONSOLIDATE_DIR) clusters those
// findings by location, counts how many models independently flagged each one,
// and renders ONE comment: an agreement-ranked table up top (cross-model
// agreement is the strongest real-vs-false-positive signal we have), with each
// model's full review folded below for drill-down.
//
// This file owns: the per-model artifact (modelFindings), writing it
// (writeFindingsOut), reading the directory back, clustering, and rendering the
// consensus markdown (renderConsensus). It depends only on the stdlib.
import (
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
)
// consensusMarker tags the single consolidated comment so entrypoint.sh can
// upsert it in place across re-runs (mirrors run.sh's per-model marker).
const consensusMarker = "<!-- gadfly-consensus -->"
// modelFindings is the per-model artifact written to GADFLY_FINDINGS_OUT. It
// carries enough to rebuild the consolidated comment without re-running models:
// the structured findings (for clustering) plus the full rendered review (for
// the folded per-model drill-down).
type modelFindings struct {
Model string `json:"model"`
Provider string `json:"provider"`
Verdict string `json:"verdict"` // worst lens verdict, as a label
Errored bool `json:"errored"` // the model produced no usable review (every lens failed, or the run crashed)
Markdown string `json:"markdown"` // full rendered per-model review (findings block already stripped)
Findings []outFinding `json:"findings"`
}
// outFinding is one finding in the per-model artifact (a flattened `finding`
// plus its lens).
type outFinding struct {
Lens string `json:"lens"`
File string `json:"file"`
Line int `json:"line"`
Severity string `json:"severity"`
Confidence string `json:"confidence"`
Title string `json:"title"`
Detail string `json:"detail"`
}
// collectFindings returns a lens result's findings with severity always filled
// in: per-finding when the structured block supplied it, else derived from the
// lens verdict (so heuristic-scraped findings still carry a canonical word). A
// clean or errored lens yields nothing. Shared by the telemetry emit and the
// per-model findings file so both agree on what a finding is.
func collectFindings(r specialistResult) []finding {
if r.errored || r.verdict == verdictClean {
return nil
}
fs := extractStructuredFindingsOrScrape(r)
lensSev := r.verdict.severity()
for i := range fs {
if fs[i].severity == "" {
fs[i].severity = lensSev
}
}
return fs
}
// writeFindingsOut writes this model's findings + rendered review to
// GADFLY_FINDINGS_OUT for the later consolidation pass. No-op unless the env is
// set. Best-effort: any error is logged to stderr and never affects the review
// (it runs after the markdown is already on stdout).
func writeFindingsOut(results []specialistResult) {
path := strings.TrimSpace(os.Getenv("GADFLY_FINDINGS_OUT"))
if path == "" {
return
}
mf := modelFindings{
Model: strings.TrimSpace(os.Getenv("GADFLY_MODEL")),
Provider: modelProvider(),
Verdict: worstVerdict(results).label(),
Errored: allErrored(results),
Markdown: renderConsolidated(results),
}
for _, r := range results {
for _, f := range collectFindings(r) {
mf.Findings = append(mf.Findings, outFinding{
Lens: r.spec.Name,
File: f.file,
Line: f.line,
Severity: f.severity,
Confidence: f.confidence,
Title: f.title,
Detail: f.detail,
})
}
}
data, err := json.Marshal(mf)
if err != nil {
fmt.Fprintln(os.Stderr, "gadfly: marshal findings out:", err)
return
}
// Defensive: make sure the parent dir exists (entrypoint creates it, but a
// missing dir would otherwise silently drop this model from the consensus).
if dir := filepath.Dir(path); dir != "" {
_ = os.MkdirAll(dir, 0o755)
}
if err := os.WriteFile(path, data, 0o644); err != nil {
fmt.Fprintln(os.Stderr, "gadfly: write findings out:", err)
}
}
// allErrored reports whether every lens of a review failed (so the model
// produced no usable findings). Such a model is recorded but excluded from the
// consensus agreement denominator — counting it would dilute every ratio with a
// model that never actually reviewed.
func allErrored(results []specialistResult) bool {
if len(results) == 0 {
return true
}
for _, r := range results {
if !r.errored {
return false
}
}
return true
}
// runConsolidate is the consolidation entry point (GADFLY_CONSOLIDATE_DIR set):
// read every per-model artifact in the directory, render the consensus comment
// to stdout. Errors are fatal to THIS process only — entrypoint.sh treats a
// failed consolidation as advisory and falls back to per-model comments.
func runConsolidate() error {
dir := strings.TrimSpace(os.Getenv("GADFLY_CONSOLIDATE_DIR"))
if dir == "" {
return errors.New("GADFLY_CONSOLIDATE_DIR is empty")
}
entries, err := os.ReadDir(dir)
if err != nil {
return fmt.Errorf("read consolidate dir: %w", err)
}
var models []modelFindings
for _, e := range entries {
if e.IsDir() || !strings.HasSuffix(e.Name(), ".json") {
continue
}
data, err := os.ReadFile(filepath.Join(dir, e.Name()))
if err != nil {
fmt.Fprintln(os.Stderr, "gadfly: read", e.Name(), err)
continue
}
var mf modelFindings
if err := json.Unmarshal(data, &mf); err != nil {
fmt.Fprintln(os.Stderr, "gadfly: parse", e.Name(), err)
continue
}
if strings.TrimSpace(mf.Model) == "" {
continue
}
models = append(models, mf)
}
if len(models) == 0 {
return errors.New("no model findings to consolidate")
}
// Cluster once, then render the consensus comment and (best-effort) the inline
// PR review from the same clusters so the two views can't drift.
clusters := clusterFindings(models)
// Lead with the marker so entrypoint.sh can upsert this comment in place
// (same pattern as run.sh's per-model marker); it appends the advisory footer.
fmt.Println(consensusMarker)
fmt.Println(renderConsensus(models, clusters))
// Inline PR review (COMMENT state) anchoring findings to changed lines.
// Best-effort: no-op without diff/API creds, never affects stdout/exit.
postInlineReview(clusters)
return nil
}
// cluster is a group of findings (across models) judged to be the same issue:
// same file, lines within lineTolerance of the cluster's current span. The span
// [line,maxLine] slides as members join, so a chain of nearby findings merges
// instead of splitting once it drifts past the first line.
type cluster struct {
file string
line int // representative (smallest) line
maxLine int // largest line in the cluster — the span's upper edge
severity string
title string
detail string // highest-severity report's detail; rendered in the inline review comment
models map[string]bool
lenses map[string]bool
}
// findingRef is one model's finding (carrying which model reported it), used
// while grouping findings into clusters.
type findingRef struct {
f outFinding
model string
}
// lineTolerance: a finding in the same file within this many lines of a
// cluster's current span is treated as the same issue (models often cite a line
// or two apart).
const lineTolerance = 3
// renderConsensus builds the single consolidated comment body from every model's
// findings. It does NOT emit the marker or advisory footer — entrypoint.sh wraps
// it (mirroring run.sh's per-model framing).
func renderConsensus(models []modelFindings, clusters []cluster) string {
// effective = models that actually produced a review. Errored models are
// shown (folded, below) but excluded from the agreement denominator so a
// failed model doesn't dilute every ratio.
effective := 0
for _, m := range models {
if !m.Errored {
effective++
}
}
errored := len(models) - effective
worst := verdictClean
for _, m := range models {
if v := parseVerdict(m.Verdict); v > worst {
worst = v
}
}
// Partition in one pass: "headline" findings (multi-model agreement, OR a
// lone CRITICAL) vs folded "single-model" lower-confidence findings. Also
// count multi-model agreements for the summary line.
var headline, folded []cluster
agreed := 0
for _, c := range clusters {
if len(c.models) >= 2 {
agreed++
}
if len(c.models) >= 2 || sevRank(c.severity) >= sevRank("critical") {
headline = append(headline, c)
} else {
folded = append(folded, c)
}
}
var b strings.Builder
fmt.Fprintf(&b, "## 🪰 Gadfly review — consensus across %d model%s", effective, plural(effective))
if errored > 0 {
fmt.Fprintf(&b, " (%d failed)", errored)
}
b.WriteString("\n\n")
fmt.Fprintf(&b, "**Verdict: %s** · %d finding%s (%d with multi-model agreement)\n",
worst.label(), len(clusters), plural(len(clusters)), agreed)
if len(headline) > 0 {
b.WriteString("\n| | Finding | Where | Models | Lens |\n|--|--|--|--|--|\n")
for _, c := range headline {
fmt.Fprintf(&b, "| %s | %s | `%s` | %d/%d | %s |\n",
sevIcon(c.severity), mdCell(c.title), mdCell(location(c.file, c.line)),
len(c.models), effective, mdCell(lensList(c.lenses)))
}
} else if len(clusters) == 0 {
b.WriteString("\nNo material issues found by consensus.\n")
}
// else: only single-model findings — they're shown folded below, so don't
// claim "no material issues" (there are some, just none with consensus).
if len(folded) > 0 {
fmt.Fprintf(&b, "\n<details><summary>%d single-model finding%s (lower confidence)</summary>\n\n",
len(folded), plural(len(folded)))
b.WriteString("| | Finding | Where | Model | Lens |\n|--|--|--|--|--|\n")
for _, c := range folded {
fmt.Fprintf(&b, "| %s | %s | `%s` | %s | %s |\n",
sevIcon(c.severity), mdCell(c.title), mdCell(location(c.file, c.line)),
mdCell(oneModel(c.models)), mdCell(lensList(c.lenses)))
}
b.WriteString("\n</details>\n")
}
// Per-model full reviews, folded for drill-down (nothing is lost).
b.WriteString("\n<details><summary>Per-model detail</summary>\n")
for _, m := range models {
body := strings.TrimSpace(m.Markdown)
if body == "" {
body = "_(no output)_"
}
verdict := m.Verdict
if m.Errored {
verdict = "⚠️ reviewer failed"
}
fmt.Fprintf(&b, "\n<details><summary><b>%s</b> (%s) — %s</summary>\n\n%s\n\n</details>\n",
mdCell(m.Model), mdCell(m.Provider), verdict, body)
}
b.WriteString("\n</details>")
return b.String()
}
// clusterFindings groups every model's findings into cross-model clusters,
// sorted by agreement (desc), then severity (desc), then location.
func clusterFindings(models []modelFindings) []cluster {
// Group by file, then greedily merge by line proximity.
byFile := map[string][]findingRef{}
for _, m := range models {
for _, f := range m.Findings {
if strings.TrimSpace(f.File) == "" {
continue
}
byFile[f.File] = append(byFile[f.File], findingRef{f, m.Model})
}
}
var clusters []cluster
for file, items := range byFile {
sort.SliceStable(items, func(i, j int) bool { return items[i].f.Line < items[j].f.Line })
// Cluster within THIS file only (clusters never span files), so the inner
// scan is over same-file clusters, not every cluster seen so far.
var fileClusters []cluster
for _, it := range items {
placed := false
for ci := range fileClusters {
c := &fileClusters[ci]
// Join if the line falls within the cluster's span, widened by the
// tolerance on both edges — so the window slides as the span grows.
if it.f.Line >= c.line-lineTolerance && it.f.Line <= c.maxLine+lineTolerance {
mergeIntoCluster(c, it.f, it.model)
placed = true
break
}
}
if !placed {
c := cluster{
file: file,
line: it.f.Line,
maxLine: it.f.Line,
severity: it.f.Severity,
title: it.f.Title,
detail: it.f.Detail,
models: map[string]bool{},
lenses: map[string]bool{},
}
mergeIntoCluster(&c, it.f, it.model)
fileClusters = append(fileClusters, c)
}
}
clusters = append(clusters, fileClusters...)
}
sort.SliceStable(clusters, func(i, j int) bool {
if len(clusters[i].models) != len(clusters[j].models) {
return len(clusters[i].models) > len(clusters[j].models)
}
if sevRank(clusters[i].severity) != sevRank(clusters[j].severity) {
return sevRank(clusters[i].severity) > sevRank(clusters[j].severity)
}
if clusters[i].file != clusters[j].file {
return clusters[i].file < clusters[j].file
}
return clusters[i].line < clusters[j].line
})
return clusters
}
// mergeIntoCluster folds one finding into a cluster: union the model/lens sets,
// widen the [line,maxLine] span, and keep the highest-severity report's title.
func mergeIntoCluster(c *cluster, f outFinding, model string) {
if model != "" {
c.models[model] = true
}
if f.Lens != "" {
c.lenses[f.Lens] = true
}
if f.Line > 0 && (c.line == 0 || f.Line < c.line) {
c.line = f.Line
}
if f.Line > c.maxLine {
c.maxLine = f.Line
}
// Backfill an empty title/detail from any report, regardless of severity, so a
// higher-severity-but-terse finding doesn't leave the cluster without context.
if strings.TrimSpace(c.title) == "" && strings.TrimSpace(f.Title) != "" {
c.title = f.Title
}
if strings.TrimSpace(c.detail) == "" && strings.TrimSpace(f.Detail) != "" {
c.detail = f.Detail
}
// A strictly-higher-severity report takes over the title/detail.
if sevRank(f.Severity) > sevRank(c.severity) {
c.severity = f.Severity
if strings.TrimSpace(f.Title) != "" {
c.title = f.Title
}
if strings.TrimSpace(f.Detail) != "" {
c.detail = f.Detail
}
}
}
// sevRank orders the canonical severity words for sorting/comparison.
func sevRank(s string) int {
switch strings.ToLower(strings.TrimSpace(s)) {
case "critical":
return 5
case "high":
return 4
case "medium":
return 3
case "small":
return 2
case "trivial":
return 1
default:
return 0
}
}
// sevIcon is the at-a-glance severity badge for the consensus table.
func sevIcon(s string) string {
switch strings.ToLower(strings.TrimSpace(s)) {
case "critical", "high":
return "🔴"
case "medium":
return "🟠"
case "small":
return "🟡"
default:
return "⚪"
}
}
func location(file string, line int) string {
if line > 0 {
return fmt.Sprintf("%s:%d", file, line)
}
return file
}
func lensList(lenses map[string]bool) string {
out := make([]string, 0, len(lenses))
for l := range lenses {
out = append(out, l)
}
sort.Strings(out)
return strings.Join(out, ", ")
}
func oneModel(models map[string]bool) string {
for m := range models {
return m
}
return ""
}
// mdCell makes a string safe for a one-line markdown table cell: collapse
// newlines, escape pipes (which delimit columns), and neutralize backticks
// (a stray one would break an inline-code span — a backslash can't escape it
// inside code, so replace with an apostrophe). Inputs are model-influenced, so
// this keeps a malformed file path or title from breaking the table.
func mdCell(s string) string {
s = strings.ReplaceAll(s, "\n", " ")
s = strings.ReplaceAll(s, "|", "\\|")
s = strings.ReplaceAll(s, "`", "'")
return strings.TrimSpace(s)
}
func plural(n int) string {
if n == 1 {
return ""
}
return "s"
}
+217
View File
@@ -0,0 +1,217 @@
package main
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
)
func TestClusterFindingsAgreementAndTolerance(t *testing.T) {
models := []modelFindings{
{Model: "m1", Verdict: "Blocking issues found", Findings: []outFinding{
{Lens: "security", File: "a.go", Line: 10, Severity: "high", Title: "auth bypass"},
{Lens: "perf", File: "b.go", Line: 5, Severity: "trivial", Title: "tiny nit"},
}},
{Model: "m2", Verdict: "Minor issues", Findings: []outFinding{
{Lens: "security", File: "a.go", Line: 11, Severity: "critical", Title: "auth bypass (crit)"}, // within tolerance of a.go:10
}},
{Model: "m3", Verdict: "Minor issues", Findings: []outFinding{
{Lens: "correctness", File: "a.go", Line: 10, Severity: "medium", Title: "auth bypass"},
}},
}
clusters := clusterFindings(models)
if len(clusters) != 2 {
t.Fatalf("want 2 clusters (a.go:10±, b.go:5), got %d: %+v", len(clusters), clusters)
}
// First cluster (highest agreement) is the a.go auth one: 3 models, severity
// escalated to critical, representative line the smallest (10).
c := clusters[0]
if len(c.models) != 3 {
t.Errorf("want 3 models on the top cluster, got %d", len(c.models))
}
if c.severity != "critical" {
t.Errorf("want escalated severity critical, got %q", c.severity)
}
if c.line != 10 {
t.Errorf("want representative line 10, got %d", c.line)
}
if !c.lenses["security"] || !c.lenses["correctness"] {
t.Errorf("want union of lenses, got %v", c.lenses)
}
}
func TestRenderConsensusFoldsSingleModelNits(t *testing.T) {
models := []modelFindings{
{Model: "m1", Provider: "p", Verdict: "Blocking issues found", Markdown: "m1 detail", Findings: []outFinding{
{Lens: "security", File: "a.go", Line: 10, Severity: "high", Title: "auth bypass"},
{Lens: "perf", File: "b.go", Line: 5, Severity: "trivial", Title: "tiny nit"},
}},
{Model: "m2", Provider: "p", Verdict: "Minor issues", Markdown: "m2 detail", Findings: []outFinding{
{Lens: "security", File: "a.go", Line: 10, Severity: "high", Title: "auth bypass"},
}},
}
out := renderConsensus(models, clusterFindings(models))
// Headline table: the agreed finding with a 2/2 badge.
if !strings.Contains(out, "2/2") {
t.Errorf("expected a 2/2 agreement badge in headline:\n%s", out)
}
if !strings.Contains(out, "auth bypass") || !strings.Contains(out, "a.go:10") {
t.Errorf("headline missing the consensus finding:\n%s", out)
}
// The lone trivial finding is folded, not in the headline table.
if !strings.Contains(out, "single-model finding") {
t.Errorf("expected a folded single-model section:\n%s", out)
}
// Per-model detail is preserved (folded).
if !strings.Contains(out, "m1 detail") || !strings.Contains(out, "m2 detail") {
t.Errorf("per-model detail not preserved:\n%s", out)
}
}
func TestRenderConsensusHighSeverityLoneFindingStaysHeadline(t *testing.T) {
// A single model, single critical finding must still surface in the headline
// (not be folded as "low confidence").
models := []modelFindings{
{Model: "solo", Verdict: "Blocking issues found", Markdown: "x", Findings: []outFinding{
{Lens: "security", File: "a.go", Line: 1, Severity: "critical", Title: "rce"},
}},
}
out := renderConsensus(models, clusterFindings(models))
headline := out
if i := strings.Index(out, "single-model finding"); i >= 0 {
headline = out[:i]
}
if !strings.Contains(headline, "rce") {
t.Errorf("lone critical should be in the headline, not folded:\n%s", out)
}
}
func TestClusterSlidingWindowMergesChain(t *testing.T) {
// Findings at 10, 13, 16 (each 3 apart) from three models must merge into ONE
// cluster — the window slides with the span instead of anchoring at line 10.
models := []modelFindings{
{Model: "m1", Findings: []outFinding{{Lens: "x", File: "a.go", Line: 10, Severity: "medium", Title: "t"}}},
{Model: "m2", Findings: []outFinding{{Lens: "x", File: "a.go", Line: 13, Severity: "medium", Title: "t"}}},
{Model: "m3", Findings: []outFinding{{Lens: "x", File: "a.go", Line: 16, Severity: "medium", Title: "t"}}},
}
clusters := clusterFindings(models)
if len(clusters) != 1 {
t.Fatalf("chain 10/13/16 should merge into 1 cluster, got %d", len(clusters))
}
if len(clusters[0].models) != 3 {
t.Errorf("want 3 models in the merged cluster, got %d", len(clusters[0].models))
}
}
func TestRenderConsensusExcludesErroredFromDenominator(t *testing.T) {
models := []modelFindings{
{Model: "m1", Verdict: "Minor issues", Markdown: "a", Findings: []outFinding{
{Lens: "security", File: "a.go", Line: 9, Severity: "medium", Title: "leak"}}},
{Model: "m2", Verdict: "Minor issues", Markdown: "b", Findings: []outFinding{
{Lens: "security", File: "a.go", Line: 9, Severity: "medium", Title: "leak"}}},
{Model: "broken", Verdict: "reviewer failed", Errored: true, Markdown: "boom"},
}
out := renderConsensus(models, clusterFindings(models))
// Denominator is the 2 effective models, not 3; the failure is noted.
if !strings.Contains(out, "2/2") {
t.Errorf("errored model must be excluded from the denominator (want 2/2):\n%s", out)
}
if !strings.Contains(out, "1 failed") {
t.Errorf("expected a '1 failed' note:\n%s", out)
}
if !strings.Contains(out, "reviewer failed") {
t.Errorf("errored model should still appear (folded) as failed:\n%s", out)
}
}
func TestRenderConsensusLoneHighFolds(t *testing.T) {
// A single-model HIGH (not critical) folds — only consensus or a lone CRITICAL
// earns the headline, so a lone Blocking-lens finding doesn't reintroduce noise.
models := []modelFindings{
{Model: "solo", Verdict: "Blocking issues found", Markdown: "x", Findings: []outFinding{
{Lens: "security", File: "a.go", Line: 1, Severity: "high", Title: "maybe-bug"}}},
}
out := renderConsensus(models, clusterFindings(models))
head := out
if i := strings.Index(out, "single-model finding"); i >= 0 {
head = out[:i]
}
if strings.Contains(head, "maybe-bug") {
t.Errorf("a lone HIGH should fold, not headline:\n%s", out)
}
}
func TestWriteAndConsolidateRoundTrip(t *testing.T) {
dir := t.TempDir()
// Two model artifacts on disk.
write := func(name string, mf modelFindings) {
data, _ := json.Marshal(mf)
if err := os.WriteFile(filepath.Join(dir, name), data, 0o644); err != nil {
t.Fatal(err)
}
}
write("m1.json", modelFindings{Model: "m1", Provider: "ollama", Verdict: "Minor issues", Markdown: "md1",
Findings: []outFinding{{Lens: "security", File: "x.go", Line: 3, Severity: "medium", Title: "leak"}}})
write("m2.json", modelFindings{Model: "m2", Provider: "ollama", Verdict: "Minor issues", Markdown: "md2",
Findings: []outFinding{{Lens: "security", File: "x.go", Line: 3, Severity: "high", Title: "leak"}}})
// A junk file must be skipped, not crash consolidation.
if err := os.WriteFile(filepath.Join(dir, "notes.txt"), []byte("ignore me"), 0o644); err != nil {
t.Fatal(err)
}
t.Setenv("GADFLY_CONSOLIDATE_DIR", dir)
// runConsolidate prints to stdout; capture it.
out := captureStdout(t, func() {
if err := runConsolidate(); err != nil {
t.Fatalf("runConsolidate: %v", err)
}
})
if !strings.HasPrefix(strings.TrimSpace(out), consensusMarker) {
t.Errorf("consolidated output must lead with the marker:\n%s", out)
}
if !strings.Contains(out, "2/2") || !strings.Contains(out, "x.go:3") {
t.Errorf("expected the agreed x.go:3 finding at 2/2:\n%s", out)
}
}
func TestRunConsolidateEmptyDirErrors(t *testing.T) {
t.Setenv("GADFLY_CONSOLIDATE_DIR", t.TempDir())
if err := runConsolidate(); err == nil {
t.Error("want an error for an empty consolidate dir (entrypoint falls back)")
}
}
// captureStdout redirects os.Stdout for the duration of fn and returns what was
// written.
func captureStdout(t *testing.T, fn func()) string {
t.Helper()
orig := os.Stdout
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
os.Stdout = w
defer func() { os.Stdout = orig }() // restore even if fn panics
done := make(chan string)
go func() {
var sb strings.Builder
buf := make([]byte, 4096)
for {
n, err := r.Read(buf)
if n > 0 {
sb.Write(buf[:n])
}
if err != nil {
break
}
}
r.Close()
done <- sb.String()
}()
fn()
w.Close()
return <-done
}
+141
View File
@@ -0,0 +1,141 @@
package main
import (
"fmt"
"strings"
)
// verdict is the severity a specialist (or the whole review) lands on.
type verdict int
const (
verdictUnknown verdict = iota // couldn't parse; treated below Minor for display, not blocking
verdictClean // "No material issues found"
verdictMinor // "Minor issues"
verdictBlocking // "Blocking issues found"
)
func (v verdict) label() string {
switch v {
case verdictClean:
return "No material issues found"
case verdictMinor:
return "Minor issues"
case verdictBlocking:
return "Blocking issues found"
default:
return "Reviewed"
}
}
// severity maps a verdict to a canonical severity word (the same vocabulary as
// the structured findings: critical/high/medium/small/trivial). Used as the
// raw_severity for heuristic-scraped findings, which carry no per-finding
// severity of their own — so the telemetry store sees one consistent vocabulary
// instead of a mix of canonical words and full verdict phrases.
func (v verdict) severity() string {
switch v {
case verdictBlocking:
return "high"
case verdictMinor:
return "small"
default:
return "trivial"
}
}
// parseVerdict extracts a specialist's self-reported verdict from its output.
// The base prompt tells each lens to lead with one of the three phrases.
func parseVerdict(out string) verdict {
l := strings.ToLower(out)
// Match the EARLIEST-appearing verdict phrase (the lead verdict), and match
// leniently — models write "Blocking issues found", "**Blocking issues**",
// "blocking issue", etc. (A strict "blocking issues found" check let a
// gpt-oss section that said "**Blocking issues**" fall through to unknown,
// so the overall header wrongly read "No material issues found".)
best, bestIdx := verdictUnknown, -1
for _, c := range []struct {
phrase string
v verdict
}{
{"blocking issue", verdictBlocking},
{"minor issue", verdictMinor},
{"no material issue", verdictClean},
} {
if i := strings.Index(l, c.phrase); i >= 0 && (bestIdx == -1 || i < bestIdx) {
best, bestIdx = c.v, i
}
}
return best
}
// specialistResult pairs a specialist with its rendered review and verdict.
type specialistResult struct {
spec Specialist
out string
verdict verdict
errored bool // the review pass failed (timeout/model error) — not a clean result
}
// worstVerdict returns the most severe verdict across results. The optional
// "improvements" lens never escalates the overall verdict (it's advisory).
func worstVerdict(results []specialistResult) verdict {
worst := verdictClean
for _, r := range results {
if r.spec.Name == "improvements" {
continue
}
if r.verdict > worst {
worst = r.verdict
}
}
return worst
}
// renderConsolidated assembles the single comment body: an overall verdict line
// followed by one verbatim section per specialist. run.sh wraps this with the
// "🪰 Gadfly review — <model>" header and the advisory footer.
func renderConsolidated(results []specialistResult) string {
errored := 0
for _, r := range results {
if r.errored {
errored++
}
}
headline := "Verdict: " + worstVerdict(results).label()
if len(results) > 0 && errored == len(results) {
// Every lens errored — do NOT report this as "clean".
headline = "Review incomplete — all lenses errored"
} else if errored > 0 {
headline += fmt.Sprintf(" · ⚠️ %d/%d lens(es) errored", errored, len(results))
}
var b strings.Builder
fmt.Fprintf(&b, "**%s** — %d reviewers: %s\n",
headline, len(results), strings.Join(specialistNames(results), ", "))
for _, r := range results {
// Strip the machine-readable ```gadfly-findings block — it's for tooling
// (telemetry/consolidation), not for human readers of the comment.
body := strings.TrimSpace(stripFindingsBlock(r.out))
if body == "" {
body = "_(no output)_"
}
summary := r.verdict.label()
if r.errored {
summary = "⚠️ could not complete"
}
fmt.Fprintf(&b, "\n<details><summary><b>%s</b> — %s</summary>\n\n%s\n\n</details>\n",
r.spec.Title, summary, body)
}
return strings.TrimRight(b.String(), "\n")
}
func specialistNames(results []specialistResult) []string {
names := make([]string, 0, len(results))
for _, r := range results {
names = append(names, r.spec.Name)
}
return names
}
+65
View File
@@ -0,0 +1,65 @@
package main
import (
"context"
"fmt"
"strings"
"gitea.stevedudenhoeffer.com/steve/majordomo/agent"
llm "gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
// defaultWorkerMaxSteps bounds a delegated worker run.
const defaultWorkerMaxSteps = 8
// workerSystemPrompt drives a delegated investigation sub-agent. It does the
// grunt work (grep/read) and returns a tight, evidence-cited digest so the
// top reviewer reasons over a summary instead of raw file dumps.
const workerSystemPrompt = "You are a fast code investigator. Use the read-only tools " +
"(read_file, grep, find_files, list_dir, get_diff) to answer the SPECIFIC question you are " +
"given about this repository — nothing more. Be precise and concise: report concrete findings " +
"with `path:line` evidence, quote only the few lines that matter, and state plainly when " +
"something is absent. Do NOT speculate or pad. When you have the answer, stop and report it."
type delegateArgs struct {
Task string `json:"task" description:"A specific investigation to carry out by reading/grepping the repo, e.g. 'find every caller of parseAmount and list which ones don't check the error'. Ask for a concrete answer, not an opinion."`
}
// delegateTool lets the reviewer offload mechanical investigation to a (cheaper)
// worker model. Present only when GADFLY_WORKER_MODEL is configured.
func (r *repoFS) delegateTool() llm.Tool {
return llm.DefineTool[delegateArgs](
"delegate_investigation",
"Delegate focused legwork to a fast worker agent that reads/greps this repo and reports back "+
"a concise, evidence-cited digest. Use it to offload mechanical investigation (tracing all "+
"callers, gathering every usage of a symbol, checking a pattern across many files) so you can "+
"reason over the summary instead of pulling raw files into your own context.",
func(ctx context.Context, args delegateArgs) (any, error) {
task := strings.TrimSpace(args.Task)
if task == "" {
return nil, fmt.Errorf("task is required")
}
box, err := r.workerToolbox()
if err != nil {
return nil, err
}
w := agent.New(r.worker, workerSystemPrompt,
agent.WithToolbox(box),
agent.WithMaxSteps(envInt("GADFLY_WORKER_MAX_STEPS", defaultWorkerMaxSteps)),
agent.WithToolErrorLimits(3, 3),
)
res, runErr := w.Run(ctx, task)
out := ""
if res != nil {
out = strings.TrimSpace(res.Output)
}
if out == "" {
if runErr != nil {
return nil, fmt.Errorf("worker investigation failed: %w", runErr)
}
return "worker returned no findings", nil
}
return out, nil
},
)
}
+395
View File
@@ -0,0 +1,395 @@
package main
// Optional findings telemetry. After a review, Gadfly can POST the run and the
// findings it surfaced to a "gadfly-reports" store (gitea.stevedudenhoeffer.com-style
// HTTP API) so model quality can be tracked over time and graded later.
//
// This is strictly ADVISORY plumbing and must never affect the review itself:
// - It is OFF unless GADFLY_FINDINGS_URL is set.
// - It writes ONLY to stderr on any error; it never touches stdout (stdout is
// the review markdown consumed by run.sh) and never changes the exit code.
// - It depends only on the Go stdlib (net/http).
//
// Findings come PRIMARILY from each lens's machine-readable ```gadfly-findings
// block (findings.go: exact file/line + per-finding severity/confidence). When a
// model emits no parseable block, emit falls back to a heuristic prose scrape: a
// "path:line" reference (e.g. run/executor.go:166) anchors a finding, whose title
// is the nearest preceding markdown heading / numbered item / bold lead-in (else
// the first sentence of the finding's paragraph). The scrape is best-effort
// signal for the store to aggregate, not a structured contract the reviewer
// guarantees.
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"regexp"
"strconv"
"strings"
"time"
)
const (
// findingsHTTPTimeout bounds each telemetry POST. Short on purpose — a slow
// or dead store must never delay the review's stdout (already printed) or the
// process exit.
findingsHTTPTimeout = 10 * time.Second
// maxFindingsPerLens caps how many findings a single lens contributes, so a
// pathological lens output can't blow up the /reports payload.
maxFindingsPerLens = 20
)
// pathLineRe matches a source-path-with-extension followed by ":<line>", e.g.
// run/executor.go:166 or compact/compactor.go:225. The path may contain
// directories, dots, dashes and underscores; backtracking lets the final ".ext"
// be split off the greedy path run. Surrounding backticks/parens are ignored
// because they are not in the character class.
var pathLineRe = regexp.MustCompile(`([A-Za-z0-9_./-]+\.[A-Za-z0-9]+):(\d+)`)
// numberedRe matches a numbered list/heading lead-in: "1. Title" or "1) Title".
var numberedRe = regexp.MustCompile(`^\d+[.)]\s+(.+)$`)
// finding is one extracted issue: where it points and a derived human title.
// severity/confidence are populated from the structured ```gadfly-findings block
// (extractStructuredFindings); they are empty for findings recovered by the
// heuristic prose scrape (parseFindings), which has no per-finding signal.
type finding struct {
file string
line int
title string
detail string
severity string // per-finding: critical/high/medium/small/trivial ("" if heuristic)
confidence string // per-finding: high/medium/low ("" if heuristic)
}
// runPayload is the POST /runs body. Field names match the gadfly-reports API EXACTLY.
// Token/cost fields are pointers so they marshal to JSON null when unknown
// (Gadfly does not currently meter tokens or cost).
type runPayload struct {
RunID string `json:"run_id"`
Repo string `json:"repo"`
PR int `json:"pr"`
Model string `json:"model"`
Provider string `json:"provider"`
Lenses int `json:"lenses"`
DurationSecs float64 `json:"duration_secs"`
InputTokens *int `json:"input_tokens"`
OutputTokens *int `json:"output_tokens"`
CostUSD *float64 `json:"cost_usd"`
}
// reportPayload is one element of the POST /reports JSON array. Field names match
// the gadfly-reports API EXACTLY.
type reportPayload struct {
Repo string `json:"repo"`
PR int `json:"pr"`
Lens string `json:"lens"`
File string `json:"file"`
Line int `json:"line"`
Title string `json:"title"`
Model string `json:"model"`
Provider string `json:"provider"`
RunID string `json:"run_id"`
RawSeverity string `json:"raw_severity"`
Confidence string `json:"confidence"` // per-finding high/medium/low ("" if heuristic)
Detail string `json:"detail"`
}
// emit posts the run and its findings to the gadfly-reports store. It is a no-op unless
// GADFLY_FINDINGS_URL is set. Best-effort: any error is logged to stderr and
// otherwise ignored. It NEVER returns an error, never calls os.Exit, and never
// writes to stdout.
func emit(results []specialistResult, elapsed time.Duration) {
base := strings.TrimRight(strings.TrimSpace(os.Getenv("GADFLY_FINDINGS_URL")), "/")
if base == "" {
return // telemetry disabled
}
repo := strings.TrimSpace(os.Getenv("GADFLY_REPO"))
pr := envAtoi("GADFLY_PR")
model := strings.TrimSpace(os.Getenv("GADFLY_MODEL"))
provider := modelProvider()
runID := fmt.Sprintf("%s#%d:%s", repo, pr, model)
run := runPayload{
RunID: runID,
Repo: repo,
PR: pr,
Model: model,
Provider: provider,
Lenses: len(results),
DurationSecs: elapsed.Seconds(),
// InputTokens/OutputTokens/CostUSD stay nil -> JSON null (not metered).
}
// collectFindings (consensus.go) skips clean/errored lenses and fills in each
// finding's severity (per-finding when structured, else derived from the lens
// verdict), so every reportPayload carries a canonical raw_severity.
var reports []reportPayload
for _, r := range results {
for _, f := range collectFindings(r) {
reports = append(reports, reportPayload{
Repo: repo,
PR: pr,
Lens: r.spec.Name,
File: f.file,
Line: f.line,
Title: f.title,
Model: model,
Provider: provider,
RunID: runID,
RawSeverity: f.severity,
Confidence: f.confidence,
Detail: f.detail,
})
}
}
client := &http.Client{Timeout: findingsHTTPTimeout}
token := strings.TrimSpace(os.Getenv("GADFLY_FINDINGS_TOKEN"))
if err := postJSON(client, base+"/runs", token, run); err != nil {
fmt.Fprintln(os.Stderr, "gadfly: findings emit (/runs) failed:", err)
}
if len(reports) > 0 {
if err := postJSON(client, base+"/reports", token, reports); err != nil {
fmt.Fprintln(os.Stderr, "gadfly: findings emit (/reports) failed:", err)
}
}
}
// parseFindings extracts findings from one lens's markdown output. A lens whose
// result errored (its output is the inline failure notice) yields nothing.
// Findings are anchored on "path:line" references, deduped by (file,line) keeping
// the first occurrence, and capped at maxFindingsPerLens.
func parseFindings(sp Specialist, out string) []finding {
_ = sp // signature kept symmetric with the caller; lens identity isn't needed here
trimmed := strings.TrimSpace(out)
if trimmed == "" || strings.HasPrefix(trimmed, "⚠️") {
return nil // empty or a "reviewer failed" notice
}
lines := strings.Split(out, "\n")
var findings []finding
seen := map[string]bool{}
for _, loc := range pathLineRe.FindAllStringSubmatchIndex(out, -1) {
file := out[loc[2]:loc[3]]
lineStr := out[loc[4]:loc[5]]
ln, err := strconv.Atoi(lineStr)
if err != nil {
continue
}
key := file + ":" + lineStr
if seen[key] {
continue
}
seen[key] = true
li := strings.Count(out[:loc[0]], "\n") // line index of the match
findings = append(findings, finding{
file: file,
line: ln,
title: deriveTitle(lines, li),
detail: paragraphAt(lines, li),
})
if len(findings) >= maxFindingsPerLens {
break
}
}
return findings
}
// deriveTitle finds the nearest preceding markdown heading / numbered item /
// bold lead-in at or above line li; failing that, the first sentence of the
// paragraph containing li.
func deriveTitle(lines []string, li int) string {
for i := li; i >= 0 && i < len(lines); i-- {
if t, ok := headingTitle(lines[i]); ok {
return t
}
}
return paragraphTitle(lines, li)
}
// headingTitle returns the title carried by a markdown heading-like line:
// an ATX "#"/"##"/"###" heading, a numbered "1." lead-in, or a "**bold**"
// lead-in (optionally behind a list bullet). ok is false for ordinary lines.
func headingTitle(line string) (string, bool) {
t := strings.TrimSpace(line)
if t == "" {
return "", false
}
if strings.HasPrefix(t, "#") {
h := strings.TrimSpace(strings.TrimLeft(t, "#"))
if h != "" {
return cleanHeading(h), true
}
return "", false
}
if m := numberedRe.FindStringSubmatch(t); m != nil {
if rest := strings.TrimSpace(m[1]); rest != "" {
return cleanHeading(rest), true
}
return "", false
}
// Bold lead-in, possibly behind a "- "/"* "/"+ " bullet. Strip only the
// bullet marker (NOT the bold asterisks) before testing for "**".
b := t
for _, pfx := range []string{"- ", "* ", "+ "} {
if strings.HasPrefix(b, pfx) {
b = strings.TrimSpace(b[len(pfx):])
break
}
}
if strings.HasPrefix(b, "**") {
if title := firstBold(b); title != "" {
return title, true
}
}
return "", false
}
// cleanHeading normalizes an extracted heading into a concise title: prefer a
// leading "**bold**" span, otherwise trim a trailing colon and truncate.
func cleanHeading(s string) string {
s = strings.TrimSpace(s)
if strings.HasPrefix(s, "**") {
if b := firstBold(s); b != "" {
return b
}
}
s = strings.TrimRight(s, ": ")
return truncate(s, 120)
}
// paragraphTitle falls back to the first sentence/line of the paragraph (the
// contiguous block of non-blank lines) containing line li, stripped of markdown
// list/emphasis markers.
func paragraphTitle(lines []string, li int) string {
if li < 0 || li >= len(lines) {
return ""
}
start := li
for start > 0 && strings.TrimSpace(lines[start-1]) != "" {
start--
}
first := leadingSentence(stripMarkers(lines[start]))
return truncate(strings.TrimSpace(first), 120)
}
// leadingSentence returns text up to the first sentence terminator that is
// followed by a space (". " / "! " / "? "), so a dot inside a filename like
// "util.go" does not prematurely cut the sentence. With no internal terminator
// it returns the line minus any trailing terminal punctuation. (auto.go's
// firstSentence splits on any ".", which would mis-cut a "path:line" sentence.)
func leadingSentence(s string) string {
best := -1
for _, sep := range []string{". ", "! ", "? "} {
if i := strings.Index(s, sep); i >= 0 && (best == -1 || i < best) {
best = i
}
}
if best >= 0 {
return s[:best]
}
return strings.TrimRight(s, ".!?")
}
// paragraphAt returns the paragraph (contiguous non-blank lines) containing li,
// joined onto one line and truncated — used as the finding's detail context.
func paragraphAt(lines []string, li int) string {
if li < 0 || li >= len(lines) {
return ""
}
start, end := li, li
for start > 0 && strings.TrimSpace(lines[start-1]) != "" {
start--
}
for end < len(lines)-1 && strings.TrimSpace(lines[end+1]) != "" {
end++
}
var parts []string
for i := start; i <= end; i++ {
if t := strings.TrimSpace(lines[i]); t != "" {
parts = append(parts, t)
}
}
return truncate(strings.Join(parts, " "), 500)
}
// stripMarkers removes a leading list bullet / numbered marker and surrounding
// emphasis/backticks from a line, leaving plain text.
func stripMarkers(s string) string {
s = strings.TrimSpace(s)
for _, pfx := range []string{"- ", "* ", "+ "} {
if strings.HasPrefix(s, pfx) {
s = strings.TrimSpace(s[len(pfx):])
break
}
}
if m := numberedRe.FindStringSubmatch(s); m != nil {
s = strings.TrimSpace(m[1])
}
return strings.Trim(s, "*`# ")
}
// firstBold returns the text inside the first "**...**" span of s, or "".
func firstBold(s string) string {
i := strings.Index(s, "**")
if i < 0 {
return ""
}
rest := s[i+2:]
j := strings.Index(rest, "**")
if j < 0 {
return ""
}
return strings.TrimSpace(rest[:j])
}
// truncate shortens s to at most n runes, appending an ellipsis when cut.
func truncate(s string, n int) string {
r := []rune(s)
if len(r) <= n {
return s
}
return strings.TrimSpace(string(r[:n])) + "…"
}
// postJSON marshals payload and POSTs it as application/json, attaching a Bearer
// token when non-empty. A non-2xx response is an error. The body is drained and
// closed so the connection can be reused.
func postJSON(client *http.Client, url, token string, payload any) error {
body, err := json.Marshal(payload)
if err != nil {
return err
}
req, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
if token != "" {
req.Header.Set("Authorization", "Bearer "+token)
}
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
_, _ = io.Copy(io.Discard, resp.Body)
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("%s: status %d", url, resp.StatusCode)
}
return nil
}
// envAtoi reads an integer env var, returning 0 when unset or unparseable.
func envAtoi(name string) int {
n, _ := strconv.Atoi(strings.TrimSpace(os.Getenv(name)))
return n
}
+257
View File
@@ -0,0 +1,257 @@
package main
import (
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"sync/atomic"
"testing"
"time"
)
// sampleLensMarkdown is a realistic lens output: a bold verdict lead-in, an ATX
// "###" heading anchoring one finding, a numbered + bold lead-in anchoring
// another, plus a duplicate "path:line" reference that must be deduped away.
const sampleLensMarkdown = "" +
"**Blocking issues found**\n" +
"\n" +
"### Swallowed error\n" +
"In `run/executor.go:166` the returned error is ignored, so a failed step is\n" +
"reported as success.\n" +
"\n" +
"1. **Missing nil check** — `compact/compactor.go:225` dereferences `cfg`\n" +
" which can be nil when the caller passes an empty config.\n" +
"\n" +
"See again run/executor.go:166 for the same root cause.\n"
func TestParseFindings(t *testing.T) {
got := parseFindings(Specialist{Name: "security"}, sampleLensMarkdown)
if len(got) != 2 {
t.Fatalf("parseFindings returned %d findings, want 2 (deduped): %+v", len(got), got)
}
if got[0].file != "run/executor.go" || got[0].line != 166 {
t.Errorf("finding[0] location = %s:%d, want run/executor.go:166", got[0].file, got[0].line)
}
if got[0].title != "Swallowed error" {
t.Errorf("finding[0] title = %q, want %q (nearest ### heading)", got[0].title, "Swallowed error")
}
if got[1].file != "compact/compactor.go" || got[1].line != 225 {
t.Errorf("finding[1] location = %s:%d, want compact/compactor.go:225", got[1].file, got[1].line)
}
if got[1].title != "Missing nil check" {
t.Errorf("finding[1] title = %q, want %q (numbered bold lead-in)", got[1].title, "Missing nil check")
}
if got[1].detail == "" {
t.Error("finding[1] detail should carry the paragraph context, got empty")
}
}
func TestParseFindings_ErroredLensYieldsNone(t *testing.T) {
notice := "⚠️ This reviewer failed to complete: context deadline exceeded (file.go:10)"
if f := parseFindings(Specialist{Name: "security"}, notice); len(f) != 0 {
t.Errorf("a failure-notice lens should yield no findings, got %+v", f)
}
if f := parseFindings(Specialist{Name: "security"}, " "); len(f) != 0 {
t.Errorf("empty output should yield no findings, got %+v", f)
}
}
func TestParseFindings_FallbackParagraphTitle(t *testing.T) {
md := "No material issues found.\n\nThe helper in src/util.go:42 returns the wrong code. Investigate.\n"
got := parseFindings(Specialist{Name: "correctness"}, md)
if len(got) != 1 {
t.Fatalf("want 1 finding, got %d: %+v", len(got), got)
}
if got[0].title != "The helper in src/util.go:42 returns the wrong code" {
t.Errorf("fallback title = %q, want first sentence of the paragraph", got[0].title)
}
}
func TestEmit_PostsRunsAndReports(t *testing.T) {
var runs, reports int32
var runBody map[string]any
var reportBody []map[string]any
var sawAuth string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if a := r.Header.Get("Authorization"); a != "" {
sawAuth = a
}
data, _ := io.ReadAll(r.Body)
switch r.URL.Path {
case "/runs":
atomic.AddInt32(&runs, 1)
_ = json.Unmarshal(data, &runBody)
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte(`{"run_id":"x"}`))
case "/reports":
atomic.AddInt32(&reports, 1)
_ = json.Unmarshal(data, &reportBody)
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte(`{"finding_ids":[1,2]}`))
default:
http.Error(w, "not found", http.StatusNotFound)
}
}))
defer srv.Close()
t.Setenv("GADFLY_FINDINGS_URL", srv.URL)
t.Setenv("GADFLY_FINDINGS_TOKEN", "secret-token")
t.Setenv("GADFLY_REPO", "owner/repo")
t.Setenv("GADFLY_PR", "7")
t.Setenv("GADFLY_MODEL", "ollama-cloud/qwen3")
t.Setenv("GADFLY_PROVIDER", "")
results := []specialistResult{
{spec: Specialist{Name: "security", Title: "🔒 Security"}, out: sampleLensMarkdown, verdict: verdictBlocking},
}
emit(results, 3*time.Second)
if got := atomic.LoadInt32(&runs); got != 1 {
t.Fatalf("/runs received %d times, want exactly 1", got)
}
if got := atomic.LoadInt32(&reports); got != 1 {
t.Fatalf("/reports received %d times, want exactly 1", got)
}
if sawAuth != "Bearer secret-token" {
t.Errorf("Authorization header = %q, want %q", sawAuth, "Bearer secret-token")
}
// --- /runs body: exact field names + values ---
for _, k := range []string{"run_id", "repo", "pr", "model", "provider", "lenses", "duration_secs", "input_tokens", "output_tokens", "cost_usd"} {
if _, ok := runBody[k]; !ok {
t.Errorf("/runs body missing field %q (got keys %v)", k, keysOf(runBody))
}
}
if runBody["run_id"] != "owner/repo#7:ollama-cloud/qwen3" {
t.Errorf("run_id = %v, want owner/repo#7:ollama-cloud/qwen3", runBody["run_id"])
}
if runBody["repo"] != "owner/repo" {
t.Errorf("repo = %v, want owner/repo", runBody["repo"])
}
if runBody["pr"] != float64(7) {
t.Errorf("pr = %v, want 7", runBody["pr"])
}
if runBody["model"] != "ollama-cloud/qwen3" {
t.Errorf("model = %v, want ollama-cloud/qwen3", runBody["model"])
}
if runBody["provider"] != "ollama-cloud" {
t.Errorf("provider = %v, want ollama-cloud", runBody["provider"])
}
if runBody["lenses"] != float64(1) {
t.Errorf("lenses = %v, want 1", runBody["lenses"])
}
if runBody["duration_secs"] != float64(3) {
t.Errorf("duration_secs = %v, want 3", runBody["duration_secs"])
}
if v, ok := runBody["input_tokens"]; !ok || v != nil {
t.Errorf("input_tokens = %v, want JSON null", v)
}
// --- /reports body: exact field names + values ---
if len(reportBody) != 2 {
t.Fatalf("/reports array length = %d, want 2", len(reportBody))
}
for _, k := range []string{"repo", "pr", "lens", "file", "line", "title", "model", "provider", "run_id", "raw_severity", "confidence", "detail"} {
if _, ok := reportBody[0][k]; !ok {
t.Errorf("/reports[0] missing field %q (got keys %v)", k, keysOf(reportBody[0]))
}
}
if reportBody[0]["lens"] != "security" {
t.Errorf("reports[0].lens = %v, want security", reportBody[0]["lens"])
}
if reportBody[0]["file"] != "run/executor.go" {
t.Errorf("reports[0].file = %v, want run/executor.go", reportBody[0]["file"])
}
if reportBody[0]["line"] != float64(166) {
t.Errorf("reports[0].line = %v, want 166", reportBody[0]["line"])
}
// No structured block in sampleLensMarkdown, so this is a heuristic-scraped
// finding: raw_severity is the canonical word derived from the lens verdict
// (Blocking -> "high"), not the full verdict phrase.
if reportBody[0]["raw_severity"] != "high" {
t.Errorf("reports[0].raw_severity = %v, want 'high'", reportBody[0]["raw_severity"])
}
if reportBody[0]["run_id"] != "owner/repo#7:ollama-cloud/qwen3" {
t.Errorf("reports[0].run_id = %v, want owner/repo#7:ollama-cloud/qwen3", reportBody[0]["run_id"])
}
}
// A clean "No material issues found" lens must NOT emit findings, even though
// its markdown contains path:line references (those are verification notes, not
// problems). /runs is still posted; /reports is not.
func TestEmit_SkipsCleanVerdictLens(t *testing.T) {
var runs, reports int32
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/runs":
atomic.AddInt32(&runs, 1)
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte(`{"run_id":"x"}`))
case "/reports":
atomic.AddInt32(&reports, 1)
w.WriteHeader(http.StatusOK)
default:
http.Error(w, "not found", http.StatusNotFound)
}
}))
defer srv.Close()
t.Setenv("GADFLY_FINDINGS_URL", srv.URL)
t.Setenv("GADFLY_REPO", "owner/repo")
t.Setenv("GADFLY_PR", "7")
t.Setenv("GADFLY_MODEL", "ollama-cloud/qwen3")
// Clean verdict, but the markdown is full of path:line "verified X" notes.
cleanMarkdown := "No material issues found.\n\nVerified `run/executor.go:166` handles the error.\n"
results := []specialistResult{
{spec: Specialist{Name: "security"}, out: cleanMarkdown, verdict: verdictClean},
}
emit(results, time.Second)
if got := atomic.LoadInt32(&runs); got != 1 {
t.Fatalf("/runs received %d times, want 1 (the run is still recorded)", got)
}
if got := atomic.LoadInt32(&reports); got != 0 {
t.Fatalf("/reports received %d times, want 0 (clean lens emits no findings)", got)
}
}
func TestEmit_DisabledMakesNoRequests(t *testing.T) {
var hits int32
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
atomic.AddInt32(&hits, 1)
w.WriteHeader(http.StatusOK)
}))
defer srv.Close()
t.Setenv("GADFLY_FINDINGS_URL", "") // disabled
t.Setenv("GADFLY_REPO", "owner/repo")
t.Setenv("GADFLY_PR", "7")
t.Setenv("GADFLY_MODEL", "ollama-cloud/qwen3")
results := []specialistResult{
{spec: Specialist{Name: "security"}, out: sampleLensMarkdown, verdict: verdictBlocking},
}
// Must not panic and must not hit the server.
emit(results, time.Second)
emit(nil, 0)
if got := atomic.LoadInt32(&hits); got != 0 {
t.Fatalf("telemetry disabled but server received %d requests, want 0", got)
}
}
func keysOf(m map[string]any) []string {
ks := make([]string, 0, len(m))
for k := range m {
ks = append(ks, k)
}
return ks
}
+268
View File
@@ -0,0 +1,268 @@
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"os"
"os/exec"
"strconv"
"strings"
"syscall"
"unicode/utf8"
llm "gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
// reviewEngine runs a single agent pass against the checked-out repo and returns
// the model's text answer. It is the one primitive both review passes use — the
// draft review and the adversarial recheck — so the rest of the pipeline
// (specialist composition, recheck orchestration, consolidation, emit) is
// engine-agnostic. Two implementations:
//
// - majordomoEngine: the original path — a majordomo tool-using agent loop
// (read_file/grep/… over a sandboxed repoFS).
// - claudeCodeEngine: shells out to the `claude` CLI in print mode, which
// brings its OWN repo tools; gadfly just feeds it the prompt and reads back
// the final text.
//
// maxSteps is the tool-step budget for engines that have one (majordomo); the
// claude-code engine manages its own loop and ignores it.
type reviewEngine interface {
runPass(ctx context.Context, system, task string, maxSteps int) (string, error)
}
// majordomoEngine drives the in-process review path over the repo sandbox. It no
// longer calls majordomo's agent loop directly: each pass runs through an
// executus run.Executor (see executus.go), which adds context compaction, run
// bounding, the per-PR budget gate, and the wrap-up critic. mdl is retained only
// so auto-select can fall back to the review model as its selector.
type majordomoEngine struct {
rex *reviewExecutor
mdl llm.Model
}
func (e *majordomoEngine) runPass(ctx context.Context, system, task string, maxSteps int) (string, error) {
return e.rex.run(ctx, system, task, maxSteps)
}
// claudeCodeEngine reviews by shelling out to the `claude` CLI (Claude Code) in
// non-interactive print mode. Claude Code reads the checked-out tree with its
// own read tools (so it verifies findings against real code, like the agentic
// majordomo path), and we parse its final answer out of `--output-format json`.
//
// Auth is inherited from the environment: the default backend is a Pro/Max
// subscription via CLAUDE_CODE_OAUTH_TOKEN (no `--bare`). See README.
type claudeCodeEngine struct {
bin string // CLI binary (GADFLY_CLAUDE_BIN, default "claude")
model string // --model value ("" = CLI default)
repoDir string // cwd for the CLI, so its tools read the checked-out tree
permissionMode string // --permission-mode (default "plan": read-only, no edits)
allowedTools string // --allowedTools value, passed verbatim ("" = omit)
extraArgs []string // appended verbatim (GADFLY_CLAUDE_EXTRA_ARGS)
thinkingTokens int // MAX_THINKING_TOKENS for the subprocess; 0 = leave default
}
// maxThinkingTokens is the extended-thinking budget used for a "claude-code/<model>:max"
// spec — Claude Code's high "ultrathink" tier. Set as MAX_THINKING_TOKENS on the
// subprocess; harmless (a no-op) if the CLI build doesn't honor it.
const maxThinkingTokens = 31999
// isClaudeCodeSpec reports whether a GADFLY_MODEL spec selects the claude-code
// engine: the bare id "claude-code" or a "claude-code/<model>" form.
func isClaudeCodeSpec(model string) bool {
m := strings.TrimSpace(model)
return m == "claude-code" || strings.HasPrefix(m, "claude-code/")
}
// newClaudeCodeEngine builds the engine from the GADFLY_MODEL spec and the
// optional GADFLY_CLAUDE_* overrides. The part after the slash in
// "claude-code/<model>[:<thinking>]" becomes --model (e.g. "claude-code/sonnet"
// → "sonnet"), with an optional thinking suffix: ":max" forces the high
// extended-thinking budget and ":<n>" sets a specific MAX_THINKING_TOKENS.
// GADFLY_CLAUDE_MODEL overrides the model id (the thinking suffix still applies).
// It does not verify the CLI is installed — a missing binary surfaces as a normal
// pass error (advisory, never fatal).
func newClaudeCodeEngine(spec, repoDir string) *claudeCodeEngine {
var model string
thinking := 0
if _, after, ok := strings.Cut(strings.TrimSpace(spec), "/"); ok {
after = strings.TrimSpace(after)
// Optional ":<thinking>" suffix. Claude model aliases/ids contain no ":",
// so a colon unambiguously separates the thinking tier here.
if m, t, hasColon := strings.Cut(after, ":"); hasColon {
model = strings.TrimSpace(m)
thinking = parseThinking(strings.TrimSpace(t))
} else {
model = after
}
}
if env := strings.TrimSpace(os.Getenv("GADFLY_CLAUDE_MODEL")); env != "" {
model = env
}
return &claudeCodeEngine{
bin: envOr("GADFLY_CLAUDE_BIN", "claude"),
model: model,
repoDir: repoDir,
permissionMode: envOr("GADFLY_CLAUDE_PERMISSION_MODE", "plan"),
allowedTools: strings.TrimSpace(os.Getenv("GADFLY_CLAUDE_ALLOWED_TOOLS")),
extraArgs: strings.Fields(os.Getenv("GADFLY_CLAUDE_EXTRA_ARGS")),
thinkingTokens: thinking,
}
}
// parseThinking maps a spec thinking suffix to a MAX_THINKING_TOKENS budget:
// "max" → maxThinkingTokens, a positive integer → itself, anything else → 0 (off).
func parseThinking(s string) int {
if strings.EqualFold(s, "max") {
return maxThinkingTokens
}
if n, err := strconv.Atoi(s); err == nil && n > 0 {
return n
}
return 0
}
// args assembles the `claude` argv for one pass. Factored out (and pure) so it
// can be unit-tested without invoking the CLI. The system prompt is layered on
// top of Claude Code's own via --append-system-prompt; the task is the -p
// prompt.
func (e *claudeCodeEngine) args(system, task string) []string {
a := []string{"-p", task, "--output-format", "json", "--append-system-prompt", system}
if e.model != "" {
a = append(a, "--model", e.model)
}
if e.permissionMode != "" {
a = append(a, "--permission-mode", e.permissionMode)
}
if e.allowedTools != "" {
a = append(a, "--allowedTools", e.allowedTools)
}
return append(a, e.extraArgs...)
}
// claudeResult is the subset of `claude --output-format json` we read.
type claudeResult struct {
Result string `json:"result"`
IsError bool `json:"is_error"`
Subtype string `json:"subtype"`
}
func (e *claudeCodeEngine) runPass(ctx context.Context, system, task string, _ int) (string, error) {
cmd := exec.CommandContext(ctx, e.bin, e.args(system, task)...)
cmd.Dir = e.repoDir
cmd.Env = claudeEnv() // minimal env — don't hand GITEA_TOKEN et al. to the CLI
if e.thinkingTokens > 0 {
// Force an extended-thinking budget for this run (a "...:max" spec).
cmd.Env = append(cmd.Env, "MAX_THINKING_TOKENS="+strconv.Itoa(e.thinkingTokens))
}
// Put the CLI and the Node children it spawns in their own process group and
// kill the WHOLE group on context cancel, so a timed-out lens can't leave
// orphaned claude/node processes behind in the container.
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
cmd.Cancel = func() error {
if cmd.Process != nil {
_ = syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
}
return nil
}
var stdout, stderr bytes.Buffer
cmd.Stdout = &stdout
cmd.Stderr = &stderr
runErr := cmd.Run()
// A cancelled/timed-out run must surface as an error, never as whatever
// partial bytes the CLI flushed before it was killed.
if ctx.Err() != nil {
return "", fmt.Errorf("claude -p %v", ctx.Err())
}
var res claudeResult
parsed := json.Unmarshal(bytes.TrimSpace(stdout.Bytes()), &res) == nil
// Clean exit: trust the parsed JSON answer, and ONLY it — never fall back to
// the raw JSON envelope when the result is empty.
if runErr == nil && parsed {
if res.IsError {
return "", fmt.Errorf("claude reported error (%s): %s", res.Subtype, truncateForErr(res.Result))
}
if out := strings.TrimSpace(res.Result); out != "" {
return out, nil
}
return "", fmt.Errorf("claude -p returned an empty result")
}
if runErr != nil {
// Prefer the CLI's own structured error message when it gave one.
if parsed && res.IsError && strings.TrimSpace(res.Result) != "" {
return "", fmt.Errorf("claude reported error (%s): %s", res.Subtype, truncateForErr(res.Result))
}
detail := truncateForErr(stderr.String())
if detail == "" {
detail = truncateForErr(stdout.String())
}
if detail != "" {
return "", fmt.Errorf("claude -p failed: %v: %s", runErr, detail)
}
return "", fmt.Errorf("claude -p failed: %v", runErr)
}
// Clean exit but stdout wasn't the expected JSON envelope: degrade to the raw
// text so a CLI format change still yields a review instead of nothing.
if raw := strings.TrimSpace(stdout.String()); raw != "" {
return raw, nil
}
return "", fmt.Errorf("claude -p produced no parseable output")
}
// claudeEnv builds a minimal environment for the `claude` subprocess: only what
// the CLI needs (PATH/HOME, its auth tokens, locale, Node/XDG/GADFLY_CLAUDE_*
// knobs), deliberately dropping the rest of the runner's secrets — GITEA_TOKEN,
// GADFLY_FINDINGS_TOKEN, provider keys — so they never reach the third-party
// CLI. Defense in depth: the parent already holds them, but the CLI has no need.
func claudeEnv() []string {
keep := func(k string) bool {
switch k {
case "PATH", "HOME", "USER", "LOGNAME", "TMPDIR", "LANG", "TERM", "SHELL", "MAX_THINKING_TOKENS":
return true
}
return strings.HasPrefix(k, "LC_") ||
strings.HasPrefix(k, "CLAUDE_") ||
strings.HasPrefix(k, "ANTHROPIC_") ||
strings.HasPrefix(k, "GADFLY_CLAUDE_") ||
strings.HasPrefix(k, "NODE_") ||
strings.HasPrefix(k, "XDG_")
}
var env []string
for _, kv := range os.Environ() {
if k, _, ok := strings.Cut(kv, "="); ok && keep(k) {
env = append(env, kv)
}
}
return env
}
// truncateForErr caps CLI error detail so a stderr dump can't bloat the comment,
// cutting on a rune boundary so it never emits invalid UTF-8.
func truncateForErr(s string) string {
s = strings.TrimSpace(s)
const max = 800
if len(s) <= max {
return s
}
cut := max
for cut > 0 && !utf8.RuneStart(s[cut]) {
cut--
}
return s[:cut] + "…"
}
// envOr returns the env var value or a default when unset/blank.
func envOr(name, def string) string {
if v := strings.TrimSpace(os.Getenv(name)); v != "" {
return v
}
return def
}
+281
View File
@@ -0,0 +1,281 @@
package main
import (
"context"
"os"
"slices"
"strings"
"testing"
"unicode/utf8"
)
func TestIsClaudeCodeSpec(t *testing.T) {
cases := map[string]bool{
"claude-code": true,
"claude-code/sonnet": true,
"claude-code/opus": true,
"claude-code/claude-opus-4-8": true,
" claude-code ": true, // trimmed
"qwen3-coder:480b-cloud": false,
"claude-code-extra": false, // not the bare id, not a "/" form
"sonnet": false,
"": false,
}
for spec, want := range cases {
if got := isClaudeCodeSpec(spec); got != want {
t.Errorf("isClaudeCodeSpec(%q) = %v, want %v", spec, got, want)
}
}
}
func TestNewClaudeCodeEngineModel(t *testing.T) {
// model derived from the spec's "/<model>" suffix
t.Setenv("GADFLY_CLAUDE_MODEL", "")
if e := newClaudeCodeEngine("claude-code/sonnet", "/repo"); e.model != "sonnet" {
t.Errorf("model = %q, want sonnet", e.model)
}
// bare spec → CLI default (no --model)
if e := newClaudeCodeEngine("claude-code", "/repo"); e.model != "" {
t.Errorf("model = %q, want empty for bare spec", e.model)
}
// GADFLY_CLAUDE_MODEL overrides the spec suffix
t.Setenv("GADFLY_CLAUDE_MODEL", "opus")
if e := newClaudeCodeEngine("claude-code/sonnet", "/repo"); e.model != "opus" {
t.Errorf("model = %q, want opus (env override)", e.model)
}
}
func TestClaudeCodeEngineDefaults(t *testing.T) {
t.Setenv("GADFLY_CLAUDE_BIN", "")
t.Setenv("GADFLY_CLAUDE_PERMISSION_MODE", "")
t.Setenv("GADFLY_CLAUDE_ALLOWED_TOOLS", "")
t.Setenv("GADFLY_CLAUDE_EXTRA_ARGS", "")
e := newClaudeCodeEngine("claude-code", "/repo")
if e.bin != "claude" {
t.Errorf("bin = %q, want claude", e.bin)
}
if e.permissionMode != "plan" {
t.Errorf("permissionMode = %q, want plan", e.permissionMode)
}
if e.repoDir != "/repo" {
t.Errorf("repoDir = %q, want /repo", e.repoDir)
}
}
// argAfter returns the value following flag in args, or "" if absent.
func argAfter(args []string, flag string) string {
if i := slices.Index(args, flag); i >= 0 && i+1 < len(args) {
return args[i+1]
}
return ""
}
func TestClaudeCodeArgs(t *testing.T) {
t.Setenv("GADFLY_CLAUDE_MODEL", "")
t.Setenv("GADFLY_CLAUDE_PERMISSION_MODE", "")
t.Setenv("GADFLY_CLAUDE_ALLOWED_TOOLS", "Read,Grep,Glob")
t.Setenv("GADFLY_CLAUDE_EXTRA_ARGS", "--max-turns 30")
e := newClaudeCodeEngine("claude-code/sonnet", "/repo")
args := e.args("SYS-PROMPT", "TASK-PROMPT")
// task is the -p value; json output; system appended; model + policy present.
if argAfter(args, "-p") != "TASK-PROMPT" {
t.Errorf("-p = %q, want TASK-PROMPT", argAfter(args, "-p"))
}
if argAfter(args, "--output-format") != "json" {
t.Errorf("--output-format = %q, want json", argAfter(args, "--output-format"))
}
if argAfter(args, "--append-system-prompt") != "SYS-PROMPT" {
t.Errorf("--append-system-prompt = %q, want SYS-PROMPT", argAfter(args, "--append-system-prompt"))
}
if argAfter(args, "--model") != "sonnet" {
t.Errorf("--model = %q, want sonnet", argAfter(args, "--model"))
}
if argAfter(args, "--permission-mode") != "plan" {
t.Errorf("--permission-mode = %q, want plan", argAfter(args, "--permission-mode"))
}
if argAfter(args, "--allowedTools") != "Read,Grep,Glob" {
t.Errorf("--allowedTools = %q, want Read,Grep,Glob", argAfter(args, "--allowedTools"))
}
// extra args appended verbatim (split on whitespace)
if !strings.Contains(strings.Join(args, " "), "--max-turns 30") {
t.Errorf("extra args not appended: %v", args)
}
}
func TestClaudeCodeArgsBareModelOmitsFlag(t *testing.T) {
t.Setenv("GADFLY_CLAUDE_MODEL", "")
t.Setenv("GADFLY_CLAUDE_ALLOWED_TOOLS", "") // omit when blank
t.Setenv("GADFLY_CLAUDE_EXTRA_ARGS", "")
e := newClaudeCodeEngine("claude-code", "/repo")
args := e.args("s", "t")
if slices.Contains(args, "--model") {
t.Errorf("--model should be omitted for a bare claude-code spec: %v", args)
}
if slices.Contains(args, "--allowedTools") {
t.Errorf("--allowedTools should be omitted when blank: %v", args)
}
}
func TestClaudeEnvFilters(t *testing.T) {
t.Setenv("GITEA_TOKEN", "secret-gitea")
t.Setenv("OLLAMA_API_KEY", "secret-ollama")
t.Setenv("GADFLY_API_KEY", "secret-gadfly")
t.Setenv("GADFLY_FINDINGS_TOKEN", "secret-findings")
t.Setenv("CLAUDE_CODE_OAUTH_TOKEN", "keep-claude")
t.Setenv("ANTHROPIC_API_KEY", "keep-anthropic")
t.Setenv("GADFLY_CLAUDE_MODEL", "keep-knob")
env := claudeEnv()
has := func(k string) bool {
for _, kv := range env {
if strings.HasPrefix(kv, k+"=") {
return true
}
}
return false
}
// kept: the CLI's auth + its own knobs + PATH
for _, k := range []string{"CLAUDE_CODE_OAUTH_TOKEN", "ANTHROPIC_API_KEY", "GADFLY_CLAUDE_MODEL", "PATH"} {
if !has(k) {
t.Errorf("claudeEnv dropped %s, but it should be kept", k)
}
}
// dropped: the runner's secrets the CLI doesn't need
for _, k := range []string{"GITEA_TOKEN", "OLLAMA_API_KEY", "GADFLY_API_KEY", "GADFLY_FINDINGS_TOKEN"} {
if has(k) {
t.Errorf("claudeEnv leaked %s into the subprocess env", k)
}
}
}
func TestTruncateForErrRuneSafe(t *testing.T) {
// 900 multibyte runes (3 bytes each) -> well over the 800-byte cap; the cut
// must land on a rune boundary so the result stays valid UTF-8.
s := strings.Repeat("€", 900)
got := truncateForErr(s)
if !utf8.ValidString(got) {
t.Fatalf("truncateForErr produced invalid UTF-8")
}
if !strings.HasSuffix(got, "…") {
t.Fatalf("truncateForErr should append an ellipsis when truncating")
}
// short strings pass through untouched
if truncateForErr(" hi ") != "hi" {
t.Fatalf("truncateForErr should trim and pass short strings through")
}
}
// stubClaude writes an executable shell stub that prints body and exits code,
// and returns an engine pointed at it.
func stubClaude(t *testing.T, body string, code int) *claudeCodeEngine {
t.Helper()
dir := t.TempDir()
path := dir + "/claude-stub.sh"
script := "#!/bin/sh\nprintf '%s' " + shSingleQuote(body) + "\nexit " + itoa(code) + "\n"
if err := os.WriteFile(path, []byte(script), 0o755); err != nil {
t.Fatal(err)
}
return &claudeCodeEngine{bin: path, repoDir: dir}
}
func shSingleQuote(s string) string { return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'" }
func itoa(i int) string { return string(rune('0' + i)) } // single-digit exit codes only
func TestRunPassCleanResult(t *testing.T) {
e := stubClaude(t, `{"result":"REVIEW TEXT","is_error":false}`, 0)
out, err := e.runPass(context.Background(), "sys", "task", 0)
if err != nil || out != "REVIEW TEXT" {
t.Fatalf("clean result: got (%q, %v), want (REVIEW TEXT, nil)", out, err)
}
}
func TestRunPassEmptyResultIsError(t *testing.T) {
// JSON parses, exit 0, but result empty: must NOT return the raw JSON blob.
e := stubClaude(t, `{"result":"","is_error":false}`, 0)
out, err := e.runPass(context.Background(), "sys", "task", 0)
if err == nil {
t.Fatalf("empty result should be an error, got out=%q", out)
}
if strings.Contains(out, "{") {
t.Fatalf("empty result must not leak raw JSON, got %q", out)
}
}
func TestRunPassIsErrorFlag(t *testing.T) {
e := stubClaude(t, `{"result":"boom","is_error":true,"subtype":"error_max_turns"}`, 0)
_, err := e.runPass(context.Background(), "sys", "task", 0)
if err == nil || !strings.Contains(err.Error(), "claude reported error") {
t.Fatalf("is_error should surface as an error, got %v", err)
}
}
func TestRunPassNonZeroNoJSON(t *testing.T) {
e := stubClaude(t, "fatal: auth failed", 1)
_, err := e.runPass(context.Background(), "sys", "task", 0)
if err == nil || !strings.Contains(err.Error(), "claude -p failed") {
t.Fatalf("non-zero exit should error with detail, got %v", err)
}
}
func TestClaudeCodeThinking(t *testing.T) {
t.Setenv("GADFLY_CLAUDE_MODEL", "")
cases := []struct {
spec string
wantModel string
wantThink int
}{
{"claude-code/opus", "opus", 0},
{"claude-code/opus:max", "opus", maxThinkingTokens},
{"claude-code/sonnet:20000", "sonnet", 20000},
{"claude-code/opus:bogus", "opus", 0}, // unrecognized suffix -> off
{"claude-code", "", 0},
}
for _, c := range cases {
e := newClaudeCodeEngine(c.spec, "/repo")
if e.model != c.wantModel || e.thinkingTokens != c.wantThink {
t.Errorf("newClaudeCodeEngine(%q) = (model %q, think %d), want (%q, %d)",
c.spec, e.model, e.thinkingTokens, c.wantModel, c.wantThink)
}
}
}
func TestClaudeCodeThinkingEnvOverrideKeepsSuffix(t *testing.T) {
// GADFLY_CLAUDE_MODEL overrides the model id, but the :max thinking from the
// spec still applies.
t.Setenv("GADFLY_CLAUDE_MODEL", "claude-opus-4-8")
e := newClaudeCodeEngine("claude-code/opus:max", "/repo")
if e.model != "claude-opus-4-8" {
t.Errorf("model = %q, want claude-opus-4-8 (env override)", e.model)
}
if e.thinkingTokens != maxThinkingTokens {
t.Errorf("thinkingTokens = %d, want %d (suffix still applies)", e.thinkingTokens, maxThinkingTokens)
}
}
// TestRunPassInjectsThinkingTokens verifies the engine actually puts
// MAX_THINKING_TOKENS into the subprocess env for a ":max" spec (and not for a
// plain spec). The stub echoes the value it received back as the result.
func TestRunPassInjectsThinkingTokens(t *testing.T) {
t.Setenv("GADFLY_CLAUDE_MODEL", "")
t.Setenv("MAX_THINKING_TOKENS", "") // not in the test's own env
dir := t.TempDir()
stub := dir + "/claude-stub.sh"
// Report the env var the CLI was launched with.
script := "#!/bin/sh\nprintf '{\"result\":\"MTT=%s\",\"is_error\":false}' \"${MAX_THINKING_TOKENS:-unset}\"\n"
if err := os.WriteFile(stub, []byte(script), 0o755); err != nil {
t.Fatal(err)
}
maxEng := newClaudeCodeEngine("claude-code/opus:max", dir)
maxEng.bin = stub
if out, err := maxEng.runPass(context.Background(), "s", "t", 0); err != nil || out != "MTT=31999" {
t.Fatalf(":max run: got (%q, %v), want (MTT=31999, nil)", out, err)
}
plainEng := newClaudeCodeEngine("claude-code/opus", dir)
plainEng.bin = stub
if out, err := plainEng.runPass(context.Background(), "s", "t", 0); err != nil || out != "MTT=unset" {
t.Fatalf("plain run: got (%q, %v), want (MTT=unset, nil)", out, err)
}
}
+371
View File
@@ -0,0 +1,371 @@
package main
// executus.go wires gadfly's agentic review path onto the executus run kernel
// (gitea.stevedudenhoeffer.com/steve/executus), layered above majordomo. The
// majordomoEngine no longer drives majordomo's agent loop directly; it builds a
// run.Executor that gives gadfly, for free:
//
// - context compaction (executus/compact): once the transcript a step would
// SEND crosses a token threshold derived from the model's real context
// window, the runaway middle is folded into a one-paragraph summary by a
// cheap summarizer model — so a big diff + accumulating read_file/grep
// results can't balloon every re-sent step (the large-PR burn).
// - run bounding + a per-PR spend budget (executus/run Ports.Budget): a hard
// token/seconds ceiling so a pathological PR can't drain the usage block.
// - the wrap-up nudge, re-expressed as an executus Critic (Ports.Critic): the
// steer that tells a step-hungry model to stop investigating and write its
// answer is now the critic seam, not a bespoke RunOption.
//
// Everything degrades to today's behavior when unconfigured: nil summarizer or a
// 0 context window disables compaction; nil budget disables the ceiling; the
// claude-code engine shells out and is unaffected by any of this.
//
// gadfly keeps its own model.go resolution (so GADFLY_ENDPOINT_<NAME> http
// aliases, failover chains, and the claude-code engine all survive) — the
// run.Executor is handed gadfly's already-resolved model via a trivial resolver,
// not routed through executus's tier table.
import (
"context"
"errors"
"fmt"
"os"
"strconv"
"strings"
"sync"
"sync/atomic"
"time"
llm "gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/executus/compact"
"gitea.stevedudenhoeffer.com/steve/executus/model"
exrun "gitea.stevedudenhoeffer.com/steve/executus/run"
exectool "gitea.stevedudenhoeffer.com/steve/executus/tool"
)
const (
// defaultCompactRatio is the fraction of the model's context window at which
// compaction fires. It is deliberately LOWER than executus's own 0.7 default:
// on the large-PR burn the per-step transcript is the embedded diff (~17K) plus
// accumulating read_file results, which rarely reaches 0.7×262K≈183K — so a
// 0.7 threshold never bites. ~0.45×262K≈118K folds the runaway middle while a
// transcript is still well under the cap. Override with GADFLY_COMPACT_RATIO.
defaultCompactRatio = 0.45
// defaultCompactKeepRecent / defaultCompactSummaryWords mirror executus's own
// compactor defaults; surfaced as gadfly env knobs for tuning.
defaultCompactKeepRecent = 8
defaultCompactSummaryWords = 200
// contextTokenLookupTimeout bounds the one-shot /api/show call that resolves a
// cloud model's context window at executor-build time. Kept short so a slow or
// unreachable endpoint adds at most this to startup before degrading to
// no-compaction (rather than the provider cache's default 15s).
contextTokenLookupTimeout = 5 * time.Second
)
// runSeq mints a unique-per-process RunID suffix for each executor run so audit
// and the run kernel can tell one pass from another within a binary process.
var runSeq atomic.Uint64
// wrappedTool adapts an already-built majordomo llm.Tool (gadfly's sandboxed
// read_file/grep/get_diff/… closures over the repoFS) to executus's tool.Tool
// interface so the run kernel can build a toolbox from them by name. gadfly's
// tools need no caller/channel identity, so BuildLLM ignores the Invocation and
// returns the pre-built tool; Permission is the zero value (private, ungated).
type wrappedTool struct{ t llm.Tool }
func (w wrappedTool) Name() string { return w.t.Name }
func (w wrappedTool) Description() string { return w.t.Description }
func (w wrappedTool) Permission() exectool.Permission { return exectool.Permission{} }
func (w wrappedTool) BuildLLM(_ exectool.Invocation) llm.Tool { return w.t }
// gadflyToolRegistry registers the repo's read-only tools (plus the optional
// delegate_investigation worker tool) in a fresh executus tool.Registry and
// returns it along with the tool names for RunnableAgent.LowLevelTools.
func gadflyToolRegistry(fs *repoFS) (exectool.Registry, []string, error) {
reg := exectool.NewRegistry()
tools := fs.allTools()
names := make([]string, 0, len(tools))
for _, t := range tools {
if err := reg.Register(wrappedTool{t: t}); err != nil {
return nil, nil, fmt.Errorf("register tool %q: %w", t.Name, err)
}
names = append(names, t.Name)
}
return reg, names, nil
}
// gadflyBudget is gadfly's per-PR spend ceiling, satisfying run.Ports.Budget.
// It gates a run BEFORE it makes any model call (Check) once the process has
// spent its token or wall-clock allowance on this PR. Tokens are fed in
// out-of-band via addUsage (the Budget interface's Commit only carries seconds);
// the engine calls addUsage after each pass with run.Result.Usage. A nil
// *gadflyBudget is never installed — caps of 0 mean "unlimited", so the port is
// only wired when at least one cap is set.
//
// The guard is PASS-granular: Check runs before each pass, so it stops the NEXT
// pass once the budget is spent but cannot abort a single runaway pass mid-flight.
// The swarm-wide GADFLY_PR_BUDGET_SECS wall-clock backstop (entrypoint.sh) is what
// bounds a mid-pass runaway.
type gadflyBudget struct {
mu sync.Mutex
maxTokens int64
maxSeconds float64
tokens int64
seconds float64
}
// newPRBudget builds the per-PR budget from env, or nil when neither cap is set
// (the default — the swarm-wide ceiling lives in entrypoint.sh; this is the
// per-process belt to its suspenders).
func newPRBudget() *gadflyBudget {
toks := envInt("GADFLY_PR_TOKEN_BUDGET", 0)
secs := envInt("GADFLY_PR_TIME_BUDGET_SECS", 0)
if toks <= 0 && secs <= 0 {
return nil
}
return &gadflyBudget{maxTokens: int64(toks), maxSeconds: float64(secs)}
}
func (b *gadflyBudget) Check(_ context.Context, _ string) error {
if b == nil {
return nil
}
b.mu.Lock()
defer b.mu.Unlock()
if b.maxTokens > 0 && b.tokens >= b.maxTokens {
return fmt.Errorf("gadfly: per-PR token budget exhausted (%d/%d)", b.tokens, b.maxTokens)
}
if b.maxSeconds > 0 && b.seconds >= b.maxSeconds {
return fmt.Errorf("gadfly: per-PR time budget exhausted (%.0f/%.0fs)", b.seconds, b.maxSeconds)
}
return nil
}
func (b *gadflyBudget) Commit(_ context.Context, _ string, runtimeSeconds float64) {
if b == nil {
return
}
b.mu.Lock()
defer b.mu.Unlock()
b.seconds += runtimeSeconds
}
// addUsage records a finished pass's token spend toward the budget. Safe on nil.
func (b *gadflyBudget) addUsage(u llm.Usage) {
if b == nil {
return
}
b.mu.Lock()
defer b.mu.Unlock()
b.tokens += int64(u.InputTokens) + int64(u.OutputTokens)
}
// wrapUpCritic re-expresses gadfly's wrap-up nudge as an executus run.Critic:
// once a run comes within wrapUpReserve steps of its cap, Steer() injects the
// "stop calling tools and write your final answer" message so a thorough model
// spends its last steps finalizing instead of hard-failing empty. It sets no
// hard deadline (Deadline()==zero) and never raises the step ceiling
// (MaxSteps()==0, defer to the run's MaxIterations) — it is purely the nudge.
type wrapUpCritic struct{ reserve int }
func (c *wrapUpCritic) Monitor(_ context.Context, info exrun.RunInfo, _ time.Duration) exrun.CriticHandle {
return &wrapUpHandle{maxSteps: info.MaxIterations, reserve: c.reserve}
}
type wrapUpHandle struct {
mu sync.Mutex
maxSteps int
reserve int
done int // steps completed so far
nudged bool
}
func (h *wrapUpHandle) RecordStep(iter int, _ *llm.Response) {
h.mu.Lock()
h.done = iter + 1
h.mu.Unlock()
}
func (h *wrapUpHandle) RecordToolStart(string, string) {}
func (h *wrapUpHandle) Steer() []llm.Message {
h.mu.Lock()
defer h.mu.Unlock()
at := h.maxSteps - h.reserve
if at < 1 {
at = 1
}
if !h.nudged && h.maxSteps > 0 && h.done >= at {
h.nudged = true
return []llm.Message{llm.UserText(wrapUpInstruction)}
}
return nil
}
func (h *wrapUpHandle) Deadline() time.Time { return time.Time{} }
func (h *wrapUpHandle) MaxSteps() int { return 0 }
func (h *wrapUpHandle) KillCause() error { return nil }
func (h *wrapUpHandle) Stop() {}
// reviewExecutor bundles a run.Executor with the per-run wiring the engine needs
// for each pass (the tool names to expose, the model spec to report as the tier,
// the per-PR caller id, and the budget to feed token usage into).
type reviewExecutor struct {
ex *exrun.Executor
toolNames []string
modelSpec string
callerID string
budget *gadflyBudget
}
// newReviewExecutor builds the run.Executor for the in-process majordomo review
// path. mdl is gadfly's already-resolved review model; summarizer is the cheap
// model the compactor uses (nil disables compaction). Compaction also needs the
// model's context window (resolved once here, not per pass); a 0 window likewise
// disables it. The budget (may be nil) becomes the run.Ports.Budget gate.
func newReviewExecutor(fs *repoFS, mdl, summarizer llm.Model, modelSpec string, budget *gadflyBudget) (*reviewExecutor, error) {
reg, names, err := gadflyToolRegistry(fs)
if err != nil {
return nil, err
}
// gadfly resolves exactly one model per process; the run kernel's resolver
// just hands that model back regardless of the tier string it is asked for.
modelsResolver := func(ctx context.Context, _ string) (context.Context, llm.Model, error) {
return ctx, mdl, nil
}
var compactor compact.CompactorFactory
var ctxTokens func(string) int
if summarizer != nil && compactionEnabled() {
if window := resolveContextTokens(modelSpec); window > 0 {
sumResolver := func(ctx context.Context, _ string) (context.Context, llm.Model, error) {
return ctx, summarizer, nil
}
compactor = compact.NewCompactor(compact.CompactorConfig{
Models: sumResolver,
KeepRecent: envInt("GADFLY_COMPACT_KEEP_RECENT", defaultCompactKeepRecent),
SummaryWordCap: envInt("GADFLY_COMPACT_SUMMARY_WORDS", defaultCompactSummaryWords),
})
ctxTokens = func(string) int { return window } // memoized: one window per process
}
}
var ports exrun.Ports
ports.Critic = &wrapUpCritic{reserve: wrapUpReserve()}
if budget != nil {
ports.Budget = budget
}
cfg := exrun.Config{
Registry: reg,
Models: modelsResolver,
Compactor: compactor,
ContextTokens: ctxTokens,
Defaults: exrun.Defaults{
// MaxIterations/MaxRuntime are intentionally omitted: every pass sets its
// own per-run cap on the RunnableAgent below (the review and recheck caps
// differ), so a Defaults value here would always be overridden — dead. This
// leaves only the cross-pass guards + the compaction ratio.
MaxConsecutiveToolErrors: 4,
MaxSameToolCallRepeats: 4,
CompactionThresholdRatio: compactRatio(),
FallbackTier: modelSpec,
},
Ports: ports,
}
return &reviewExecutor{
ex: exrun.New(cfg),
toolNames: names,
modelSpec: modelSpec,
callerID: prCallerID(),
budget: budget,
}, nil
}
// run executes one agent pass (review or recheck) through the run kernel and
// returns the model's final text. An empty answer with no error is reported as
// an error so the caller (reviewWithSpecialist) renders the advisory "reviewer
// failed to complete" notice rather than a blank section.
func (r *reviewExecutor) run(ctx context.Context, system, task string, maxSteps int) (string, error) {
res := r.ex.Run(ctx, exrun.RunnableAgent{
Name: "gadfly-review",
SystemPrompt: system,
ModelTier: r.modelSpec,
MaxIterations: maxSteps,
MaxRuntime: reviewTimeout(),
LowLevelTools: r.toolNames,
Critic: exrun.CriticConfig{Enabled: true},
}, exectool.Invocation{
RunID: fmt.Sprintf("gadfly-%d", runSeq.Add(1)),
CallerID: r.callerID,
}, task)
// Feed token spend toward the per-PR budget out-of-band (Commit carries only
// seconds; the executor already called it). Safe on a nil budget.
r.budget.addUsage(res.Usage)
if res.Err != nil {
return "", res.Err
}
if out := strings.TrimSpace(res.Output); out != "" {
return out, nil
}
return "", errors.New("agent produced no output")
}
// prCallerID is the budget/audit caller key: the repo + PR, so a budget keyed on
// it is naturally per-PR. Falls back to "local" for an out-of-CI run.
func prCallerID() string {
repo := strings.TrimSpace(os.Getenv("GADFLY_REPO"))
pr := strings.TrimSpace(os.Getenv("GADFLY_PR"))
if repo == "" && pr == "" {
return "local"
}
return repo + "#" + pr
}
// compactionEnabled reports whether context compaction should be wired. On
// unless GADFLY_COMPACT is explicitly falsey.
func compactionEnabled() bool { return envBool("GADFLY_COMPACT", true) }
// compactRatio is the compaction threshold as a fraction of the model context
// window (GADFLY_COMPACT_RATIO), clamped to (0,1]; default defaultCompactRatio.
func compactRatio() float64 {
v := strings.TrimSpace(os.Getenv("GADFLY_COMPACT_RATIO"))
if v == "" {
return defaultCompactRatio
}
f, err := strconv.ParseFloat(v, 64)
if err != nil || f <= 0 || f > 1 {
return defaultCompactRatio
}
return f
}
// resolveContextTokens returns the review model's context window in tokens, used
// to set the compaction threshold. GADFLY_MODEL_CONTEXT_TOKENS overrides it
// (needed for custom/self-hosted endpoints executus can't introspect); otherwise
// it asks executus/model, which knows the static catalog and can fetch an
// Ollama Cloud model's limit via /api/show (one call, at executor-build time).
// Returns 0 — disabling compaction — for an unknown model, mirroring executus's
// "unknown ⇒ don't budget" contract.
func resolveContextTokens(modelSpec string) int {
if v := envInt("GADFLY_MODEL_CONTEXT_TOKENS", 0); v > 0 {
return v
}
key := strings.TrimSpace(os.Getenv("GADFLY_API_KEY"))
if key == "" {
key = strings.TrimSpace(os.Getenv("OLLAMA_API_KEY"))
}
cache := model.NewCloudOllamaLimitCache("", key, nil)
ctx, cancel := context.WithTimeout(context.Background(), contextTokenLookupTimeout)
defer cancel()
if n, ok := model.MaxContextTokensResolving(ctx, modelSpec, cache); ok {
return n
}
// Unknown model or a failed lookup (e.g. no key / unreachable endpoint): don't
// guess — compaction is disabled. Log it so a misconfiguration is debuggable
// rather than silently dropping the protection. Set GADFLY_MODEL_CONTEXT_TOKENS
// to force a window for an endpoint executus can't introspect.
fmt.Fprintf(os.Stderr, "gadfly: no context window resolved for %q; compaction disabled (set GADFLY_MODEL_CONTEXT_TOKENS to enable it)\n", modelSpec)
return 0
}
+140
View File
@@ -0,0 +1,140 @@
package main
import (
"context"
"testing"
llm "gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
func TestGadflyBudget(t *testing.T) {
ctx := context.Background()
// A nil budget never blocks and never panics.
var nilB *gadflyBudget
if err := nilB.Check(ctx, "pr"); err != nil {
t.Errorf("nil budget Check should be nil, got %v", err)
}
nilB.Commit(ctx, "pr", 10)
nilB.addUsage(llm.Usage{InputTokens: 5})
// Token ceiling: passes until usage crosses it.
b := &gadflyBudget{maxTokens: 100}
if err := b.Check(ctx, "pr"); err != nil {
t.Fatalf("fresh budget should pass, got %v", err)
}
b.addUsage(llm.Usage{InputTokens: 60, OutputTokens: 50}) // 110 >= 100
if err := b.Check(ctx, "pr"); err == nil {
t.Error("budget over the token cap should reject the next run")
}
// Seconds ceiling, accumulated via Commit.
s := &gadflyBudget{maxSeconds: 30}
s.Commit(ctx, "pr", 31)
if err := s.Check(ctx, "pr"); err == nil {
t.Error("budget over the time cap should reject the next run")
}
}
func TestNewPRBudget(t *testing.T) {
t.Setenv("GADFLY_PR_TOKEN_BUDGET", "")
t.Setenv("GADFLY_PR_TIME_BUDGET_SECS", "")
if newPRBudget() != nil {
t.Error("no caps set should yield a nil (disabled) budget")
}
t.Setenv("GADFLY_PR_TOKEN_BUDGET", "1000")
if b := newPRBudget(); b == nil || b.maxTokens != 1000 {
t.Errorf("token cap should build a budget with maxTokens=1000, got %+v", b)
}
}
func TestCompactRatio(t *testing.T) {
t.Setenv("GADFLY_COMPACT_RATIO", "")
if got := compactRatio(); got != defaultCompactRatio {
t.Errorf("default ratio = %v, want %v", got, defaultCompactRatio)
}
t.Setenv("GADFLY_COMPACT_RATIO", "0.6")
if got := compactRatio(); got != 0.6 {
t.Errorf("ratio override = %v, want 0.6", got)
}
for _, bad := range []string{"0", "-1", "2", "nope"} {
t.Setenv("GADFLY_COMPACT_RATIO", bad)
if got := compactRatio(); got != defaultCompactRatio {
t.Errorf("invalid ratio %q should fall back to the default, got %v", bad, got)
}
}
}
func TestResolveContextTokensOverride(t *testing.T) {
// The explicit override short-circuits any model introspection (no network).
t.Setenv("GADFLY_MODEL_CONTEXT_TOKENS", "123456")
if got := resolveContextTokens("anything"); got != 123456 {
t.Errorf("explicit context-token override = %d, want 123456", got)
}
}
func TestPRCallerID(t *testing.T) {
t.Setenv("GADFLY_REPO", "")
t.Setenv("GADFLY_PR", "")
if got := prCallerID(); got != "local" {
t.Errorf("no repo/PR should be %q, got %q", "local", got)
}
t.Setenv("GADFLY_REPO", "steve/mort")
t.Setenv("GADFLY_PR", "1367")
if got := prCallerID(); got != "steve/mort#1367" {
t.Errorf("callerID = %q, want steve/mort#1367", got)
}
}
func TestCompactionEnabled(t *testing.T) {
for _, v := range []string{"", "1", "true", "yes"} {
t.Setenv("GADFLY_COMPACT", v)
if !compactionEnabled() {
t.Errorf("GADFLY_COMPACT=%q should be enabled", v)
}
}
for _, v := range []string{"0", "false", "no", "off"} {
t.Setenv("GADFLY_COMPACT", v)
if compactionEnabled() {
t.Errorf("GADFLY_COMPACT=%q should be disabled", v)
}
}
}
func TestGadflyToolRegistry(t *testing.T) {
fs, err := newRepoFS(t.TempDir(), "diff")
if err != nil {
t.Fatal(err)
}
_, names, err := gadflyToolRegistry(fs)
if err != nil {
t.Fatalf("gadflyToolRegistry: %v", err)
}
want := map[string]bool{"read_file": true, "list_dir": true, "grep": true, "find_files": true, "get_diff": true}
has := func(n string) bool {
for _, g := range names {
if g == n {
return true
}
}
return false
}
for n := range want {
if !has(n) {
t.Errorf("registry missing tool %q (got %v)", n, names)
}
}
if has("delegate_investigation") {
t.Error("delegate_investigation must be absent without a worker model")
}
// With a worker model the delegate tool is registered too.
fs.worker = fakeModel(t, "x")
_, names, err = gadflyToolRegistry(fs)
if err != nil {
t.Fatalf("gadflyToolRegistry with worker: %v", err)
}
if !has("delegate_investigation") {
t.Errorf("delegate_investigation should be registered with a worker model, got %v", names)
}
}
+236
View File
@@ -0,0 +1,236 @@
package main
// Structured findings: the machine-readable contract between a lens's output and
// the telemetry/consolidation pipeline. Each lens is asked (see
// scripts/system-prompt.txt) to append a fenced ```gadfly-findings code block
// holding a JSON array of its findings. Parsing that exact block is far more
// reliable than scraping prose with a path:line regex (emit.go's heuristic),
// and it carries PER-FINDING severity + confidence the prose verdict can't.
//
// Everything here degrades gracefully: a missing, unterminated, or malformed
// block makes extractStructuredFindings return ok=false (and yield no findings),
// so the caller falls back to the heuristic scrape — a weak model that ignores
// the contract still contributes findings, exactly as before.
import (
"encoding/json"
"strconv"
"strings"
)
// structuredFinding mirrors one element of the ```gadfly-findings JSON array.
// Line is json.Number so we tolerate both 123 and "123" from less-precise models.
type structuredFinding struct {
File string `json:"file"`
Line json.Number `json:"line"`
Severity string `json:"severity"`
Confidence string `json:"confidence"`
Title string `json:"title"`
Detail string `json:"detail"` // optional; the prose paragraph is used when absent
}
// findingsFence is the info-string that tags the machine-readable block.
const findingsFence = "gadfly-findings"
// extractStructuredFindings parses the ```gadfly-findings JSON block out of a
// lens's markdown. It returns the findings and ok=true when a TERMINATED block is
// present AND parses as a JSON array (an empty array is valid "nothing found" —
// ok is still true). A missing, unterminated, or unparseable block returns
// ok=false so the caller falls back to the heuristic scrape.
//
// Findings are deduped by file:line (keeping the first, matching parseFindings),
// findings with no usable file are dropped, and each title/detail is backfilled
// from the prose when the JSON omits it (best of both: exact location + the human
// context the model already wrote).
func extractStructuredFindings(out string) ([]finding, bool) {
lines := strings.Split(out, "\n")
start, end, ok := findingsSpan(lines)
if !ok {
return nil, false
}
var raw []structuredFinding
if err := json.Unmarshal([]byte(strings.Join(lines[start+1:end], "\n")), &raw); err != nil {
return nil, false
}
findings := make([]finding, 0, len(raw))
seen := map[string]bool{}
var prose map[string]string // built lazily — only if a finding lacks its own detail
for _, sf := range raw {
file := strings.TrimSpace(sf.File)
if file == "" {
continue
}
ln := 0
if n, err := strconv.Atoi(strings.TrimSpace(sf.Line.String())); err == nil && n > 0 {
ln = n
}
key := file + ":" + strconv.Itoa(ln)
if ln > 0 { // only dedupe concrete locations; unknown-line findings are kept
if seen[key] {
continue
}
seen[key] = true
}
detail := strings.TrimSpace(sf.Detail)
if detail == "" && ln > 0 {
if prose == nil {
prose = proseParagraphs(out)
}
detail = prose[key]
}
title := strings.TrimSpace(sf.Title)
if title == "" { // never empty: fall back to the prose detail, then the location
if detail != "" {
title = truncate(detail, 120)
} else if ln > 0 {
title = key
} else {
title = file
}
}
findings = append(findings, finding{
file: file,
line: ln,
title: title,
detail: truncate(detail, 500),
severity: normalizeSeverity(sf.Severity),
confidence: normalizeConfidence(sf.Confidence),
})
if len(findings) >= maxFindingsPerLens {
break
}
}
return findings, true
}
// extractStructuredFindingsOrScrape returns a lens's findings, preferring the
// structured ```gadfly-findings block and falling back to the heuristic prose
// scrape when the block is absent, unterminated/malformed, OR parsed to zero
// usable findings (e.g. an empty [] emitted alongside real prose findings).
// Factored out of emit() so the fallback rule is unit-testable.
func extractStructuredFindingsOrScrape(r specialistResult) []finding {
if fs, _ := extractStructuredFindings(r.out); len(fs) > 0 {
return fs
}
return parseFindings(r.spec, r.out)
}
// stripFindingsBlock removes every TERMINATED ```gadfly-findings block from out so
// the machine-readable JSON never shows in the rendered comment. An UNTERMINATED
// fence is left in place — treating it as a block would swallow the rest of the
// comment (e.g. when a model's output was truncated mid-block). Trailing
// whitespace is trimmed.
func stripFindingsBlock(out string) string {
lines := strings.Split(out, "\n")
for {
start, end, ok := findingsSpan(lines)
if !ok {
break
}
lines = append(lines[:start:start], lines[end+1:]...)
}
return strings.TrimRight(strings.Join(lines, "\n"), "\n")
}
// findingsSpan returns the [start,end] inclusive line indices of the first
// TERMINATED ```gadfly-findings block in lines (start = the opening fence, end =
// the closing fence), or ok=false when there is none or it is unterminated.
// extract and strip share it so they always agree on what is (and isn't) a block.
func findingsSpan(lines []string) (start, end int, ok bool) {
start = -1
for i, ln := range lines {
if isFindingsOpen(ln) {
start = i
break
}
}
if start < 0 {
return 0, 0, false
}
for j := start + 1; j < len(lines); j++ {
if isFenceClose(lines[j]) {
return start, j, true
}
}
return 0, 0, false // unterminated
}
// fenceInfo returns the info-string (text after the backticks) of a code-fence
// line and whether the line opens/closes a fence at all. A bare ``` yields ("",
// true); ```gadfly-findings yields ("gadfly-findings", true).
func fenceInfo(line string) (string, bool) {
t := strings.TrimSpace(line)
if !strings.HasPrefix(t, "```") {
return "", false
}
return strings.TrimSpace(strings.TrimLeft(t, "`")), true
}
// isFindingsOpen reports whether line opens a ```gadfly-findings block, matching
// the info-string EXACTLY (not as a substring) so a fence like ```not-findings
// can't masquerade as ours.
func isFindingsOpen(line string) bool {
info, ok := fenceInfo(line)
return ok && info == findingsFence
}
// isFenceClose reports whether line is a bare closing fence (``` with no info).
func isFenceClose(line string) bool {
info, ok := fenceInfo(line)
return ok && info == ""
}
// proseParagraphs maps "file:line" -> the prose paragraph that first references
// it, so a structured finding without its own detail can borrow the human
// context the model already wrote. Built from the markdown OUTSIDE the findings
// block (the block is JSON, not prose).
func proseParagraphs(out string) map[string]string {
prose := stripFindingsBlock(out)
lines := strings.Split(prose, "\n")
m := map[string]string{}
for _, loc := range pathLineRe.FindAllStringSubmatchIndex(prose, -1) {
key := prose[loc[2]:loc[3]] + ":" + prose[loc[4]:loc[5]]
if _, dup := m[key]; dup {
continue
}
li := strings.Count(prose[:loc[0]], "\n")
m[key] = paragraphAt(lines, li)
}
return m
}
// normalizeSeverity maps a model's severity word onto the canonical set
// (critical/high/medium/small/trivial), accepting common synonyms. An
// unrecognized value is returned lowercased so the store still sees the raw word.
func normalizeSeverity(s string) string {
switch t := strings.ToLower(strings.TrimSpace(s)); t {
case "critical", "crit", "blocker", "blocking":
return "critical"
case "high", "major", "severe":
return "high"
case "medium", "moderate":
return "medium"
case "small", "low", "minor":
return "small"
case "trivial", "nit", "nitpick", "info", "informational", "style", "cosmetic":
return "trivial"
default:
return t
}
}
// normalizeConfidence maps a model's confidence word onto high/medium/low,
// accepting common synonyms; an unrecognized value is returned lowercased.
func normalizeConfidence(s string) string {
switch t := strings.ToLower(strings.TrimSpace(s)); t {
case "high", "certain", "confirmed", "verified":
return "high"
case "medium", "med", "moderate":
return "medium"
case "low", "unsure", "tentative", "unverified", "speculative":
return "low"
default:
return t
}
}
+146
View File
@@ -0,0 +1,146 @@
package main
import (
"strings"
"testing"
)
const sampleReview = "**Blocking issues found**\n\n" +
"- **Unauthenticated endpoint** — `model.go:184` leaks PR content to a third party.\n" +
"- A nit about naming in `util.go:12`.\n\n" +
"```gadfly-findings\n" +
"[\n" +
" {\"file\": \"model.go\", \"line\": 184, \"severity\": \"high\", \"confidence\": \"high\", \"title\": \"Unauthenticated endpoint\"},\n" +
" {\"file\": \"util.go\", \"line\": 12, \"severity\": \"nit\", \"confidence\": \"medium\", \"title\": \"naming\"}\n" +
"]\n" +
"```\n"
func TestExtractStructuredFindings(t *testing.T) {
fs, ok := extractStructuredFindings(sampleReview)
if !ok {
t.Fatal("expected ok=true for a well-formed block")
}
if len(fs) != 2 {
t.Fatalf("want 2 findings, got %d", len(fs))
}
if fs[0].file != "model.go" || fs[0].line != 184 || fs[0].severity != "high" || fs[0].confidence != "high" {
t.Errorf("finding[0] mismatch: %+v", fs[0])
}
// "nit" must normalize to the canonical "trivial".
if fs[1].severity != "trivial" {
t.Errorf("want severity normalized to trivial, got %q", fs[1].severity)
}
// Detail is borrowed from the prose paragraph referencing the same file:line.
if fs[0].detail == "" {
t.Error("expected detail borrowed from prose, got empty")
}
}
func TestExtractStructuredFindingsEmptyArray(t *testing.T) {
out := "No material issues found\n\n```gadfly-findings\n[]\n```\n"
fs, ok := extractStructuredFindings(out)
if !ok {
t.Fatal("an empty array is a valid block; want ok=true")
}
if len(fs) != 0 {
t.Fatalf("want 0 findings, got %d", len(fs))
}
}
func TestExtractStructuredFindingsFallback(t *testing.T) {
// No block at all -> ok=false so the caller uses the heuristic scrape.
if _, ok := extractStructuredFindings("Minor issues\n\n- something at `x.go:1`\n"); ok {
t.Error("want ok=false when there is no block")
}
// Malformed JSON -> ok=false (graceful fallback).
bad := "Minor issues\n\n```gadfly-findings\n{not json}\n```\n"
if _, ok := extractStructuredFindings(bad); ok {
t.Error("want ok=false for malformed JSON")
}
}
func TestExtractStructuredFindingsStringLine(t *testing.T) {
// Tolerate a quoted line number from a less-precise model.
out := "Minor issues\n\n```gadfly-findings\n[{\"file\":\"a.go\",\"line\":\"42\",\"severity\":\"medium\",\"title\":\"x\"}]\n```\n"
fs, ok := extractStructuredFindings(out)
if !ok || len(fs) != 1 || fs[0].line != 42 {
t.Fatalf("want one finding at line 42, got ok=%v %+v", ok, fs)
}
}
func TestStripFindingsBlock(t *testing.T) {
stripped := stripFindingsBlock(sampleReview)
if strings.Contains(stripped, findingsFence) {
t.Errorf("block not stripped: %q", stripped)
}
// The prose findings must survive.
if !strings.Contains(stripped, "Unauthenticated endpoint") || !strings.Contains(stripped, "util.go:12") {
t.Errorf("prose lost during strip: %q", stripped)
}
}
func TestStripFindingsBlockUnterminated(t *testing.T) {
// A truncated, unterminated block must NOT swallow the prose before it.
out := "Minor issues\n\n- real finding at `x.go:1`\n\n```gadfly-findings\n[{\"file\":\"x.go\""
got := stripFindingsBlock(out)
if !strings.Contains(got, "real finding at `x.go:1`") {
t.Errorf("unterminated block swallowed the prose: %q", got)
}
}
func TestStripFindingsBlockNoBlock(t *testing.T) {
in := "Minor issues\n\n- finding at `x.go:9`"
if out := stripFindingsBlock(in); out != in {
t.Errorf("strip changed block-free text: %q != %q", out, in)
}
}
func TestNormalizeSeverity(t *testing.T) {
cases := map[string]string{
"Critical": "critical", "blocker": "critical",
"major": "high", "HIGH": "high",
"moderate": "medium",
"minor": "small", "low": "small", // "minor" and "low" both map to small (consistently)
"nit": "trivial", "Style": "trivial",
"weird": "weird", // unknown passes through, lowercased
}
for in, want := range cases {
if got := normalizeSeverity(in); got != want {
t.Errorf("normalizeSeverity(%q) = %q, want %q", in, got, want)
}
}
}
func TestNormalizeConfidence(t *testing.T) {
cases := map[string]string{
"High": "high", "confirmed": "high",
"MEDIUM": "medium", "moderate": "medium",
"low": "low", "unverified": "low",
"hunch": "hunch", // unknown passes through, lowercased
}
for in, want := range cases {
if got := normalizeConfidence(in); got != want {
t.Errorf("normalizeConfidence(%q) = %q, want %q", in, got, want)
}
}
}
func TestExtractStructuredFindingsFallbackOnEmpty(t *testing.T) {
// A non-clean lens that emitted an empty [] but listed prose findings must
// fall through to the heuristic scrape, not silently drop everything.
out := "Minor issues\n\n- bug at `pkg/a.go:7`\n\n```gadfly-findings\n[]\n```\n"
r := specialistResult{spec: Specialist{Name: "correctness"}, out: out, verdict: verdictMinor}
fs := extractStructuredFindingsOrScrape(r)
if len(fs) == 0 {
t.Fatal("empty [] must fall back to the heuristic scrape, got no findings")
}
if fs[0].file != "pkg/a.go" || fs[0].line != 7 {
t.Errorf("heuristic fallback wrong: %+v", fs[0])
}
}
func TestVerdictSeverity(t *testing.T) {
if verdictBlocking.severity() != "high" || verdictMinor.severity() != "small" || verdictUnknown.severity() != "trivial" {
t.Error("verdict.severity mapping changed unexpectedly")
}
}
+189
View File
@@ -0,0 +1,189 @@
package main
import (
"context"
"sync"
"testing"
"time"
llm "gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/majordomo/provider/fake"
)
// trackingModel wraps a real llm.Model and records the maximum number of
// concurrent Generate calls, holding each call open briefly so overlap is
// observable. The concurrency tracking + sleep happen OUTSIDE the wrapped
// model's call, so (unlike the fake provider, which holds its mutex while
// invoking its default func) this measures genuine caller-side concurrency.
type trackingModel struct {
inner llm.Model
onEnter func()
onExit func()
}
func (m *trackingModel) Generate(ctx context.Context, req llm.Request, opts ...llm.Option) (*llm.Response, error) {
m.onEnter()
defer m.onExit()
time.Sleep(40 * time.Millisecond) // overlap window: concurrent lenses pile up here
return m.inner.Generate(ctx, req, opts...)
}
func (m *trackingModel) Stream(ctx context.Context, req llm.Request, opts ...llm.Option) (llm.Stream, error) {
return m.inner.Stream(ctx, req, opts...)
}
func (m *trackingModel) Capabilities() llm.Capabilities { return m.inner.Capabilities() }
// peakTrackingModel returns a model that replies with a clean verdict (so the
// recheck pass is skipped — one Generate per lens) and a getter for the peak
// number of concurrent Generate calls observed.
func peakTrackingModel(t *testing.T) (llm.Model, func() int) {
t.Helper()
p := fake.New("fake", fake.WithDefault(func(_ string, _ llm.Request) fake.Step {
return fake.Reply("No material issues found.")
}))
inner, err := p.Model("mock")
if err != nil {
t.Fatal(err)
}
var mu sync.Mutex
var inFlight, peak int
enter := func() {
mu.Lock()
defer mu.Unlock()
inFlight++
if inFlight > peak {
peak = inFlight
}
}
exit := func() {
mu.Lock()
defer mu.Unlock()
inFlight--
}
return &trackingModel{inner: inner, onEnter: enter, onExit: exit},
func() int { mu.Lock(); defer mu.Unlock(); return peak }
}
func threeLenses() []Specialist {
return []Specialist{
{Name: "security", Title: "🔒 Security", Focus: "x"},
{Name: "correctness", Title: "🎯 Correctness", Focus: "y"},
{Name: "error-handling", Title: "🧯 Errors", Focus: "z"},
}
}
func assertOrderClean(t *testing.T, specs []Specialist, results []specialistResult) {
t.Helper()
if len(results) != len(specs) {
t.Fatalf("got %d results, want %d", len(results), len(specs))
}
for i, r := range results {
if r.spec.Name != specs[i].Name {
t.Errorf("result %d is %q, want %q (order must follow the specialist list, not finish order)", i, r.spec.Name, specs[i].Name)
}
if r.errored {
t.Errorf("lens %q unexpectedly errored", r.spec.Name)
}
if r.verdict != verdictClean {
t.Errorf("lens %q verdict = %v, want clean", r.spec.Name, r.verdict)
}
}
}
// TestRunSpecialists_FansOut: with GADFLY_LENS_CONCURRENCY=3 all three lenses run
// at once (peak == 3), and results still come back in specialist order.
func TestRunSpecialists_FansOut(t *testing.T) {
t.Setenv("GADFLY_LENS_CONCURRENCY", "3")
t.Setenv("GADFLY_PROVIDER_LENS_CONCURRENCY", "")
mdl, peak := peakTrackingModel(t)
fs, err := newRepoFS(t.TempDir(), "diff --git a/x b/x\n+y\n")
if err != nil {
t.Fatal(err)
}
specs := threeLenses()
results := runSpecialists(testEngine(t, mdl, fs), "sys", specs, "task", "diff")
if got := peak(); got != 3 {
t.Errorf("peak concurrent lenses = %d, want 3", got)
}
assertOrderClean(t, specs, results)
}
// TestRunSpecialists_SequentialByDefault: an unset GADFLY_LENS_CONCURRENCY keeps
// the suite strictly serial (peak == 1) — the historical behavior.
func TestRunSpecialists_SequentialByDefault(t *testing.T) {
t.Setenv("GADFLY_LENS_CONCURRENCY", "")
t.Setenv("GADFLY_PROVIDER_LENS_CONCURRENCY", "")
mdl, peak := peakTrackingModel(t)
fs, err := newRepoFS(t.TempDir(), "diff --git a/x b/x\n+y\n")
if err != nil {
t.Fatal(err)
}
specs := threeLenses()
results := runSpecialists(testEngine(t, mdl, fs), "sys", specs, "task", "diff")
if got := peak(); got != 1 {
t.Errorf("peak concurrent lenses = %d, want 1 (sequential by default)", got)
}
assertOrderClean(t, specs, results)
}
// TestRunSpecialists_PerProviderFanOut: a per-provider override fans this model's
// lenses out even when the scalar default is serial — the model's lane (from
// GADFLY_MODEL's prefix) is matched against GADFLY_PROVIDER_LENS_CONCURRENCY.
func TestRunSpecialists_PerProviderFanOut(t *testing.T) {
t.Setenv("GADFLY_MODEL", "m1/qwen3:14b")
t.Setenv("GADFLY_LENS_CONCURRENCY", "1") // scalar default: serial
t.Setenv("GADFLY_PROVIDER_LENS_CONCURRENCY", "m1=3") // but the m1 lane fans out
mdl, peak := peakTrackingModel(t)
fs, err := newRepoFS(t.TempDir(), "diff --git a/x b/x\n+y\n")
if err != nil {
t.Fatal(err)
}
specs := threeLenses()
results := runSpecialists(testEngine(t, mdl, fs), "sys", specs, "task", "diff")
if got := peak(); got != 3 {
t.Errorf("peak concurrent lenses = %d, want 3 (m1 per-provider override)", got)
}
assertOrderClean(t, specs, results)
}
// TestLensConcurrency covers the resolution matrix: scalar default, scalar
// override, and per-provider override keyed by the model's resolved lane (same
// lane rule entrypoint.sh uses for GADFLY_PROVIDER_CONCURRENCY).
func TestLensConcurrency(t *testing.T) {
tests := []struct {
name string
model string
provider string
scalar string
provMap string
want int
}{
{"default", "", "", "", "", 1},
{"scalar", "", "", "4", "", 4},
{"scalar invalid falls back to default", "", "", "nope", "", 1},
{"bare id uses GADFLY_PROVIDER lane", "qwen3-coder:480b-cloud", "ollama-cloud", "1", "ollama-cloud=3", 3},
{"bare id with no provider defaults to ollama-cloud lane", "qwen3-coder:480b-cloud", "", "1", "ollama-cloud=7", 7},
{"prefixed spec uses its prefix lane", "m1/qwen3:14b", "", "1", "m1=2,ollama-cloud=3", 2},
{"no lane match falls back to scalar", "m5/qwen3.6:35b-mlx", "", "5", "m1=2", 5},
{"override wins over scalar", "ollama-cloud/x", "", "9", "ollama-cloud=2", 2},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Setenv("GADFLY_MODEL", tt.model)
t.Setenv("GADFLY_PROVIDER", tt.provider)
t.Setenv("GADFLY_LENS_CONCURRENCY", tt.scalar)
t.Setenv("GADFLY_PROVIDER_LENS_CONCURRENCY", tt.provMap)
if got := lensConcurrency(); got != tt.want {
t.Errorf("lensConcurrency() = %d, want %d", got, tt.want)
}
})
}
}
+250 -128
View File
@@ -20,23 +20,42 @@
//
// Inputs (env):
//
// OLLAMA_API_KEY Ollama Cloud bearer key (required).
// GADFLY_MODEL model id, e.g. "qwen3-coder:480b-cloud" (required).
// GADFLY_MODEL model id, or a full "provider/model" spec / majordomo
// alias / failover chain (required). A bare id is
// prefixed with GADFLY_PROVIDER.
// GADFLY_PROVIDER provider for bare model ids (default "ollama-cloud";
// e.g. "ollama" for a local daemon, "openai", …).
// GADFLY_BASE_URL override the backend endpoint (OpenAI/Ollama-compatible
// servers, remote Ollama, gateways). See model.go.
// GADFLY_API_KEY provider key; optional — falls back to the provider's
// standard env (OLLAMA_API_KEY / OPENAI_API_KEY /
// ANTHROPIC_API_KEY / GOOGLE_API_KEY|GEMINI_API_KEY).
// GADFLY_SPECIALISTS csv of review lenses, "all", or "auto" (see specialists.go).
// GADFLY_SELECTOR_MODEL model that picks lenses in "auto" mode (default: review model).
// GADFLY_WORKER_MODEL cheap model for the delegate_investigation tool (unset = off).
// GADFLY_REPO_DIR path to the checked-out repo (required; the FS sandbox root).
// GADFLY_DIFF_FILE path to a file holding the full unified diff (required).
// GADFLY_SYSTEM_FILE path to the reviewer system prompt (required).
// GADFLY_TITLE PR title (optional).
// GADFLY_BODY PR description (optional).
// GADFLY_MAX_STEPS review-pass step cap (optional, default 24).
// GADFLY_WRAPUP_RESERVE steps before the cap at which the agent is told to
// stop investigating and write its answer (optional,
// default 4). Plus a tool-free finalization fallback
// guarantees a step-exhausted pass still emits output.
// GADFLY_WRAPUP_RESERVE steps before the cap at which the wrap-up critic nudges
// the agent to stop investigating and write its answer
// (optional, default 4).
// GADFLY_RECHECK set to 0/false to skip the recheck pass (optional, default on).
// GADFLY_RECHECK_MAX_STEPS recheck-pass step cap (optional, default 16).
// GADFLY_TIMEOUT_SECS overall deadline in seconds, shared by both passes (optional, default 300).
// GADFLY_MAX_DIFF_CHARS diff chars embedded in the prompt (optional, default 60000;
// the full diff is always available via the get_diff tool).
// GADFLY_LENS_CONCURRENCY how many specialist lenses run concurrently within this
// model (optional, default 1 = sequential). Total in-flight
// model requests ≈ this × entrypoint.sh's per-provider model
// concurrency, so keep the product within the backend's budget.
// GADFLY_PROVIDER_LENS_CONCURRENCY per-provider override for the above, as a
// "provider=N,provider=N" map keyed by the SAME provider
// lanes as GADFLY_PROVIDER_CONCURRENCY (e.g.
// "ollama-cloud=3,m1=1"). Wins over GADFLY_LENS_CONCURRENCY
// for the model's provider; falls back to it otherwise.
// GADFLY_MAX_DIFF_CHARS diff chars embedded in the review prompt (optional, default 60000;
// the full diff is reachable via the paginated get_diff tool).
//
// On success it prints the review to stdout and exits 0. On a usage/config or
// model error it prints a diagnostic to stderr and exits non-zero; run.sh then
@@ -52,21 +71,29 @@ import (
"strings"
"time"
"gitea.stevedudenhoeffer.com/steve/majordomo/agent"
llm "gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/majordomo/provider/ollama"
"gitea.stevedudenhoeffer.com/steve/executus/fanout"
)
const (
defaultMaxSteps = 24
defaultMaxSteps = 24
// defaultTimeoutSecs is the deadline for EACH specialist's passes (review +
// recheck). It is per-lens, not shared across the suite, so one slow lens
// (e.g. a big local model) can't starve the others. Slow local models may
// need this raised (and a higher job timeout to match the suite total).
defaultTimeoutSecs = 300
defaultMaxDiffChars = 60000
// autoSelectTimeout bounds the dynamic specialist-selection call.
autoSelectTimeout = 120 * time.Second
// defaultWrapUpReserve is how many steps before the cap the agent is told
// to stop investigating and write its final answer. Reserving a margin is
// what keeps a thorough reviewer from spending its whole budget on tool
// calls and then hard-failing with "max steps reached without a final
// answer" — it always has a few steps left to wrap up.
defaultWrapUpReserve = 4
// defaultLensConcurrency is how many specialist lenses run at once within a
// single model. 1 keeps the suite sequential (the historical behavior);
// higher values overlap the independent per-lens passes. See runSpecialists.
defaultLensConcurrency = 1
)
// wrapUpInstruction is steered into a running agent once it comes within the
@@ -77,13 +104,6 @@ const wrapUpInstruction = "⚠️ You are almost out of your investigation budge
"Do not begin any new investigation. If a finding could not be confirmed, drop it or mark it explicitly as unverified. " +
"Output the review in the required format right now."
// finalizeInstruction is the user message sent on the tool-free fallback pass
// when the agent exhausted its budget (or tripped a loop guard) without ever
// producing a final answer. It forces the model to synthesize whatever it has.
const finalizeInstruction = "You have run out of investigation steps. Do NOT call any tools. " +
"Based solely on what you have already gathered above, write your final answer now in the required format. " +
"If you could not confirm some findings, omit them or mark them as unverified, but produce the answer."
func main() {
if err := run(); err != nil {
fmt.Fprintln(os.Stderr, "gadfly:", err)
@@ -92,16 +112,20 @@ func main() {
}
func run() error {
apiKey := os.Getenv("OLLAMA_API_KEY")
if apiKey == "" {
return errors.New("OLLAMA_API_KEY is required")
start := time.Now()
// Consolidation mode: not a review at all — read the per-model findings the
// swarm wrote and print the single cross-model consensus comment. entrypoint.sh
// runs this once, after every model has finished.
if strings.TrimSpace(os.Getenv("GADFLY_CONSOLIDATE_DIR")) != "" {
return runConsolidate()
}
model := os.Getenv("GADFLY_MODEL")
repoDir := os.Getenv("GADFLY_REPO_DIR")
diffFile := os.Getenv("GADFLY_DIFF_FILE")
systemFile := os.Getenv("GADFLY_SYSTEM_FILE")
if model == "" || repoDir == "" || diffFile == "" || systemFile == "" {
return errors.New("GADFLY_MODEL, GADFLY_REPO_DIR, GADFLY_DIFF_FILE and GADFLY_SYSTEM_FILE are all required")
if repoDir == "" || diffFile == "" || systemFile == "" {
return errors.New("GADFLY_REPO_DIR, GADFLY_DIFF_FILE and GADFLY_SYSTEM_FILE are all required")
}
diffBytes, err := os.ReadFile(diffFile)
@@ -123,123 +147,201 @@ func run() error {
return err
}
mdl, err := ollama.Cloud(ollama.WithToken(apiKey)).Model(model)
if err != nil {
return fmt.Errorf("build model %q: %w", model, err)
// Resolve the review engine. The claude-code engine shells out to the
// `claude` CLI (its own repo tools); every other spec is a majordomo model.
// auto-selection and the delegate worker are majordomo-only — with
// claude-code they're skipped (Claude Code does its own legwork).
ccSpec := isClaudeCodeSpec(os.Getenv("GADFLY_MODEL"))
var eng reviewEngine
if ccSpec {
eng = newClaudeCodeEngine(os.Getenv("GADFLY_MODEL"), fsTools.root)
} else {
mdl, merr := resolveModel()
if merr != nil {
return fmt.Errorf("resolve model: %w", merr)
}
// Optional cheap worker for delegate_investigation. Non-fatal: a bad
// worker spec just disables delegation rather than sinking the review.
if worker, werr := resolveWorkerModel(); werr != nil {
fmt.Fprintln(os.Stderr, "gadfly: worker model disabled:", werr)
} else if worker != nil {
fsTools.worker = worker
}
// The context compactor needs a cheap summarizer; reuse the worker model
// when present, else the review model. A bad explicit GADFLY_COMPACT_MODEL
// just disables compaction rather than sinking the review.
summarizer, serr := resolveSummarizerModel(mdl, fsTools.worker)
if serr != nil {
fmt.Fprintln(os.Stderr, "gadfly: compaction summarizer disabled:", serr)
}
rex, rerr := newReviewExecutor(fsTools, mdl, summarizer, os.Getenv("GADFLY_MODEL"), newPRBudget())
if rerr != nil {
return fmt.Errorf("build review executor: %w", rerr)
}
eng = &majordomoEngine{rex: rex, mdl: mdl}
}
timeout := time.Duration(envInt("GADFLY_TIMEOUT_SECS", defaultTimeoutSecs)) * time.Second
ctx, cancel := context.WithTimeout(context.Background(), timeout)
specialists, registry, auto, serrs := resolveSpecialists(repoDir)
for _, e := range serrs {
fmt.Fprintln(os.Stderr, "gadfly:", e)
}
// Dynamic selection: a (cheap) model picks the lenses this diff needs.
// Majordomo-only — the selector is an llm.Model.
if auto {
if ccSpec {
fmt.Fprintln(os.Stderr, "gadfly: auto-select is not supported with the claude-code engine; using the default suite")
specialists = suiteFromRegistry(registry, defaultSuite)
} else {
selector, serr := resolveSelectorModel(eng.(*majordomoEngine).mdl)
if serr != nil {
return fmt.Errorf("resolve selector model: %w", serr)
}
selCtx, cancel := context.WithTimeout(context.Background(), autoSelectTimeout)
picked, aerr := autoSelectSpecialists(selCtx, selector, os.Getenv("GADFLY_TITLE"), os.Getenv("GADFLY_BODY"), diff, registry)
cancel()
if aerr != nil {
fmt.Fprintln(os.Stderr, "gadfly: auto-select failed; falling back to the default suite:", aerr)
specialists = suiteFromRegistry(registry, defaultSuite)
} else {
specialists = picked
fmt.Fprintln(os.Stderr, "gadfly: auto-selected specialists:", specialistNamesOf(specialists))
}
}
}
if len(specialists) == 0 {
return errors.New("no specialists resolved to run")
}
base := string(systemBytes)
task := buildTask(diff)
results := runSpecialists(eng, base, specialists, task, diff)
fmt.Println(renderConsolidated(results))
// Optional, best-effort telemetry. OFF unless GADFLY_FINDINGS_URL is set;
// any failure is logged to stderr and never affects stdout or the exit code.
emit(results, time.Since(start))
// Optional per-model findings artifact for the cross-model consolidation
// pass. No-op unless GADFLY_FINDINGS_OUT is set (entrypoint sets it for a
// multi-model swarm). Best-effort, never affects stdout or the exit code.
writeFindingsOut(results)
return nil
}
// runSpecialists reviews the diff through each lens and returns the results in
// the SAME order as specialists, regardless of finish order. It uses executus's
// fanout primitive: up to GADFLY_LENS_CONCURRENCY lenses run concurrently (the
// default of 1 keeps the suite sequential, exactly as before), and fanout.Run
// returns one result per lens in input order. Each lens already runs under its
// own per-lens timeout (reviewWithSpecialist) and the lenses only read the
// immutable repoFS, so concurrency simply overlaps independent passes.
//
// Caution: this fans out WITHIN one model. It multiplies with entrypoint.sh's
// per-provider model concurrency, so total concurrent backend requests ≈
// (models at once) × (lenses at once). To fan lenses out without oversubscribing
// the backend, run models one at a time (provider lane cap 1) and raise this.
func runSpecialists(eng reviewEngine, base string, specialists []Specialist, task, diff string) []specialistResult {
// Optional live status board: publishes this model's per-lens progress to a
// file the entrypoint board renders. Inert (no-op) unless GADFLY_STATUS_FILE
// is set, so plain runs are unaffected.
sw := newStatusWriter(os.Getenv("GADFLY_MODEL"), modelProvider(), specialists)
fanResults := fanout.Run(context.Background(), specialists, fanout.Options[Specialist]{
MaxConcurrent: lensConcurrency(),
}, func(_ context.Context, sp Specialist) (res specialistResult, _ error) {
// A panic in one lens must not crash the whole binary (which would kill
// every other lens's output) or leave this lens stuck at "running" on the
// status board. fanout does not recover fn panics, so we do it here:
// record the panic as an errored result and mark the lens finished.
defer func() {
if r := recover(); r != nil {
res = specialistResult{spec: sp, out: fmt.Sprintf("⚠️ This reviewer panicked: %v", r), verdict: verdictUnknown, errored: true}
sw.set(sp.Name, lensFinished, "", true)
}
}()
sw.set(sp.Name, lensRunning, "", false)
out, errored := reviewWithSpecialist(eng, base, sp, task, diff)
v := parseVerdict(out)
sw.set(sp.Name, lensFinished, v.label(), errored)
return specialistResult{spec: sp, out: out, verdict: v, errored: errored}, nil
})
// fanout guarantees input order; its Result.Err is set only when the context
// is cancelled before a lens ran (reviewWithSpecialist embeds its own failures
// in the result), so surface that as an errored lens rather than dropping it.
results := make([]specialistResult, len(specialists))
for i, r := range fanResults {
if r.Err != nil {
results[i] = specialistResult{spec: specialists[i], out: fmt.Sprintf("⚠️ This reviewer did not run: %v", r.Err), verdict: verdictUnknown, errored: true}
sw.set(specialists[i].Name, lensFinished, "", true)
continue
}
results[i] = r.Value
}
return results
}
// lensConcurrency resolves how many specialist lenses run at once for THIS run's
// model. It mirrors entrypoint.sh's per-provider MODEL concurrency: a
// per-provider override in GADFLY_PROVIDER_LENS_CONCURRENCY ("provider=N,...")
// wins for the model's provider, otherwise the GADFLY_LENS_CONCURRENCY scalar
// (default 1). The provider is resolved by modelProvider() — the SAME lane rule
// entrypoint uses for GADFLY_PROVIDER_CONCURRENCY — so e.g.
// "ollama-cloud=3,m1=1" fans cloud lenses out while keeping a slow local box
// serial, exactly the way the model map does for whole models.
func lensConcurrency() int {
if n, ok := providerOverride("GADFLY_PROVIDER_LENS_CONCURRENCY", modelProvider()); ok {
return n
}
return envInt("GADFLY_LENS_CONCURRENCY", defaultLensConcurrency)
}
// providerOverride parses a "provider=N,provider=N" env map and returns the
// value for provider when present and valid (>0). Mirrors entrypoint.sh's
// provider_cap lookup so the two concurrency maps share one syntax.
func providerOverride(envName, provider string) (int, bool) {
for _, item := range strings.Split(os.Getenv(envName), ",") {
k, v, ok := strings.Cut(item, "=")
if !ok || strings.TrimSpace(k) != provider {
continue
}
if n, err := strconv.Atoi(strings.TrimSpace(v)); err == nil && n > 0 {
return n, true
}
}
return 0, false
}
// reviewWithSpecialist runs one lens end-to-end under its OWN timeout, so a slow
// model on one lens can't starve the others: a review pass under the
// specialist's composed prompt, then the shared adversarial recheck pass. The
// returned bool is true when the review pass failed (rendered as an inline
// notice — advisory; one lens failing never sinks the others or the job).
func reviewWithSpecialist(eng reviewEngine, base string, sp Specialist, task, diff string) (string, bool) {
ctx, cancel := context.WithTimeout(context.Background(), reviewTimeout())
defer cancel()
// Pass 1 — review: produce the draft.
draft, err := runAgent(ctx, mdl, fsTools, string(systemBytes), buildTask(diff),
draft, err := eng.runPass(ctx, composeSpecialistPrompt(base, sp), task,
envInt("GADFLY_MAX_STEPS", defaultMaxSteps))
if err != nil {
return fmt.Errorf("review pass: %w", err)
fmt.Fprintf(os.Stderr, "gadfly: specialist %q review pass failed: %v\n", sp.Name, err)
return fmt.Sprintf("⚠️ This reviewer failed to complete: %v", err), true
}
// Pass 2 — recheck: adversarially re-verify the draft's findings and drop
// the unconfirmed ones. Skipped for a clean draft (nothing to verify) or
// when disabled. A recheck failure is non-fatal — we emit the unverified
// draft rather than losing the review entirely.
final := draft
if shouldRecheck(draft) {
rechecked, rerr := runAgent(ctx, mdl, fsTools, recheckSystemPrompt, buildRecheckTask(draft, diff),
rechecked, rerr := eng.runPass(ctx, recheckSystemPrompt, buildRecheckTask(draft, diff),
envInt("GADFLY_RECHECK_MAX_STEPS", defaultRecheckMaxSteps))
if rerr != nil {
fmt.Fprintln(os.Stderr, "gadfly: recheck pass failed; emitting unverified draft:", rerr)
fmt.Fprintf(os.Stderr, "gadfly: specialist %q recheck failed; emitting unverified draft: %v\n", sp.Name, rerr)
} else {
final = rechecked
}
}
fmt.Println(final)
return nil
}
// runAgent runs one agent pass (its own fresh toolbox over the sandbox) and
// returns the final answer. An empty answer is an error — the caller decides
// whether that is fatal (review pass) or recoverable (recheck pass). A
// non-empty answer that ended on a budget/guard error is still returned: the
// model wrote its output, then ran out of steps.
//
// Two mechanisms keep a step-hungry model from hard-failing with no output:
// 1. A wrap-up steer: once the run comes within wrapUpReserve steps of the
// cap, a forceful "stop calling tools, write your final answer" message is
// injected so the model spends its remaining steps finalizing.
// 2. A finalization fallback: if the loop still ends empty (the model ignored
// the nudge, or a loop guard tripped), one tool-free model call forces a
// final answer out of the transcript already gathered.
func runAgent(ctx context.Context, mdl llm.Model, fsTools *repoFS, system, task string, maxSteps int) (string, error) {
box, err := fsTools.toolbox()
if err != nil {
return "", err
}
loop := agent.New(mdl, system,
agent.WithToolbox(box),
agent.WithMaxSteps(maxSteps),
// Guard rails: stop the model from spinning on failing or identical
// tool calls instead of writing its answer.
agent.WithToolErrorLimits(4, 4),
)
wrapUpAt := maxSteps - wrapUpReserve()
if wrapUpAt < 1 {
wrapUpAt = 1
}
var completed int // steps finished so far (updated after each step)
nudged := false
res, runErr := loop.Run(ctx, task,
agent.OnStep(func(s agent.Step) { completed = s.Index + 1 }),
agent.WithSteer(func() []llm.Message {
if !nudged && completed >= wrapUpAt {
nudged = true
return []llm.Message{llm.UserText(wrapUpInstruction)}
}
return nil
}),
)
out := ""
if res != nil {
out = strings.TrimSpace(res.Output)
}
if out != "" {
return out, nil
}
// No final answer. If we still have budget on the clock and a transcript to
// work from, force a tool-free finalization rather than losing the pass.
if res != nil && len(res.Messages) > 0 && ctx.Err() == nil {
if forced := forceFinalAnswer(ctx, mdl, system, res.Messages); forced != "" {
return forced, nil
}
}
if runErr != nil {
return "", runErr
}
return "", errors.New("agent produced no output")
}
// forceFinalAnswer makes one tool-free model call to squeeze a final answer out
// of an agent that exhausted its step budget without producing one. Tools are
// forbidden (ToolChoice "none") so the model must synthesize from the transcript
// instead of investigating further. Best-effort: any error or empty reply
// returns "" and the caller falls back to its normal empty-output handling.
func forceFinalAnswer(ctx context.Context, mdl llm.Model, system string, transcript []llm.Message) string {
msgs := append(append([]llm.Message(nil), transcript...), llm.UserText(finalizeInstruction))
resp, err := mdl.Generate(ctx, llm.Request{
System: system,
Messages: msgs,
ToolChoice: "none",
})
if err != nil || resp == nil {
return ""
}
return strings.TrimSpace(resp.Text())
return final, false
}
// wrapUpReserve is how many steps before the cap the wrap-up nudge fires,
@@ -258,7 +360,7 @@ func buildTask(diff string) string {
truncNote := ""
if maxDiff > 0 && len(diff) > maxDiff {
diff = diff[:maxDiff]
truncNote = fmt.Sprintf("\n\n[NOTE: diff truncated to %d chars in this message; call get_diff for the full text.]", maxDiff)
truncNote = fmt.Sprintf("\n\n[NOTE: diff truncated to %d chars in this message; page the full diff with get_diff (paginated; pass a `path` to scope it to one file) or read the changed files.]", maxDiff)
}
var b strings.Builder
@@ -268,7 +370,7 @@ func buildTask(diff string) string {
if strings.TrimSpace(body) != "" {
fmt.Fprintf(&b, "PR description:\n%s\n\n", body)
}
b.WriteString("Review the following unified diff. Before reporting any cross-file or compile-correctness issue, use your tools (read_file, grep, find_files) to verify it against the actual checked-out code — do not rely on the diff alone.\n\n")
b.WriteString("Review the following unified diff. Before reporting any cross-file or compile-correctness issue, use your repository read tools to verify it against the actual checked-out code — do not rely on the diff alone.\n\n")
fmt.Fprintf(&b, "```diff\n%s\n```%s", diff, truncNote)
return b.String()
}
@@ -285,3 +387,23 @@ func envInt(name string, def int) int {
}
return n
}
// envBool reads a boolean-ish env var: def when unset, false for an explicit
// falsey value (0/false/no/off), true otherwise. The shared spelling for
// gadfly's "on unless disabled" opt-out flags (GADFLY_RECHECK, GADFLY_COMPACT).
func envBool(name string, def bool) bool {
switch strings.ToLower(strings.TrimSpace(os.Getenv(name))) {
case "":
return def
case "0", "false", "no", "off":
return false
default:
return true
}
}
// reviewTimeout is the per-specialist-lens deadline (GADFLY_TIMEOUT_SECS), shared
// across a lens's review+recheck passes and applied as each pass's run cap.
func reviewTimeout() time.Duration {
return time.Duration(envInt("GADFLY_TIMEOUT_SECS", defaultTimeoutSecs)) * time.Second
}
+282
View File
@@ -0,0 +1,282 @@
package main
import (
"fmt"
"os"
"strings"
"gitea.stevedudenhoeffer.com/steve/majordomo"
llm "gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/majordomo/provider/anthropic"
"gitea.stevedudenhoeffer.com/steve/majordomo/provider/google"
"gitea.stevedudenhoeffer.com/steve/majordomo/provider/llamaswap"
"gitea.stevedudenhoeffer.com/steve/majordomo/provider/ollama"
"gitea.stevedudenhoeffer.com/steve/majordomo/provider/openai"
)
// defaultProvider is the provider used when GADFLY_MODEL is a bare model id
// (no "provider/" prefix). It keeps existing Ollama Cloud configs — where the
// model list is just ids like "qwen3-coder:480b-cloud" — working unchanged.
const defaultProvider = "ollama-cloud"
// resolveModel builds the review model from the environment. Gadfly is powered
// by majordomo, so it can target any provider majordomo supports — Ollama
// (local or cloud), OpenAI, Anthropic, Google, or any OpenAI/Ollama-compatible
// endpoint — without code changes.
//
// Env:
//
// GADFLY_MODEL model id, or a full "provider/model" spec, or a
// majordomo failover chain / alias (required).
// GADFLY_PROVIDER provider prefix applied when GADFLY_MODEL has no "/"
// (default "ollama-cloud"). e.g. "ollama" for a local daemon.
// GADFLY_BASE_URL override the backend endpoint (OpenAI/Ollama-compatible
// servers, a remote Ollama, an OpenRouter-style gateway…).
// When set, the provider is constructed directly at that URL.
// GADFLY_API_KEY bearer/API key for the chosen provider. Optional; when
// unset the provider falls back to its standard env var
// (OLLAMA_API_KEY / OPENAI_API_KEY / ANTHROPIC_API_KEY /
// GOOGLE_API_KEY|GEMINI_API_KEY). Local Ollama needs none.
//
// With GADFLY_BASE_URL unset, resolution goes through majordomo's registry, so
// LLM_* env DSNs and registered aliases/tiers work too.
func resolveModel() (llm.Model, error) {
// Register any env-defined endpoints/aliases first so they're resolvable as
// "<name>/<model>" specs below. Best-effort: a malformed entry is logged and
// skipped rather than failing the whole review.
for _, err := range registerEnvProviders() {
fmt.Fprintln(os.Stderr, "gadfly: ignoring bad endpoint/alias:", err)
}
model := strings.TrimSpace(os.Getenv("GADFLY_MODEL"))
if model == "" {
return nil, fmt.Errorf("GADFLY_MODEL is required")
}
provider := strings.TrimSpace(os.Getenv("GADFLY_PROVIDER"))
if provider == "" {
provider = defaultProvider
}
baseURL := strings.TrimSpace(os.Getenv("GADFLY_BASE_URL"))
apiKey := os.Getenv("GADFLY_API_KEY")
// No endpoint override: let majordomo's registry resolve the spec. This
// path supports built-in providers (reading their standard key envs),
// LLM_* env DSNs, and aliases/failover chains.
if baseURL == "" {
return majordomo.Parse(buildSpec(provider, model))
}
// Endpoint override: construct the provider directly at the given URL.
switch provider {
case "openai", "openai-compatible":
opts := []openai.Option{openai.WithBaseURL(baseURL)}
if apiKey != "" {
opts = append(opts, openai.WithAPIKey(apiKey))
}
return openai.New(opts...).Model(model)
case "ollama", "ollama-cloud":
opts := []ollama.Option{ollama.WithBaseURL(baseURL)}
if apiKey != "" {
opts = append(opts, ollama.WithToken(apiKey))
}
return ollama.New(opts...).Model(model)
case "llamaswap", "llamaswaps", "llama-swap", "llama-swaps":
// llama-swap (model-swapping proxy). Accept every spelling: hyphenated
// ("llama-swap"/"llama-swaps") mirrors majordomo's DSN schemes (http vs
// https), and the un-hyphenated forms are accepted too. With an explicit
// GADFLY_BASE_URL the scheme is whatever the URL says, so all behave the same.
opts := []llamaswap.Option{llamaswap.WithBaseURL(baseURL)}
if apiKey != "" {
opts = append(opts, llamaswap.WithToken(apiKey))
}
return llamaswap.New(opts...).Model(model)
case "foreman":
// foreman (gitea.stevedudenhoeffer.com/steve/foreman) is a native-Ollama
// queue daemon; the preset also smooths its non-streaming/long-poll
// quirks. Base URL is used verbatim, so a plaintext http:// foreman works.
return ollama.Foreman(baseURL, apiKey).Model(model)
case "anthropic":
opts := []anthropic.Option{anthropic.WithBaseURL(baseURL)}
if apiKey != "" {
opts = append(opts, anthropic.WithAPIKey(apiKey))
}
return anthropic.New(opts...).Model(model)
case "google", "gemini":
opts := []google.Option{google.WithBaseURL(baseURL)}
if apiKey != "" {
opts = append(opts, google.WithAPIKey(apiKey))
}
return google.New(opts...).Model(model)
default:
return nil, fmt.Errorf("GADFLY_BASE_URL is set but GADFLY_PROVIDER %q has no endpoint-override support (use openai/openai-compatible/ollama/llama-swap/foreman/anthropic/google, or unset GADFLY_BASE_URL to resolve via majordomo)", provider)
}
}
// resolveWorkerModel builds the optional worker model for delegate_investigation
// from GADFLY_WORKER_MODEL (a cheap tier is ideal). Returns (nil, nil) when
// unset — delegation is simply off. Honors GADFLY_PROVIDER for a bare id.
func resolveWorkerModel() (llm.Model, error) {
spec := strings.TrimSpace(os.Getenv("GADFLY_WORKER_MODEL"))
if spec == "" {
return nil, nil
}
provider := strings.TrimSpace(os.Getenv("GADFLY_PROVIDER"))
if provider == "" {
provider = defaultProvider
}
return majordomo.Parse(buildSpec(provider, spec))
}
// resolveSummarizerModel picks the model the context compactor uses to compress
// the runaway middle of a transcript. It should be CHEAP, since it fires once per
// compaction: GADFLY_COMPACT_MODEL if set (honoring GADFLY_PROVIDER for a bare
// id), else the delegate worker model when one is configured (already cheap by
// design), else the review model itself. Returns (nil, err) only on an explicit
// bad GADFLY_COMPACT_MODEL spec — the caller logs it and simply runs without
// compaction rather than failing the review.
func resolveSummarizerModel(review, worker llm.Model) (llm.Model, error) {
if spec := strings.TrimSpace(os.Getenv("GADFLY_COMPACT_MODEL")); spec != "" {
provider := strings.TrimSpace(os.Getenv("GADFLY_PROVIDER"))
if provider == "" {
provider = defaultProvider
}
return majordomo.Parse(buildSpec(provider, spec))
}
if worker != nil {
return worker, nil
}
return review, nil
}
// buildSpec turns (provider, model) into a majordomo spec. A model id that
// already carries a "provider/" prefix (or is a multi-element failover chain)
// is passed through verbatim; a bare id is prefixed with the provider.
func buildSpec(provider, model string) string {
if strings.Contains(model, "/") || strings.Contains(model, ",") {
return model
}
return provider + "/" + model
}
// modelProvider returns the provider "lane" for THIS run's model, mirroring
// entrypoint.sh's provider_of: the segment before the first "/" in GADFLY_MODEL,
// else GADFLY_PROVIDER, else the default (ollama-cloud). The binary reviews one
// model per invocation, so this is that model's provider — used to resolve
// per-provider policy (e.g. lens concurrency) against the SAME provider keys
// entrypoint uses for GADFLY_PROVIDER_CONCURRENCY.
func modelProvider() string {
model := strings.TrimSpace(os.Getenv("GADFLY_MODEL"))
if pfx, _, ok := strings.Cut(model, "/"); ok {
return strings.TrimSpace(pfx)
}
if p := strings.TrimSpace(os.Getenv("GADFLY_PROVIDER")); p != "" {
return p
}
return defaultProvider
}
// registerEnvProviders reads named endpoints and aliases from the environment
// and registers them with majordomo's default registry, so they can be used as
// "<name>/<model>" specs (or bare aliases) in GADFLY_MODEL.
//
// Two env families (NAME is lowercased to form the registry name, like
// majordomo's own LLM_* convention — GADFLY_ENDPOINT_BIGBOX → "bigbox"):
//
// GADFLY_ENDPOINT_<NAME> = "<provider>|<base-url>[|<key>]"
// Registers a provider at an explicit endpoint. Unlike majordomo's LLM_*
// DSNs (which are HTTPS-only), the base URL is used verbatim, so a
// plaintext local Ollama (or foreman queue) works:
// GADFLY_ENDPOINT_BIGBOX="ollama|http://192.168.1.50:11434"
// GADFLY_MODEL=bigbox/qwen2.5-coder:7b
// provider is one of ollama/llama-swap(s)/foreman/openai/anthropic/google; "foreman"
// targets a foreman daemon (native Ollama on the wire):
// GADFLY_ENDPOINT_M1="foreman|http://foreman-m1:8080|tok"
//
// GADFLY_ALIAS_<NAME> = "<majordomo spec>"
// Registers a plain alias that expands inline (a model, or a failover
// chain): GADFLY_ALIAS_FAST="bigbox/qwen2.5-coder:7b,ollama-cloud/gpt-oss:120b-cloud".
//
// Returns one error per malformed entry; valid entries still register.
func registerEnvProviders() []error {
var errs []error
for _, kv := range os.Environ() {
key, val, _ := strings.Cut(kv, "=")
switch {
case strings.HasPrefix(key, "GADFLY_ENDPOINT_") && len(key) > len("GADFLY_ENDPOINT_"):
name := strings.ToLower(strings.TrimPrefix(key, "GADFLY_ENDPOINT_"))
p, err := endpointProvider(name, val)
if err != nil {
errs = append(errs, fmt.Errorf("%s: %w", key, err))
continue
}
majordomo.RegisterProvider(p)
case strings.HasPrefix(key, "GADFLY_ALIAS_") && len(key) > len("GADFLY_ALIAS_"):
name := strings.ToLower(strings.TrimPrefix(key, "GADFLY_ALIAS_"))
spec := strings.TrimSpace(val)
if spec == "" {
errs = append(errs, fmt.Errorf("%s: empty alias spec", key))
continue
}
majordomo.RegisterAlias(name, spec)
}
}
return errs
}
// endpointProvider builds a named provider from a "provider|base-url[|key]"
// value. The base URL is honored verbatim (http or https).
func endpointProvider(name, raw string) (llm.Provider, error) {
parts := strings.SplitN(raw, "|", 3)
if len(parts) < 2 {
return nil, fmt.Errorf("want \"<provider>|<base-url>[|<key>]\", got %q", raw)
}
provider := strings.TrimSpace(parts[0])
baseURL := strings.TrimSpace(parts[1])
key := ""
if len(parts) == 3 {
key = strings.TrimSpace(parts[2])
}
if baseURL == "" {
return nil, fmt.Errorf("missing base URL in %q", raw)
}
switch provider {
case "ollama", "ollama-cloud":
opts := []ollama.Option{ollama.WithName(name), ollama.WithBaseURL(baseURL)}
if key != "" {
opts = append(opts, ollama.WithToken(key))
}
return ollama.New(opts...), nil
case "llamaswap", "llamaswaps", "llama-swap", "llama-swaps":
opts := []llamaswap.Option{llamaswap.WithName(name), llamaswap.WithBaseURL(baseURL)}
if key != "" {
opts = append(opts, llamaswap.WithToken(key))
}
return llamaswap.New(opts...), nil
case "foreman":
// foreman is native-Ollama on the wire; the preset additionally handles
// its non-streaming degradation. Unlike the HTTPS-only LLM_* foreman://
// DSN, the base URL here is verbatim, so a plaintext http:// foreman works.
return ollama.Foreman(baseURL, key, ollama.WithName(name)), nil
case "openai", "openai-compatible":
opts := []openai.Option{openai.WithName(name), openai.WithBaseURL(baseURL)}
if key != "" {
opts = append(opts, openai.WithAPIKey(key))
}
return openai.New(opts...), nil
case "anthropic":
opts := []anthropic.Option{anthropic.WithName(name), anthropic.WithBaseURL(baseURL)}
if key != "" {
opts = append(opts, anthropic.WithAPIKey(key))
}
return anthropic.New(opts...), nil
case "google", "gemini":
opts := []google.Option{google.WithName(name), google.WithBaseURL(baseURL)}
if key != "" {
opts = append(opts, google.WithAPIKey(key))
}
return google.New(opts...), nil
default:
return nil, fmt.Errorf("unknown provider %q (use ollama/llama-swap(s)/foreman/openai/openai-compatible/anthropic/google)", provider)
}
}
+91
View File
@@ -0,0 +1,91 @@
package main
import "testing"
func TestEndpointProvider(t *testing.T) {
t.Run("ollama http endpoint registers under its name", func(t *testing.T) {
p, err := endpointProvider("bigbox", "ollama|http://192.168.1.50:11434")
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if p.Name() != "bigbox" {
t.Errorf("Name() = %q, want %q", p.Name(), "bigbox")
}
})
t.Run("openai compatible with key", func(t *testing.T) {
if _, err := endpointProvider("gpu", "openai|http://gpu.lan:8000/v1|sk-x"); err != nil {
t.Fatalf("unexpected error: %v", err)
}
})
t.Run("foreman queue registers under its name", func(t *testing.T) {
p, err := endpointProvider("m1", "foreman|http://foreman-m1:8080|tok")
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
// WithName(name) must win over the Foreman preset's default "foreman".
if p.Name() != "m1" {
t.Errorf("Name() = %q, want %q", p.Name(), "m1")
}
})
t.Run("foreman without token", func(t *testing.T) {
if _, err := endpointProvider("m5", "foreman|http://foreman-m5:8080"); err != nil {
t.Fatalf("unexpected error: %v", err)
}
})
t.Run("llamaswap registers under its name", func(t *testing.T) {
p, err := endpointProvider("ls", "llamaswap|http://swap.lan:8080|tok")
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if p.Name() != "ls" {
t.Errorf("Name() = %q, want %q", p.Name(), "ls")
}
})
t.Run("llamaswap without token", func(t *testing.T) {
if _, err := endpointProvider("ls2", "llamaswap|http://swap.lan:8080"); err != nil {
t.Fatalf("unexpected error: %v", err)
}
})
// All llama-swap spellings (hyphenated/TLS variants mirror majordomo's DSN
// schemes) must resolve to the llamaswap provider.
for _, name := range []string{"llama-swap", "llama-swaps", "llamaswaps"} {
t.Run(name+" alias", func(t *testing.T) {
p, err := endpointProvider("ls", name+"|https://swap.lan:8080|tok")
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if p.Name() != "ls" {
t.Errorf("Name() = %q, want %q", p.Name(), "ls")
}
})
}
for _, bad := range []string{"", "ollama", "noprovider-no-pipe", "mystery|http://x"} {
t.Run("rejects "+bad, func(t *testing.T) {
if _, err := endpointProvider("n", bad); err == nil {
t.Errorf("endpointProvider(%q) = nil error, want error", bad)
}
})
}
}
func TestBuildSpec(t *testing.T) {
tests := []struct {
name string
provider string
model string
want string
}{
{"bare id gets provider prefix", "ollama-cloud", "qwen3-coder:480b-cloud", "ollama-cloud/qwen3-coder:480b-cloud"},
{"bare id local ollama", "ollama", "llama3.1", "ollama/llama3.1"},
{"already has provider passes through", "ollama-cloud", "openai/gpt-4o", "openai/gpt-4o"},
{"slashed model name passes through verbatim", "openai", "openai/meta-llama/Llama-3.1", "openai/meta-llama/Llama-3.1"},
{"failover chain passes through", "ollama-cloud", "anthropic/opus-4.8,ollama-cloud/qwen3-coder:480b-cloud", "anthropic/opus-4.8,ollama-cloud/qwen3-coder:480b-cloud"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := buildSpec(tt.provider, tt.model); got != tt.want {
t.Errorf("buildSpec(%q, %q) = %q, want %q", tt.provider, tt.model, got, tt.want)
}
})
}
}
+22 -19
View File
@@ -2,7 +2,6 @@ package main
import (
"fmt"
"os"
"strings"
)
@@ -11,20 +10,26 @@ import (
// than discovering them.
const defaultRecheckMaxSteps = 16
// defaultRecheckDiffChars caps the diff embedded in the recheck task. It is much
// smaller than the review task's GADFLY_MAX_DIFF_CHARS: the recheck already has
// the draft findings to verify and can pull the exact hunks it needs via the
// paginated get_diff tool (optionally scoped to a path), so re-embedding the
// whole diff on every recheck step is pure burn. Override: GADFLY_RECHECK_DIFF_CHARS.
const defaultRecheckDiffChars = 20000
// recheckSystemPrompt drives the second, adversarial verification pass. The
// model is given a DRAFT review and must independently confirm each finding
// against the real code before letting it survive — the antidote to a
// single-pass reviewer that reads a couple of files, mis-connects them, and
// posts a confident but wrong "blocking" verdict.
const recheckSystemPrompt = `You are a VERIFICATION GATE for an automated adversarial code review of the
"mort" project (a large Go Discord bot). You are given a DRAFT review produced
by another model. Your job is NOT to write a new review — it is to confirm or
reject each finding in the draft against the ACTUAL code, then output the
corrected review.
const recheckSystemPrompt = `You are a VERIFICATION GATE for an automated adversarial code review. You are
given a DRAFT review produced by another model. Your job is NOT to write a new
review — it is to confirm or reject each finding in the draft against the ACTUAL
code, then output the corrected review.
You have the same read-only repository tools as the original reviewer:
- read_file(path[, start_line, limit]), list_dir([path]), grep(pattern[, path,
max_results]), find_files(name[, max_results]), get_diff().
You have read-only access to the checked-out repository — use your tools to read
files and search the code to independently verify each finding against the real
source.
For EVERY finding in the draft:
1. Independently reproduce the reasoning by reading the actual files with your
@@ -50,18 +55,16 @@ Output rules:
- Do NOT invent new findings; this is a verification gate, not a fresh review.
- Do NOT include meta-commentary about the verification process or which
findings you dropped — output only the final, corrected review markdown.
- The draft ends with a fenced ` + "`gadfly-findings`" + ` JSON block. Regenerate it
so it lists ONLY the findings that SURVIVED your verification, in the same schema
({"file","line","severity","confidence","title"}; severity one of
critical/high/medium/small/trivial, confidence one of high/medium/low). If every
finding was dropped, emit an empty array ` + "`[]`" + `. Keep the block last.
- When done investigating, STOP calling tools and reply with the review.`
// recheckEnabled reports whether the verification pass should run. On unless
// GADFLY_RECHECK is explicitly a falsey value.
func recheckEnabled() bool {
switch strings.ToLower(strings.TrimSpace(os.Getenv("GADFLY_RECHECK"))) {
case "0", "false", "no", "off":
return false
default:
return true
}
}
func recheckEnabled() bool { return envBool("GADFLY_RECHECK", true) }
// shouldRecheck decides whether to run the verification pass for a given draft.
// A clean "no material issues" draft has nothing to verify, so it is skipped
@@ -80,11 +83,11 @@ func shouldRecheck(draft string) bool {
// scrutinize, with the full diff available via get_diff (and embedded here,
// truncated, to save a tool call).
func buildRecheckTask(draft, diff string) string {
maxDiff := envInt("GADFLY_MAX_DIFF_CHARS", defaultMaxDiffChars)
maxDiff := envInt("GADFLY_RECHECK_DIFF_CHARS", defaultRecheckDiffChars)
truncNote := ""
if maxDiff > 0 && len(diff) > maxDiff {
diff = diff[:maxDiff]
truncNote = fmt.Sprintf("\n\n[NOTE: diff truncated to %d chars here; call get_diff for the full text.]", maxDiff)
truncNote = fmt.Sprintf("\n\n[NOTE: diff truncated to %d chars here; call get_diff (paginated; pass a `path` to scope it to one file) or read the changed files for the rest.]", maxDiff)
}
var b strings.Builder
+29 -8
View File
@@ -49,7 +49,7 @@ func TestRecheckEnabled(t *testing.T) {
}
func TestBuildRecheckTask(t *testing.T) {
t.Setenv("GADFLY_MAX_DIFF_CHARS", "")
t.Setenv("GADFLY_RECHECK_DIFF_CHARS", "")
draft := "VERDICT: Blocking issues found\n- foo.go:1 broken"
out := buildRecheckTask(draft, "diff --git a/x b/x\n+y\n")
if !strings.Contains(out, draft) {
@@ -77,25 +77,46 @@ func fakeModel(t *testing.T, reply string) llm.Model {
return m
}
func TestRunAgent_ReturnsOutput(t *testing.T) {
// newTestReviewExecutor builds a reviewExecutor over a fake model + repo for unit
// tests: no compaction summarizer and no budget, so it exercises the bare agent
// loop through the executus run kernel without any network call.
func newTestReviewExecutor(t *testing.T, mdl llm.Model, fs *repoFS) *reviewExecutor {
t.Helper()
rex, err := newReviewExecutor(fs, mdl, nil, "mock", nil)
if err != nil {
t.Fatal(err)
}
return rex
}
// testEngine wraps newTestReviewExecutor in a majordomoEngine for the
// runSpecialists-level tests.
func testEngine(t *testing.T, mdl llm.Model, fs *repoFS) *majordomoEngine {
t.Helper()
return &majordomoEngine{rex: newTestReviewExecutor(t, mdl, fs), mdl: mdl}
}
func TestReviewExecutor_ReturnsOutput(t *testing.T) {
fs, err := newRepoFS(t.TempDir(), "diff")
if err != nil {
t.Fatal(err)
}
mdl := fakeModel(t, " corrected review: No material issues found. ")
out, err := runAgent(context.Background(), mdl, fs, "sys", "task", 4)
rex := newTestReviewExecutor(t, mdl, fs)
out, err := rex.run(context.Background(), "sys", "task", 4)
if err != nil {
t.Fatalf("runAgent: %v", err)
t.Fatalf("run: %v", err)
}
if out != "corrected review: No material issues found." {
t.Errorf("runAgent should return trimmed model output, got %q", out)
t.Errorf("run should return trimmed model output, got %q", out)
}
}
func TestRunAgent_EmptyIsError(t *testing.T) {
func TestReviewExecutor_EmptyIsError(t *testing.T) {
fs, _ := newRepoFS(t.TempDir(), "diff")
mdl := fakeModel(t, " ")
if _, err := runAgent(context.Background(), mdl, fs, "sys", "task", 4); err == nil {
t.Error("runAgent should error on empty model output")
rex := newTestReviewExecutor(t, mdl, fs)
if _, err := rex.run(context.Background(), "sys", "task", 4); err == nil {
t.Error("run should error on empty model output")
}
}
+296
View File
@@ -0,0 +1,296 @@
package main
// Inline PR review. After the consensus comment is rendered, Gadfly also posts a
// single Gitea pull review (state COMMENT — advisory, NEVER request-changes or
// approve) whose inline comments anchor consensus findings to the exact changed
// lines. The issue comment stays the ranked overview; the review puts each
// finding next to the code it's about — the "reviewer integrated with Gitea" the
// project wanted, without ever blocking a merge.
//
// All of this is best-effort: disabled by GADFLY_INLINE_REVIEW=0 or when the diff
// / API creds aren't available, only anchors findings that land on a line present
// in the diff (Gitea rejects comments off the diff), and any error is logged to
// stderr without touching the consensus comment (already on stdout) or the exit
// code.
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"regexp"
"strconv"
"strings"
"time"
)
const (
// inlineReviewMarker tags our review body so a re-run can delete the previous
// one instead of stacking duplicate inline comments.
inlineReviewMarker = "<!-- gadfly-inline-review -->"
// maxInlineComments caps how many inline comments one review carries, so a
// huge diff can't produce a wall of annotations. Clusters are pre-sorted by
// agreement×severity, so the cap keeps the most important.
maxInlineComments = 25
inlineReviewHTTPTimeout = 20 * time.Second
)
// reviewComment is one inline comment in a Gitea pull review. Field names match
// the Gitea API EXACTLY (new_position = line in the new/head file).
type reviewComment struct {
Path string `json:"path"`
Body string `json:"body"`
NewPosition int `json:"new_position"`
}
// createReview is the POST /pulls/{n}/reviews body. event is ALWAYS "COMMENT".
type createReview struct {
Body string `json:"body"`
Event string `json:"event"`
Comments []reviewComment `json:"comments"`
}
// postInlineReview posts one COMMENT-state pull review with inline comments for
// consensus findings on changed lines. No-op + best-effort (see file comment).
func postInlineReview(clusters []cluster) {
if strings.EqualFold(strings.TrimSpace(os.Getenv("GADFLY_INLINE_REVIEW")), "0") {
return
}
api := strings.TrimRight(strings.TrimSpace(os.Getenv("GITEA_API")), "/")
token := strings.TrimSpace(os.Getenv("GITEA_TOKEN"))
pr := strings.TrimSpace(os.Getenv("GADFLY_PR"))
diffPath := strings.TrimSpace(os.Getenv("GADFLY_DIFF_FILE"))
if api == "" || token == "" || pr == "" || diffPath == "" {
return
}
diff, err := os.ReadFile(diffPath)
if err != nil {
fmt.Fprintln(os.Stderr, "gadfly: inline review: read diff:", err)
return
}
comments := inlineComments(clusters, parseDiffNewLines(string(diff)))
if len(comments) == 0 {
return // nothing anchors to a changed line; the consensus comment covers it
}
client := &http.Client{Timeout: inlineReviewHTTPTimeout}
base := fmt.Sprintf("%s/pulls/%s/reviews", api, pr)
deletePriorReviews(client, base, token) // avoid stacking on re-runs
body := fmt.Sprintf("%s\n🪰 **Gadfly consensus review** — %d inline finding%s on changed lines. See the consensus comment for the full ranked summary.\n\n<sub>Advisory only — does not block merge.</sub>",
inlineReviewMarker, len(comments), plural(len(comments)))
if err := giteaSend(client, http.MethodPost, base, token, createReview{Body: body, Event: "COMMENT", Comments: comments}); err != nil {
fmt.Fprintln(os.Stderr, "gadfly: inline review post:", err)
}
}
// inlineComments builds inline comments for the clusters that anchor to a line
// present in the diff, in priority order (clusters are pre-sorted), capped.
func inlineComments(clusters []cluster, addable map[string]map[int]bool) []reviewComment {
var out []reviewComment
for _, c := range clusters {
path := normPath(c.file)
anchor := anchorLine(addable[path], c.line, c.maxLine)
if anchor == 0 {
continue
}
out = append(out, reviewComment{Path: path, NewPosition: anchor, Body: inlineBody(c)})
if len(out) >= maxInlineComments {
break
}
}
return out
}
// anchorLine returns the first line in [lo,hi] that is an added line in the diff,
// or 0 if none. Scanning the cluster's whole span (not just its representative
// line) anchors a finding whose min line is just outside the diff but whose span
// still overlaps a changed line.
func anchorLine(added map[int]bool, lo, hi int) int {
if added == nil || lo <= 0 {
return 0
}
if hi < lo { // single-line cluster (maxLine unset)
hi = lo
}
for ln := lo; ln <= hi; ln++ {
if added[ln] {
return ln
}
}
return 0
}
// inlineBody renders one inline comment: severity + title, who flagged it, detail.
func inlineBody(c cluster) string {
var b strings.Builder
fmt.Fprintf(&b, "%s **%s**", sevIcon(c.severity), strings.TrimSpace(c.title))
fmt.Fprintf(&b, "\n\n_%s · flagged by %d model%s_", lensList(c.lenses), len(c.models), plural(len(c.models)))
if d := strings.TrimSpace(c.detail); d != "" {
fmt.Fprintf(&b, "\n\n%s", d)
}
b.WriteString("\n\n<sub>🪰 Gadfly · advisory</sub>")
return b.String()
}
// parseDiffNewLines returns, per file, the set of NEW-file line numbers that were
// ADDED in the unified diff — the safest lines for an inline comment to anchor to
// (Gitea reliably accepts comments on added lines). Context lines are walked to
// keep the line counter correct but are NOT recorded: anchoring only to added
// lines avoids the all-or-nothing review POST being rejected for an off-change
// anchor. Hunk lengths from the @@ header bound each hunk, so a content line that
// happens to start with "+++ " or "@@" is still read as content, not a header.
func parseDiffNewLines(diff string) map[string]map[int]bool {
out := map[string]map[int]bool{}
var file string
var newLine, oldRem, newRem int
inHunk := false
for _, line := range strings.Split(diff, "\n") {
if inHunk && (newRem > 0 || oldRem > 0) {
switch {
case strings.HasPrefix(line, "+"):
record(out, file, newLine) // added line — anchorable
newLine++
newRem--
case strings.HasPrefix(line, "-"):
oldRem--
case strings.HasPrefix(line, "\\"): // "\ No newline at end of file"
default: // context line (leading space, or an empty line): advance, don't record
newLine++
newRem--
oldRem--
}
if newRem <= 0 && oldRem <= 0 {
inHunk = false
}
continue
}
switch {
case strings.HasPrefix(line, "+++ "):
file = normPath(strings.TrimPrefix(line, "+++ "))
if file == "/dev/null" {
file = ""
}
case strings.HasPrefix(line, "@@"):
if m := hunkRe.FindStringSubmatch(line); m != nil && file != "" {
newLine, _ = strconv.Atoi(m[3])
oldRem = atoiOr(m[2], 1)
newRem = atoiOr(m[4], 1)
inHunk = newRem > 0 || oldRem > 0
}
}
}
return out
}
// hunkRe captures a unified-diff hunk header's old/new start+length:
// @@ -<oldStart>[,<oldLen>] +<newStart>[,<newLen>] @@
var hunkRe = regexp.MustCompile(`^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@`)
// normPath trims a unified-diff path down to a repo-relative one: strip a single
// leading "a/" or "b/" prefix (and any "./"), and surrounding whitespace. Applied
// to BOTH the diff paths and a finding's file so they match even when a model
// writes "./pkg/x.go" or the diff carries the "b/" prefix.
func normPath(p string) string {
p = strings.TrimSpace(p)
p = strings.TrimPrefix(p, "./")
if strings.HasPrefix(p, "a/") || strings.HasPrefix(p, "b/") {
p = p[2:]
}
return p
}
func record(out map[string]map[int]bool, file string, line int) {
if file == "" {
return
}
if out[file] == nil {
out[file] = map[int]bool{}
}
out[file][line] = true
}
// atoiOr parses s, returning def when s is empty or unparseable. Used for the
// optional hunk-length fields (absent => length 1).
func atoiOr(s string, def int) int {
if s == "" {
return def
}
if n, err := strconv.Atoi(s); err == nil {
return n
}
return def
}
// deletePriorReviews removes our previous inline reviews (matched by the body
// marker) so a re-run replaces rather than stacks. Best-effort and quiet.
func deletePriorReviews(client *http.Client, base, token string) {
const perPage = 50
for page := 1; page <= 10; page++ { // bound the scan, but page past 50 so a stale marked review isn't missed
req, err := http.NewRequest(http.MethodGet, fmt.Sprintf("%s?limit=%d&page=%d", base, perPage, page), nil)
if err != nil {
return
}
req.Header.Set("Authorization", "token "+token)
resp, err := client.Do(req)
if err != nil {
return
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
io.Copy(io.Discard, resp.Body)
resp.Body.Close()
return
}
var reviews []struct {
ID int `json:"id"`
Body string `json:"body"`
}
_ = json.NewDecoder(resp.Body).Decode(&reviews)
resp.Body.Close()
for _, r := range reviews {
if !strings.Contains(r.Body, inlineReviewMarker) {
continue
}
dreq, err := http.NewRequest(http.MethodDelete, fmt.Sprintf("%s/%d", base, r.ID), nil)
if err != nil {
continue
}
dreq.Header.Set("Authorization", "token "+token)
if dresp, err := client.Do(dreq); err == nil {
io.Copy(io.Discard, dresp.Body)
dresp.Body.Close()
}
}
if len(reviews) < perPage {
return // last page
}
}
}
// giteaSend marshals payload and sends it with Gitea's "token" auth scheme,
// treating a non-2xx response as an error (with a snippet of the body).
func giteaSend(client *http.Client, method, url, token string, payload any) error {
body, err := json.Marshal(payload)
if err != nil {
return err
}
req, err := http.NewRequest(method, url, bytes.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "token "+token)
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
snippet, _ := io.ReadAll(io.LimitReader(resp.Body, 2048))
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("%s %s: status %d: %s", method, url, resp.StatusCode, strings.TrimSpace(string(snippet)))
}
return nil
}
+107
View File
@@ -0,0 +1,107 @@
package main
import (
"strings"
"testing"
)
func TestParseDiffNewLines(t *testing.T) {
diff := "diff --git a/a.go b/a.go\n" +
"index 111..222 100644\n" +
"--- a/a.go\n" +
"+++ b/a.go\n" +
"@@ -1,3 +1,4 @@\n" +
" line1\n" +
"-old2\n" +
"+new2\n" +
"+new3\n" +
" line4\n"
got := parseDiffNewLines(diff)
// Only ADDED lines anchor: new2 (line 2) and new3 (line 3). Context lines 1
// and 4 are walked for counting but not recorded.
want := map[int]bool{2: true, 3: true}
if len(got["a.go"]) != len(want) {
t.Fatalf("a.go lines = %v, want %v (added-only)", got["a.go"], want)
}
for ln := range want {
if !got["a.go"][ln] {
t.Errorf("expected a.go line %d anchorable", ln)
}
}
if got["a.go"][1] || got["a.go"][4] {
t.Error("context lines 1/4 should not be anchorable (added-only)")
}
}
func TestParseDiffNewLinesContentLooksLikeHeader(t *testing.T) {
// An added line whose CONTENT is "++ weird" appears as "+++ weird" in the
// diff. Hunk-length tracking must read it as content, not a file header.
diff := "--- a/x\n+++ b/x\n@@ -1,1 +1,2 @@\n ctx\n+++ weird\n"
got := parseDiffNewLines(diff)
if got["x"][1] { // context line, not recorded (added-only)
t.Errorf("line 1 is context, should not anchor: %v", got["x"])
}
if !got["x"][2] { // the "+++ weird" added line
t.Errorf("want added line 2 anchorable, got %v", got["x"])
}
}
func TestAnchorLineScansSpan(t *testing.T) {
// A cluster spanning 10..14 whose min line (10) isn't in the diff but whose
// span includes added line 14 must anchor to 14, not be dropped.
added := map[string]map[int]bool{"a.go": {14: true}}
clusters := []cluster{{file: "a.go", line: 10, maxLine: 14, severity: "high", title: "t", models: set("m1"), lenses: set("x")}}
cs := inlineComments(clusters, added)
if len(cs) != 1 || cs[0].NewPosition != 14 {
t.Fatalf("want anchor at 14 via span scan, got %+v", cs)
}
}
func TestParseDiffNewLinesMultiFile(t *testing.T) {
diff := "diff --git a/one.go b/one.go\n--- a/one.go\n+++ b/one.go\n@@ -0,0 +1,2 @@\n+a\n+b\n" +
"diff --git a/two.go b/two.go\n--- a/two.go\n+++ b/two.go\n@@ -5,0 +6,1 @@\n+c\n"
got := parseDiffNewLines(diff)
if !got["one.go"][1] || !got["one.go"][2] {
t.Errorf("one.go: want 1,2 got %v", got["one.go"])
}
if !got["two.go"][6] {
t.Errorf("two.go: want 6 got %v", got["two.go"])
}
}
func TestInlineCommentsNormalizesPaths(t *testing.T) {
// Diff path "b/pkg/x.go" -> "pkg/x.go"; a finding written as "./pkg/x.go" must
// still anchor (both normalize to the same repo-relative path).
addable := parseDiffNewLines("--- a/pkg/x.go\n+++ b/pkg/x.go\n@@ -0,0 +1,1 @@\n+bug\n")
clusters := []cluster{{file: "./pkg/x.go", line: 1, severity: "high", title: "t", models: set("m1"), lenses: set("x")}}
if cs := inlineComments(clusters, addable); len(cs) != 1 || cs[0].Path != "pkg/x.go" {
t.Errorf("path normalization failed to anchor: %+v", cs)
}
}
func TestInlineCommentsFiltersToDiffLines(t *testing.T) {
addable := map[string]map[int]bool{"a.go": {10: true, 11: true}}
clusters := []cluster{
{file: "a.go", line: 10, severity: "high", title: "anchored", models: set("m1", "m2"), lenses: set("security"), detail: "d"},
{file: "a.go", line: 99, severity: "high", title: "off-diff line", models: set("m1"), lenses: set("security")},
{file: "b.go", line: 1, severity: "high", title: "off-diff file", models: set("m1"), lenses: set("security")},
}
cs := inlineComments(clusters, addable)
if len(cs) != 1 {
t.Fatalf("want 1 anchorable inline comment, got %d", len(cs))
}
if cs[0].Path != "a.go" || cs[0].NewPosition != 10 {
t.Errorf("wrong anchor: %+v", cs[0])
}
if !strings.Contains(cs[0].Body, "anchored") || !strings.Contains(cs[0].Body, "2 model") {
t.Errorf("inline body missing title/agreement: %q", cs[0].Body)
}
}
func set(xs ...string) map[string]bool {
m := map[string]bool{}
for _, x := range xs {
m[x] = true
}
return m
}
+252
View File
@@ -0,0 +1,252 @@
package main
import (
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"gopkg.in/yaml.v3"
)
// Specialist is one review lens: a focused instruction appended to the base
// system prompt, run as its own review (+recheck) pass and rendered as its own
// section in the consolidated comment.
type Specialist struct {
Name string // lowercase id, used in GADFLY_SPECIALISTS and section anchors
Title string // human label for the section header
Focus string // lens-specific instruction appended to the base prompt
}
// builtinSpecialists are the lenses Gadfly ships with. The default suite (below)
// is the "biggies"; the rest are opt-in by name via GADFLY_SPECIALISTS, a
// .gadfly.yml `specialists:` list, or `all`.
var builtinSpecialists = map[string]Specialist{
"security": {"security", "🔒 Security", "" +
"Authn/authz gaps, injection (SQL/command/path/template), SSRF, unsafe deserialization, " +
"secret/credential leakage, missing input validation, unsafe handling of untrusted data, " +
"and insecure defaults. Trace tainted input to where it's used."},
"correctness": {"correctness", "🎯 Correctness", "" +
"Logic bugs and incorrect behavior introduced by the change. Pay special attention to " +
"SEMANTIC / domain correctness: do NOT trust a constant, conversion factor, formula, unit, " +
"or threshold because it looks reasonable — independently RE-DERIVE the expected value from " +
"first principles (units, dimensions, boundary values) and compare. A plausible-looking magic " +
"number is exactly where real bugs hide."},
"maintainability": {"maintainability", "🧹 Code cleanliness & maintainability", "" +
"Readability and structure: dead/duplicated code, confusing names, overly long or deeply " +
"nested functions, leaky abstractions, copy-paste that should be shared, and changes that " +
"don't follow the patterns the surrounding code already uses. Prefer concrete, low-churn fixes."},
"performance": {"performance", "⚡ Performance", "" +
"Efficiency regressions: N+1 queries, unnecessary allocations or copies, work inside hot loops, " +
"unbounded growth, missing pagination/limits, blocking calls on hot paths, and avoidable " +
"quadratic behavior. Only flag impact you can justify, not micro-optimizations."},
"error-handling": {"error-handling", "🧯 Error handling & edge cases", "" +
"Ignored or swallowed errors, missing cleanup/rollback/`defer`, panics on bad input, and " +
"unhandled edge cases: nil/null, empty collections, zero/negative, integer overflow, and " +
"boundary (off-by-one) conditions. Check the unhappy paths the diff introduces."},
// --- opt-in (not in the default suite) ---
"tests": {"tests", "🧪 Tests", "" +
"Whether the change is adequately tested: new logic without tests, changed behavior whose " +
"tests weren't updated, missing edge-case/error-path coverage, and brittle or assertion-free tests."},
"docs": {"docs", "📝 Docs", "" +
"Documentation drift: public APIs/flags/config changed without updating doc comments, README, " +
"or examples; stale comments now contradicting the code; missing docs for new exported surface."},
"conventions": {"conventions", "📐 Conventions", "" +
"Adherence to THIS repo's own conventions. Discover them — read any README / CONTRIBUTING / " +
"CLAUDE.md / AGENTS.md / lint config the repo ships, and hold the change to the documented and " +
"surrounding-code patterns."},
"improvements": {"improvements", "💡 Improvements (optional)", "" +
"OPTIONAL, non-blocking suggestions only. Surface at most 12 genuinely high-value blind spots " +
"or simplifications the author likely wants to know about. Be quiet: if nothing rises to that " +
"bar, reply exactly 'No suggestions.' Never nitpick, never pad. These never affect the verdict."},
}
// defaultSuite is used when nothing selects specialists explicitly.
var defaultSuite = []string{"security", "correctness", "maintainability", "performance", "error-handling"}
// fileConfig is the optional .gadfly.yml schema.
type fileConfig struct {
// Specialists selects which run (names). Empty => the default suite.
Specialists []string `yaml:"specialists"`
// Define adds or overrides specialist definitions.
Define []struct {
Name string `yaml:"name"`
Title string `yaml:"title"`
Focus string `yaml:"focus"`
} `yaml:"define"`
}
// resolveSpecialists builds the active specialist list from (in increasing
// precedence) the built-ins, a repo .gadfly.yml, and GADFLY_SPECIALIST_<NAME>
// env definitions. The active SELECTION is GADFLY_SPECIALISTS (csv, "all", or
// "auto"), else the file's `specialists:`, else the default suite. Unknown
// names are reported as errors and skipped; valid ones still run.
//
// It also returns the full registry (the catalog of every defined specialist)
// and an auto flag: when the selection is "auto", specs is nil and the caller
// runs the dynamic selector (see auto.go) against the registry instead.
func resolveSpecialists(repoDir string) (specs []Specialist, registry map[string]Specialist, auto bool, errs []error) {
// 1. registry seeded with built-ins.
registry = make(map[string]Specialist, len(builtinSpecialists))
for k, v := range builtinSpecialists {
registry[k] = v
}
// 2. config file: definitions + (maybe) the selection.
var fileSelection []string
if cfg, ok, err := loadFileConfig(repoDir); err != nil {
errs = append(errs, err)
} else if ok {
for _, d := range cfg.Define {
name := strings.ToLower(strings.TrimSpace(d.Name))
if name == "" || strings.TrimSpace(d.Focus) == "" {
errs = append(errs, fmt.Errorf(".gadfly.yml: specialist define needs non-empty name and focus"))
continue
}
registry[name] = Specialist{Name: name, Title: titleOr(d.Title, name), Focus: d.Focus}
}
fileSelection = cfg.Specialists
}
// 3. env definitions: GADFLY_SPECIALIST_<NAME>="focus".
for _, kv := range os.Environ() {
key, val, _ := strings.Cut(kv, "=")
if !strings.HasPrefix(key, "GADFLY_SPECIALIST_") || len(key) == len("GADFLY_SPECIALIST_") {
continue
}
name := strings.ToLower(strings.TrimPrefix(key, "GADFLY_SPECIALIST_"))
if strings.TrimSpace(val) == "" {
errs = append(errs, fmt.Errorf("%s: empty specialist focus", key))
continue
}
existing, ok := registry[name]
title := titleOr("", name)
if ok {
title = existing.Title
}
registry[name] = Specialist{Name: name, Title: title, Focus: val}
}
// 4. selection: env > file > default suite.
var names []string
switch {
case strings.TrimSpace(os.Getenv("GADFLY_SPECIALISTS")) != "":
names = splitCSV(os.Getenv("GADFLY_SPECIALISTS"))
case len(fileSelection) > 0:
names = fileSelection
default:
names = defaultSuite
}
if len(names) == 1 && strings.EqualFold(names[0], "auto") {
// Dynamic selection: caller runs the selector against the registry.
return nil, registry, true, errs
}
if len(names) == 1 && strings.EqualFold(names[0], "all") {
names = sortedKeys(registry)
}
// 5. resolve names -> specialists.
var out []Specialist
seen := map[string]bool{}
for _, raw := range names {
name := strings.ToLower(strings.TrimSpace(raw))
if name == "" || seen[name] {
continue
}
seen[name] = true
sp, ok := registry[name]
if !ok {
errs = append(errs, fmt.Errorf("unknown specialist %q (built-ins: %s; or define it via .gadfly.yml / GADFLY_SPECIALIST_%s)",
name, strings.Join(sortedKeys(builtinSpecialists), ", "), strings.ToUpper(name)))
continue
}
out = append(out, sp)
}
return out, registry, false, errs
}
// suiteFromRegistry resolves names against the registry, skipping any missing.
// Used as the auto-mode fallback (the default suite always exists in builtins).
func suiteFromRegistry(registry map[string]Specialist, names []string) []Specialist {
var out []Specialist
for _, n := range names {
if sp, ok := registry[n]; ok {
out = append(out, sp)
}
}
return out
}
// specialistNamesOf returns the names of a specialist slice (for logging).
func specialistNamesOf(specs []Specialist) []string {
names := make([]string, len(specs))
for i, s := range specs {
names[i] = s.Name
}
return names
}
// composeSpecialistPrompt appends a specialist's lens to the base system prompt.
func composeSpecialistPrompt(base string, sp Specialist) string {
return strings.TrimRight(base, "\n") +
"\n\n## Your assigned lens — " + sp.Title + "\n" +
"Review the change specifically and ONLY through this lens. Scrutinize it for:\n" +
sp.Focus +
"\n\nStay in this lane: other lenses (correctness, security, performance, etc.) are reviewed " +
"separately, so don't duplicate their findings here. If nothing in your lens is materially " +
"wrong, reply with the \"No material issues found\" verdict for this lens — do not reach for " +
"another lens's issue just to have something to say."
}
func loadFileConfig(repoDir string) (fileConfig, bool, error) {
candidates := []string{}
if p := strings.TrimSpace(os.Getenv("GADFLY_CONFIG")); p != "" {
if !filepath.IsAbs(p) {
p = filepath.Join(repoDir, p)
}
candidates = append(candidates, p)
} else {
candidates = append(candidates, filepath.Join(repoDir, ".gadfly.yml"), filepath.Join(repoDir, ".gadfly.yaml"))
}
for _, p := range candidates {
data, err := os.ReadFile(p)
if err != nil {
continue // not found / unreadable: try next
}
var cfg fileConfig
if err := yaml.Unmarshal(data, &cfg); err != nil {
return fileConfig{}, false, fmt.Errorf("parse %s: %w", filepath.Base(p), err)
}
return cfg, true, nil
}
return fileConfig{}, false, nil
}
func titleOr(title, name string) string {
if strings.TrimSpace(title) != "" {
return title
}
return name
}
func splitCSV(s string) []string {
var out []string
for _, p := range strings.Split(s, ",") {
if t := strings.TrimSpace(p); t != "" {
out = append(out, t)
}
}
return out
}
func sortedKeys(m map[string]Specialist) []string {
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
sort.Strings(keys)
return keys
}
+138
View File
@@ -0,0 +1,138 @@
package main
import (
"os"
"path/filepath"
"testing"
)
func names(specs []Specialist) []string {
out := make([]string, len(specs))
for i, s := range specs {
out[i] = s.Name
}
return out
}
func eq(a, b []string) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if a[i] != b[i] {
return false
}
}
return true
}
func TestResolveSpecialists_DefaultSuite(t *testing.T) {
t.Setenv("GADFLY_SPECIALISTS", "")
specs, _, _, errs := resolveSpecialists(t.TempDir())
if len(errs) != 0 {
t.Fatalf("unexpected errors: %v", errs)
}
if !eq(names(specs), defaultSuite) {
t.Errorf("default = %v, want %v", names(specs), defaultSuite)
}
}
func TestResolveSpecialists_EnvSelection(t *testing.T) {
t.Setenv("GADFLY_SPECIALISTS", "security, tests")
specs, _, _, errs := resolveSpecialists(t.TempDir())
if len(errs) != 0 {
t.Fatalf("unexpected errors: %v", errs)
}
if !eq(names(specs), []string{"security", "tests"}) {
t.Errorf("got %v", names(specs))
}
}
func TestResolveSpecialists_AutoFlag(t *testing.T) {
t.Setenv("GADFLY_SPECIALISTS", "auto")
specs, registry, auto, errs := resolveSpecialists(t.TempDir())
if len(errs) != 0 {
t.Fatalf("unexpected errors: %v", errs)
}
if !auto {
t.Error("expected auto=true for GADFLY_SPECIALISTS=auto")
}
if specs != nil {
t.Errorf("auto mode should return nil specs, got %v", names(specs))
}
if len(registry) == 0 {
t.Error("auto mode should still return the registry catalog")
}
}
func TestResolveSpecialists_UnknownNameErrors(t *testing.T) {
t.Setenv("GADFLY_SPECIALISTS", "security,bogus")
specs, _, _, errs := resolveSpecialists(t.TempDir())
if len(errs) == 0 {
t.Fatal("expected an error for unknown specialist")
}
if !eq(names(specs), []string{"security"}) {
t.Errorf("valid ones should still resolve, got %v", names(specs))
}
}
func TestResolveSpecialists_EnvCustomDefinition(t *testing.T) {
t.Setenv("GADFLY_SPECIALIST_MIGRATIONS", "Review DB migrations for destructive ops.")
t.Setenv("GADFLY_SPECIALISTS", "migrations")
specs, _, _, errs := resolveSpecialists(t.TempDir())
if len(errs) != 0 {
t.Fatalf("unexpected errors: %v", errs)
}
if len(specs) != 1 || specs[0].Name != "migrations" || specs[0].Focus == "" {
t.Fatalf("custom specialist not registered: %+v", specs)
}
}
func TestResolveSpecialists_FileConfig(t *testing.T) {
dir := t.TempDir()
cfg := `specialists: [security, migrations]
define:
- name: migrations
title: "DB migrations"
focus: "Review schema migrations for destructive or unindexed changes."
`
if err := os.WriteFile(filepath.Join(dir, ".gadfly.yml"), []byte(cfg), 0o644); err != nil {
t.Fatal(err)
}
t.Setenv("GADFLY_SPECIALISTS", "") // let the file drive selection
specs, _, _, errs := resolveSpecialists(dir)
if len(errs) != 0 {
t.Fatalf("unexpected errors: %v", errs)
}
if !eq(names(specs), []string{"security", "migrations"}) {
t.Errorf("got %v", names(specs))
}
}
func TestParseVerdictAndWorst(t *testing.T) {
cases := map[string]verdict{
"VERDICT: No material issues found.": verdictClean,
"Minor issues\n- nit": verdictMinor,
"**Blocking issues found**": verdictBlocking,
"**Blocking issues**\n- bug": verdictBlocking, // no "found" suffix
"VERDICT: Blocking issue\n- one": verdictBlocking, // singular
"No material issues found. There are no blocking issues.": verdictClean, // earliest phrase wins
"something unparseable": verdictUnknown,
}
for in, want := range cases {
if got := parseVerdict(in); got != want {
t.Errorf("parseVerdict(%q) = %v, want %v", in, got, want)
}
}
results := []specialistResult{
{spec: Specialist{Name: "security"}, verdict: verdictMinor},
{spec: Specialist{Name: "correctness"}, verdict: verdictBlocking},
{spec: Specialist{Name: "improvements"}, verdict: verdictBlocking}, // must not count
}
if w := worstVerdict(results[:2]); w != verdictBlocking {
t.Errorf("worst = %v, want blocking", w)
}
if w := worstVerdict([]specialistResult{results[0], results[2]}); w != verdictMinor {
t.Errorf("improvements should not escalate; worst = %v, want minor", w)
}
}
+131
View File
@@ -0,0 +1,131 @@
package main
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"sync"
"time"
)
// Lens states for the live status board. A lens starts queued, becomes running
// when its pass begins, and ends finished (with a verdict, or errored).
const (
lensQueued = "queued"
lensRunning = "running"
lensFinished = "finished"
)
// lensStatus is one specialist lens's progress, as rendered by the entrypoint
// status board (scripts/status-board.sh).
type lensStatus struct {
Name string `json:"name"`
State string `json:"state"` // queued | running | finished
Verdict string `json:"verdict,omitempty"` // set when finished (the lens's label)
Errored bool `json:"errored,omitempty"` // the lens failed to complete
}
// modelStatus is the on-disk shape one model process publishes for the live
// status board: a snapshot of this model's lenses as they progress. The board
// reads every model's file and renders a single consolidated PR comment.
type modelStatus struct {
Model string `json:"model"`
Provider string `json:"provider"`
Started int64 `json:"started"` // unix seconds
Updated int64 `json:"updated"` // unix seconds, bumped on every change
Done bool `json:"done"` // all lenses finished
Lenses []lensStatus `json:"lenses"`
}
// statusWriter maintains a model's status file as its lenses progress. It is
// purely opt-in: when GADFLY_STATUS_FILE is unset the writer's path is empty and
// every method is a no-op, so a plain run (and the unit tests) never touch the
// filesystem and behave exactly as before. Writes are atomic (temp file +
// rename within the same dir) so the board never reads a half-written file even
// though lenses can finish concurrently.
type statusWriter struct {
path string
mu sync.Mutex
st modelStatus
}
// newStatusWriter seeds a writer with every lens queued and flushes the initial
// snapshot. model/provider are echoed into the file so the board can render
// them without re-deriving from the filename (which is sanitized). The status
// file path comes from GADFLY_STATUS_FILE (set by run.sh per model); when empty
// the writer is inert.
func newStatusWriter(model, provider string, specialists []Specialist) *statusWriter {
w := &statusWriter{path: strings.TrimSpace(os.Getenv("GADFLY_STATUS_FILE"))}
w.st = modelStatus{
Model: model,
Provider: provider,
Started: time.Now().Unix(),
}
for _, sp := range specialists {
w.st.Lenses = append(w.st.Lenses, lensStatus{Name: sp.Name, State: lensQueued})
}
w.flush()
return w
}
// set transitions a lens to a new state (and verdict/errored when finished),
// recomputes the overall done flag, and atomically rewrites the file. Unknown
// lens names are ignored. Safe for concurrent callers (one goroutine per lens).
func (w *statusWriter) set(name, state, verdict string, errored bool) {
if w == nil || w.path == "" {
return
}
w.mu.Lock()
defer w.mu.Unlock()
for i := range w.st.Lenses {
if w.st.Lenses[i].Name == name {
w.st.Lenses[i].State = state
w.st.Lenses[i].Verdict = verdict
w.st.Lenses[i].Errored = errored
break
}
}
done := true
for _, l := range w.st.Lenses {
if l.State != lensFinished {
done = false
break
}
}
w.st.Done = done
w.flush()
}
// flush writes the current snapshot atomically. Best-effort: any error is
// swallowed (the status board is advisory and must never affect the review).
func (w *statusWriter) flush() {
if w.path == "" {
return
}
w.st.Updated = time.Now().Unix()
data, err := json.MarshalIndent(&w.st, "", " ")
if err != nil {
return
}
dir := filepath.Dir(w.path)
tmp, err := os.CreateTemp(dir, ".status-*.tmp")
if err != nil {
return
}
tmpName := tmp.Name()
if _, err := tmp.Write(data); err != nil {
tmp.Close()
os.Remove(tmpName)
return
}
if err := tmp.Close(); err != nil {
os.Remove(tmpName)
return
}
// Rename is atomic within the same filesystem, so the board reader sees
// either the old complete file or the new complete file — never a partial.
if err := os.Rename(tmpName, w.path); err != nil {
os.Remove(tmpName)
}
}
+103
View File
@@ -0,0 +1,103 @@
package main
import (
"encoding/json"
"os"
"path/filepath"
"testing"
)
// readStatus loads a modelStatus written by the statusWriter.
func readStatus(t *testing.T, path string) modelStatus {
t.Helper()
data, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read status file: %v", err)
}
var st modelStatus
if err := json.Unmarshal(data, &st); err != nil {
t.Fatalf("unmarshal status: %v", err)
}
return st
}
func TestStatusWriterLifecycle(t *testing.T) {
path := filepath.Join(t.TempDir(), "glm.json")
t.Setenv("GADFLY_STATUS_FILE", path)
specs := []Specialist{
{Name: "security", Title: "Security"},
{Name: "correctness", Title: "Correctness"},
}
w := newStatusWriter("glm-5.2:cloud", "ollama-cloud", specs)
// Initial snapshot: both lenses queued, model not done, metadata echoed.
st := readStatus(t, path)
if st.Model != "glm-5.2:cloud" || st.Provider != "ollama-cloud" {
t.Fatalf("model/provider not echoed: %+v", st)
}
if len(st.Lenses) != 2 {
t.Fatalf("want 2 lenses, got %d", len(st.Lenses))
}
for _, l := range st.Lenses {
if l.State != lensQueued {
t.Fatalf("lens %q want queued, got %q", l.Name, l.State)
}
}
if st.Done {
t.Fatal("model marked done while lenses still queued")
}
if st.Started == 0 {
t.Fatal("started timestamp not set")
}
// Transition one lens through running -> finished; model not yet done.
w.set("security", lensRunning, "", false)
if got := readStatus(t, path); got.Lenses[0].State != lensRunning {
t.Fatalf("security want running, got %q", got.Lenses[0].State)
}
w.set("security", lensFinished, "No material issues found", false)
st = readStatus(t, path)
if st.Lenses[0].State != lensFinished || st.Lenses[0].Verdict != "No material issues found" {
t.Fatalf("security finish not recorded: %+v", st.Lenses[0])
}
if st.Done {
t.Fatal("model marked done with one lens still queued")
}
// Finish the second lens (errored) -> model done.
w.set("correctness", lensFinished, "Reviewed", true)
st = readStatus(t, path)
if !st.Done {
t.Fatal("model should be done after all lenses finished")
}
if !st.Lenses[1].Errored {
t.Fatal("errored flag not recorded for correctness")
}
if st.Updated < st.Started {
t.Fatalf("updated (%d) should be >= started (%d)", st.Updated, st.Started)
}
}
// With GADFLY_STATUS_FILE unset the writer is inert: no file, no panic.
func TestStatusWriterDisabled(t *testing.T) {
t.Setenv("GADFLY_STATUS_FILE", "")
w := newStatusWriter("m", "p", []Specialist{{Name: "security"}})
w.set("security", lensFinished, "Minor issues", false)
// Nothing to assert beyond "did not panic / did not write" — a nil-safe set
// on the disabled writer is the contract.
if w.path != "" {
t.Fatalf("expected empty path when disabled, got %q", w.path)
}
}
// set must ignore unknown lens names rather than panic or append.
func TestStatusWriterUnknownLens(t *testing.T) {
path := filepath.Join(t.TempDir(), "s.json")
t.Setenv("GADFLY_STATUS_FILE", path)
w := newStatusWriter("m", "p", []Specialist{{Name: "security"}})
w.set("does-not-exist", lensRunning, "", false)
if st := readStatus(t, path); len(st.Lenses) != 1 || st.Lenses[0].State != lensQueued {
t.Fatalf("unknown lens mutated state: %+v", st.Lenses)
}
}
+167 -18
View File
@@ -9,6 +9,7 @@ import (
"regexp"
"sort"
"strings"
"sync"
llm "gitea.stevedudenhoeffer.com/steve/majordomo/llm"
)
@@ -17,11 +18,13 @@ import (
// every tool caps how much it can pull in one call — a runaway read_file or
// grep would blow the window and stall the loop.
const (
maxFileBytes = 64 * 1024 // per read_file call
maxReadLines = 800 // per read_file call
maxGrepResults = 200 // per grep call
maxFindResults = 200 // per find_files call
maxLineLen = 400 // truncate any single returned line to this
maxFileBytes = 64 * 1024 // per read_file call
maxReadLines = 800 // per read_file call
maxGrepResults = 200 // per grep call
maxFindResults = 200 // per find_files call
maxLineLen = 400 // truncate any single returned line to this
maxGetDiffLines = 800 // per get_diff call (paginated window)
maxGetDiffBytes = 64 * 1024 // per get_diff call
)
// skipDirs are never descended into by grep / find_files — noise and bulk that
@@ -37,8 +40,15 @@ var skipDirs = map[string]bool{
// escapes (symlink or `..` traversal), so a hostile diff can never make the
// reviewer read outside the checkout.
type repoFS struct {
root string // absolute, symlink-resolved repo root
diff string // the full PR unified diff (served by get_diff)
root string // absolute, symlink-resolved repo root
diff string // the full PR unified diff (served by get_diff)
worker llm.Model // optional cheap model for delegate_investigation; nil = no delegation
// diffLines caches the split diff so paging through get_diff doesn't re-split
// the whole (possibly large) diff on every call. The diff is immutable, so the
// cache is computed once and safe to share across concurrent lenses.
diffOnce sync.Once
diffLines []string
}
// newRepoFS resolves root to an absolute, symlink-free path.
@@ -96,17 +106,47 @@ func (r *repoFS) contains(abs string) error {
return nil
}
// toolbox builds the read-only review toolbox over this sandbox.
func (r *repoFS) toolbox() (*llm.Toolbox, error) {
box := llm.NewToolbox("gadfly")
tools := []llm.Tool{
// fsTools is the set of read-only repository tools.
func (r *repoFS) fsTools() []llm.Tool {
return []llm.Tool{
r.readFileTool(),
r.listDirTool(),
r.grepTool(),
r.findFilesTool(),
r.getDiffTool(),
}
for _, t := range tools {
}
// allTools is the single source of truth for the reviewer's tool set: the
// read-only repo tools, plus delegate_investigation when a worker model is
// configured. Both the executus registry (gadflyToolRegistry, the production
// path) and toolbox() build from this list.
func (r *repoFS) allTools() []llm.Tool {
tools := r.fsTools()
if r.worker != nil {
tools = append(tools, r.delegateTool())
}
return tools
}
// toolbox builds a majordomo toolbox from allTools(). The production review path
// now goes through executus's tool.Registry (see executus.go); this remains for
// the toolbox-level tests (the `call` helper).
func (r *repoFS) toolbox() (*llm.Toolbox, error) {
box := llm.NewToolbox("gadfly")
for _, t := range r.allTools() {
if err := box.Add(t); err != nil {
return nil, fmt.Errorf("add tool %q: %w", t.Name, err)
}
}
return box, nil
}
// workerToolbox is the toolbox handed to a delegated worker sub-agent: the
// read-only repo tools only (no delegate tool — workers don't sub-delegate).
func (r *repoFS) workerToolbox() (*llm.Toolbox, error) {
box := llm.NewToolbox("gadfly-worker")
for _, t := range r.fsTools() {
if err := box.Add(t); err != nil {
return nil, fmt.Errorf("add tool %q: %w", t.Name, err)
}
@@ -374,15 +414,124 @@ func (r *repoFS) findFilesTool() llm.Tool {
)
}
type getDiffArgs struct {
Path string `json:"path,omitempty" description:"Optional changed-file path (e.g. pkg/foo/bar.go); returns ONLY that file's diff hunks. Omit for the whole diff. Use this on a large PR to pull just the file a finding is about."`
StartLine int `json:"start_line,omitempty" description:"Optional 1-based line to start from within the (whole or path-scoped) diff (default 1)."`
Limit int `json:"limit,omitempty" description:"Optional max number of diff lines to return (default/maximum 800)."`
}
func (r *repoFS) getDiffTool() llm.Tool {
return llm.DefineTool[struct{}](
return llm.DefineTool[getDiffArgs](
"get_diff",
"Return the complete unified diff under review. The diff is also included (possibly truncated) in the task message; call this to get the full, untruncated text.",
func(_ context.Context, _ struct{}) (any, error) {
if strings.TrimSpace(r.diff) == "" {
return "(empty diff)", nil
"Return the unified diff under review as a numbered, PAGINATED window (like read_file) — not the whole diff at once, so a huge PR can't blow the context window. Pass `path` to fetch just one changed file's hunks, or `start_line`/`limit` to page through. A truncated copy of the diff is also embedded in the task message.",
func(_ context.Context, args getDiffArgs) (any, error) {
scope := "the diff"
var lines []string
if p := strings.TrimSpace(args.Path); p != "" {
lines = diffLinesForPath(r.diff, p)
if len(lines) == 0 {
return fmt.Sprintf("(no diff hunks for path %q; check it against the changed-files list — the path must match a file the diff touches)", p), nil
}
scope = "the diff for " + p
} else {
lines = r.diffAllLines()
if len(lines) == 0 {
return "(empty diff)", nil
}
}
return r.diff, nil
return windowDiff(lines, scope, args.StartLine, args.Limit), nil
},
)
}
// diffAllLines returns the whole diff split into lines, cached so paging never
// re-splits the (possibly large) diff.
func (r *repoFS) diffAllLines() []string {
r.diffOnce.Do(func() { r.diffLines = splitDiffLines(r.diff) })
return r.diffLines
}
// splitDiffLines splits a unified diff into lines, dropping the single trailing
// empty element a trailing newline produces — otherwise windowDiff would emit a
// blank final line and over-count the total by one.
func splitDiffLines(diff string) []string {
lines := strings.Split(diff, "\n")
if n := len(lines); n > 0 && lines[n-1] == "" {
lines = lines[:n-1]
}
return lines
}
// windowDiff returns a numbered, paginated slice of pre-split diff lines,
// mirroring read_file's caps (maxGetDiffLines / maxGetDiffBytes / maxLineLen) so
// a single get_diff call can never dump a multi-hundred-KB diff into the
// transcript — the amplifier behind the large-PR token burn. The full diff stays
// reachable by paging with start_line, or scoped per file via the path arg.
func windowDiff(lines []string, scope string, start, limit int) string {
total := len(lines)
if start < 1 {
start = 1
}
if limit <= 0 || limit > maxGetDiffLines {
limit = maxGetDiffLines
}
if start > total {
return fmt.Sprintf("(%s has %d lines; nothing at/after line %d)", scope, total, start)
}
var b strings.Builder
emitted := 0
i := start - 1
for ; i < total; i++ {
if emitted >= limit || b.Len() >= maxGetDiffBytes {
break
}
line := lines[i]
if len(line) > maxLineLen {
line = line[:maxLineLen] + "…"
}
fmt.Fprintf(&b, "%d\t%s\n", i+1, line)
emitted++
}
if i < total {
// i (0-based) is the first line NOT emitted; line i was the last shown.
fmt.Fprintf(&b, "... (%s truncated after line %d of %d; call get_diff again with start_line=%d for the rest, or pass a `path` to scope to one file)\n", scope, i, total, i+1)
}
return b.String()
}
// diffLinesForPath returns the unified-diff lines for one changed file: from the
// `diff --git` header that names path through to the next header (or end). The
// header names two path tokens (a/<old> b/<new>); a match is on a WHOLE token,
// so path "foo.go" does not pull in "barfoo.go" — and a trailing "/" scopes to a
// directory (e.g. "pkg/foo/" matches pkg/foo/bar.go).
func diffLinesForPath(diff, path string) []string {
want := strings.TrimPrefix(strings.TrimPrefix(strings.TrimSpace(path), "a/"), "b/")
var out []string
inSection := false
for _, ln := range splitDiffLines(diff) {
if strings.HasPrefix(ln, "diff --git ") {
inSection = diffHeaderNames(ln, want)
}
if inSection {
out = append(out, ln)
}
}
return out
}
// diffHeaderNames reports whether a `diff --git a/X b/Y` header names want as one
// of its (a/b-stripped) path tokens — exact whole-token match, or a directory
// prefix when want ends with "/".
func diffHeaderNames(header, want string) bool {
fields := strings.Fields(header)
if len(fields) < 3 {
return false
}
for _, f := range fields[2:] {
p := strings.TrimPrefix(strings.TrimPrefix(f, "a/"), "b/")
if p == want || (strings.HasSuffix(want, "/") && strings.HasPrefix(p, want)) {
return true
}
}
return false
}
+81 -3
View File
@@ -197,15 +197,93 @@ func TestFindFilesTool(t *testing.T) {
func TestGetDiffTool(t *testing.T) {
root := buildFixtureRepo(t)
const diff = "diff --git a/x b/x\n+added line\n"
const diff = "diff --git a/x b/x\n--- a/x\n+++ b/x\n+added line\n" +
"diff --git a/y.go b/y.go\n--- a/y.go\n+++ b/y.go\n+y change\n"
fs, _ := newRepoFS(root, diff)
// Default: the whole diff as a NUMBERED window (paginated), not a raw dump —
// so a huge PR can't be poured into the transcript in one call.
out, err := call(t, fs, "get_diff", map[string]any{})
if err != nil {
t.Fatalf("get_diff: %v", err)
}
if !strings.Contains(out, "1\tdiff --git a/x b/x") {
t.Errorf("get_diff should return a numbered window, got:\n%s", out)
}
if !strings.Contains(out, "+added line") || !strings.Contains(out, "+y change") {
t.Errorf("the full (short) diff window should include every hunk, got:\n%s", out)
}
// path filter: only the named file's hunks come back.
out, err = call(t, fs, "get_diff", map[string]any{"path": "y.go"})
if err != nil {
t.Fatalf("get_diff path: %v", err)
}
if !strings.Contains(out, "y change") {
t.Errorf("get_diff path=y.go should include y's hunk, got:\n%s", out)
}
if strings.Contains(out, "added line") {
t.Errorf("get_diff path=y.go must NOT include x's hunk, got:\n%s", out)
}
// unknown path: a clear note, never an error.
out, err = call(t, fs, "get_diff", map[string]any{"path": "nope.txt"})
if err != nil {
t.Fatalf("get_diff unknown path: %v", err)
}
if !strings.Contains(out, "no diff hunks") {
t.Errorf("get_diff for an unknown path should note no hunks, got:\n%s", out)
}
}
// TestGetDiffTool_Paginates: a diff longer than the per-call line cap is returned
// as a truncated window with a paging hint, and start_line pages past it — the
// mechanism that stops get_diff from dumping a multi-hundred-KB diff at once.
func TestGetDiffTool_Paginates(t *testing.T) {
diff := "diff --git a/big b/big\n" + strings.Repeat("+line\n", maxGetDiffLines+50)
fs, _ := newRepoFS(t.TempDir(), diff)
out, err := call(t, fs, "get_diff", map[string]any{})
if err != nil {
t.Fatalf("get_diff: %v", err)
}
if out != diff {
t.Errorf("get_diff returned %q, want %q", out, diff)
if !strings.Contains(out, "truncated after line") {
t.Error("a diff longer than the per-call cap should be truncated with a paging hint")
}
if strings.Contains(out, "801\t") {
t.Error("the first window must stop at the line cap, not reach line 801")
}
out, err = call(t, fs, "get_diff", map[string]any{"start_line": 805})
if err != nil {
t.Fatalf("get_diff page: %v", err)
}
if !strings.Contains(out, "805\t") {
t.Error("paging with start_line=805 should include line 805")
}
}
// TestDiffLinesForPath_Anchored: get_diff path= matches on a WHOLE path token,
// so "foo.go" never pulls in "barfoo.go" (the unanchored-substring weakness the
// swarm flagged), while a trailing "/" still scopes to a directory.
func TestDiffLinesForPath_Anchored(t *testing.T) {
diff := "diff --git a/foo.go b/foo.go\n+foo change\n" +
"diff --git a/barfoo.go b/barfoo.go\n+barfoo change\n" +
"diff --git a/pkg/x.go b/pkg/x.go\n+x change\n"
joined := strings.Join(diffLinesForPath(diff, "foo.go"), "\n")
if !strings.Contains(joined, "foo change") {
t.Errorf("foo.go should match its own hunk:\n%s", joined)
}
if strings.Contains(joined, "barfoo change") {
t.Errorf("foo.go must NOT match barfoo.go (unanchored substring regression):\n%s", joined)
}
if !strings.Contains(strings.Join(diffLinesForPath(diff, "pkg/"), "\n"), "x change") {
t.Error("a trailing-slash path should scope to the directory (pkg/ -> pkg/x.go)")
}
if len(diffLinesForPath(diff, "nope.go")) != 0 {
t.Error("an unknown path should yield no lines")
}
}
+20 -51
View File
@@ -37,10 +37,11 @@ func lastUserText(req llm.Request) string {
return req.Messages[len(req.Messages)-1].Text()
}
// TestRunAgent_WrapUpNudgeProducesAnswer: a model that keeps calling tools until
// it is nudged to wrap up should still finish inside its budget — the steer
// message arrives a few steps before the cap and the model writes its answer.
func TestRunAgent_WrapUpNudgeProducesAnswer(t *testing.T) {
// TestReviewExecutor_WrapUpNudgeProducesAnswer: a model that keeps calling tools
// until it is nudged to wrap up should still finish inside its budget — the steer
// message (delivered by the executus wrap-up critic a few steps before the cap)
// arrives and the model writes its answer.
func TestReviewExecutor_WrapUpNudgeProducesAnswer(t *testing.T) {
t.Setenv("GADFLY_WRAPUP_RESERVE", "4")
final := "VERDICT: No material issues found."
@@ -60,9 +61,10 @@ func TestRunAgent_WrapUpNudgeProducesAnswer(t *testing.T) {
}
fs, _ := newRepoFS(t.TempDir(), "diff --git a/x b/x\n+y\n")
out, err := runAgent(context.Background(), mdl, fs, "sys", "task", 12)
rex := newTestReviewExecutor(t, mdl, fs)
out, err := rex.run(context.Background(), "sys", "task", 12)
if err != nil {
t.Fatalf("runAgent should succeed via wrap-up nudge, got error: %v", err)
t.Fatalf("run should succeed via wrap-up nudge, got error: %v", err)
}
if out != final {
t.Errorf("expected final review %q, got %q", final, out)
@@ -72,24 +74,19 @@ func TestRunAgent_WrapUpNudgeProducesAnswer(t *testing.T) {
}
}
// TestRunAgent_FinalizationFallback: a model that ignores the wrap-up nudge and
// spins on tools until the cap should NOT hard-fail — the tool-free finalization
// pass forces a final answer out of the transcript.
func TestRunAgent_FinalizationFallback(t *testing.T) {
// TestReviewExecutor_ExhaustionWithoutAnswerIsError: a model that ignores the
// wrap-up nudge and spins on tools until the step cap produces no final answer.
// The transcript-based forced-finalization fallback was removed in the executus
// re-platform (run.Result does not expose the loop transcript), so the pass now
// surfaces an error — which reviewWithSpecialist renders as an advisory "reviewer
// failed to complete" notice rather than a phantom success.
func TestReviewExecutor_ExhaustionWithoutAnswerIsError(t *testing.T) {
t.Setenv("GADFLY_WRAPUP_RESERVE", "2")
final := "VERDICT: Minor issues\n- something"
forcedCalled := false
n := 0
p := fake.New("fake", fake.WithDefault(func(_ string, req llm.Request) fake.Step {
// Only the tool-free finalization pass forbids tools — reply there.
if req.ToolChoice == "none" {
forcedCalled = true
return fake.Reply(final)
}
// Otherwise keep spinning, ignoring the wrap-up nudge entirely.
p := fake.New("fake", fake.WithDefault(func(_ string, _ llm.Request) fake.Step {
n++
return spinToolCall(n)
return spinToolCall(n) // spin forever, ignoring the wrap-up nudge
}))
mdl, err := p.Model("mock")
if err != nil {
@@ -97,37 +94,9 @@ func TestRunAgent_FinalizationFallback(t *testing.T) {
}
fs, _ := newRepoFS(t.TempDir(), "diff --git a/x b/x\n+y\n")
out, err := runAgent(context.Background(), mdl, fs, "sys", "task", 6)
if err != nil {
t.Fatalf("runAgent should recover via finalization fallback, got error: %v", err)
}
if !forcedCalled {
t.Error("finalization fallback was never invoked")
}
if out != final {
t.Errorf("expected forced final answer %q, got %q", final, out)
}
}
// TestRunAgent_FallbackStillEmptyIsError: if even the tool-free finalization
// yields nothing, runAgent surfaces an error rather than a phantom success.
func TestRunAgent_FallbackStillEmptyIsError(t *testing.T) {
n := 0
p := fake.New("fake", fake.WithDefault(func(_ string, req llm.Request) fake.Step {
if req.ToolChoice == "none" {
return fake.Reply(" ") // finalization produces only whitespace
}
n++
return spinToolCall(n)
}))
mdl, err := p.Model("mock")
if err != nil {
t.Fatal(err)
}
fs, _ := newRepoFS(t.TempDir(), "diff --git a/x b/x\n+y\n")
if _, err := runAgent(context.Background(), mdl, fs, "sys", "task", 4); err == nil {
t.Error("runAgent should error when the finalization fallback also yields no output")
rex := newTestReviewExecutor(t, mdl, fs)
if _, err := rex.run(context.Background(), "sys", "task", 6); err == nil {
t.Error("run should error when the model exhausts its steps without an answer")
}
}
+306 -15
View File
@@ -31,13 +31,29 @@
# COMMENT_ID comment id, for the 👀 reaction (issue_comment only)
# ACTOR github.actor (the user who triggered)
# Optional config:
# OLLAMA_REVIEW_MODELS comma-separated model ids (default below)
# GADFLY_MODELS comma-separated model ids/specs (alias: OLLAMA_REVIEW_MODELS)
# GADFLY_PROVIDER majordomo provider for bare model ids (default ollama-cloud;
# e.g. "ollama" local, "openai", "anthropic", "google")
# GADFLY_BASE_URL override backend endpoint (OpenAI/Ollama-compatible servers)
# GADFLY_API_KEY provider key (else provider's standard env: OPENAI_API_KEY, …)
# CLAUDE_CODE_OAUTH_TOKEN auth for the claude-code engine (GADFLY_MODELS entry
# "claude-code"/"claude-code/<model>"); Pro/Max subscription
# token from `claude setup-token`. Else ANTHROPIC_API_KEY.
# GADFLY_TRIGGER_PHRASE comment phrase that triggers a re-review (default "@gadfly review")
# GADFLY_ALLOWED_USERS comma-separated usernames allowed to comment-trigger;
# empty => fall back to "is a repo collaborator"
# GADFLY_FINDINGS_URL optional gadfly-reports store base URL; set to POST the run +
# findings for model-quality tracking (off when empty)
# GADFLY_FINDINGS_TOKEN optional bearer token for the gadfly-reports store
# GADFLY_CONSOLIDATE cross-model consensus comment: "auto" (default; on for >=2
# models), "1" force on, "0" force off (one comment per model)
# GADFLY_INLINE_REVIEW when consolidating, also post a COMMENT-state PR review with
# inline comments on changed lines (default on; "0" disables)
set -uo pipefail
DEFAULT_MODELS="qwen3-coder:480b-cloud,gpt-oss:120b-cloud"
# One model by default: the specialist suite already provides breadth, so a
# multi-model default would multiply cost (models × specialists × 2 passes).
DEFAULT_MODELS="qwen3-coder:480b-cloud"
TRIGGER_PHRASE="${GADFLY_TRIGGER_PHRASE:-@gadfly review}"
SCRIPTS_DIR="/app/scripts"
WORKDIR="${WORKDIR:-/tmp/gadfly}"
@@ -52,6 +68,29 @@ die() { log "ERROR: $*"; exit 1; }
API() { curl -fsS --connect-timeout 20 --max-time 30 -H "Authorization: token ${GITEA_TOKEN}" "$@"; }
# upsert_comment_body MARKER BODY — create or update (by leading MARKER) a single
# PR comment. Mirrors run.sh's per-model upsert; used for the consensus comment
# and the per-model fallback when consolidation is on.
upsert_comment_body() {
local marker="$1" body="$2" post_body existing_id="" page=1 cmts
post_body="$(jq -n --arg b "$body" '{body:$b}')"
while [ "$page" -le 10 ]; do
cmts="$(API "${GITEA_API}/issues/${PR}/comments?limit=50&page=${page}" || echo '[]')"
[ "$(echo "$cmts" | jq 'length')" = "0" ] && break
existing_id="$(echo "$cmts" | jq -r --arg m "$marker" \
'.[] | select(.body != null and (.body | startswith($m))) | .id' | head -n1)"
[ -n "$existing_id" ] && break
page=$((page+1))
done
if [ -n "$existing_id" ]; then
curl -sS --connect-timeout 20 --max-time 30 -X PATCH -H "Authorization: token ${GITEA_TOKEN}" \
-H "Content-Type: application/json" "${GITEA_API}/issues/comments/${existing_id}" -d "$post_body" >/dev/null
else
curl -sS --connect-timeout 20 --max-time 30 -X POST -H "Authorization: token ${GITEA_TOKEN}" \
-H "Content-Type: application/json" "${GITEA_API}/issues/${PR}/comments" -d "$post_body" >/dev/null
fi
}
# --- is the commenter allowed to trigger a re-review? ----------------------
actor_allowed() {
local actor="$1"
@@ -118,18 +157,270 @@ log "cloning ${REPO_PATH} @ ${BRANCH}"
git clone --depth=1 --branch "$BRANCH" "$CLONE_URL" "$REPO_DIR" 2>/dev/null \
|| die "clone of ${REPO_PATH}@${BRANCH} failed"
# --- review once per model -------------------------------------------------
MODELS="${OLLAMA_REVIEW_MODELS:-$DEFAULT_MODELS}"
log "models: ${MODELS}"
IFS=',' read -ra ARR <<< "$MODELS" || true
for raw in "${ARR[@]}"; do
m="$(echo "$raw" | tr -d '[:space:]')"
[ -z "$m" ] && continue
log "::: reviewing with ${m}"
PROVIDER=ollama \
MODEL="$m" \
GADFLY_BIN="/usr/local/bin/gadfly" \
GADFLY_REPO_DIR="$REPO_DIR" \
bash "${SCRIPTS_DIR}/run.sh" || log "model ${m} failed (continuing)"
# --- findings telemetry (optional) -----------------------------------------
# Plumb the run context to the binary (inherited through run.sh and the gadfly
# process env). The binary's emit is OFF unless GADFLY_FINDINGS_URL is set; when
# set it POSTs the run + its findings to a gadfly-reports store. GADFLY_FINDINGS_URL /
# GADFLY_FINDINGS_TOKEN come from the consumer's stub env and are re-exported so
# they reach the binary even if unset (empty => disabled).
export GADFLY_REPO="$REPO_PATH"
export GADFLY_PR="$PR"
export GADFLY_FINDINGS_URL="${GADFLY_FINDINGS_URL:-}"
export GADFLY_FINDINGS_TOKEN="${GADFLY_FINDINGS_TOKEN:-}"
# --- review once per model, with per-provider concurrency -------------------
# GADFLY_MODELS is the provider-agnostic name; OLLAMA_REVIEW_MODELS is a
# back-compat alias. GADFLY_PROVIDER / GADFLY_BASE_URL / GADFLY_API_KEY and any
# provider key envs (OPENAI_API_KEY, …) are inherited by run.sh and the binary.
#
# Concurrency: each PROVIDER is its own lane and lanes run in PARALLEL, so a fast
# cloud provider isn't stuck behind a slow local box. Within a lane, at most
# `cap` models run at once. cap = GADFLY_PROVIDER_CONCURRENCY's "provider=N"
# entry, else GADFLY_CONCURRENCY (default 1). A model's provider is the spec's
# first path segment ("m1pro/qwen3.6:35b-mlx" -> m1pro), or GADFLY_PROVIDER /
# ollama-cloud for a bare id. Default (cap 1) keeps a single-provider pool fully
# sequential, exactly as before.
MODELS="${GADFLY_MODELS:-${OLLAMA_REVIEW_MODELS:-$DEFAULT_MODELS}}"
DEFAULT_CONC="${GADFLY_CONCURRENCY:-1}"
# --- huge-PR downshift ------------------------------------------------------
# A very large diff is what burns the model budget: every review step re-sends
# it, multiplied across models × lenses × passes × steps (this is what nuked a
# whole Ollama Cloud block on one giant PR). entrypoint is the only process that
# spans the whole fleet, so the fleet-wide size decision lives here: size the PR
# diff ONCE, and above GADFLY_HUGE_DIFF_BYTES collapse to a single cheap model +
# a focused lens subset, fewer steps, no recheck, and a smaller embedded diff.
# A finished shallow review beats a budget-nuking one. All knobs override; set
# GADFLY_HUGE_DIFF_BYTES=0 to disable. Small PRs are never touched.
HUGE_PR=0
HUGE_DIFF_BYTES="${GADFLY_HUGE_DIFF_BYTES:-600000}"
if [ "$HUGE_DIFF_BYTES" -gt 0 ] 2>/dev/null; then
PR_DIFF_BYTES="$(API "${GITEA_API}/pulls/${PR}.diff" 2>/dev/null | wc -c | tr -d '[:space:]')"
[ -z "$PR_DIFF_BYTES" ] && PR_DIFF_BYTES=0
if [ "$PR_DIFF_BYTES" -gt "$HUGE_DIFF_BYTES" ] 2>/dev/null; then
HUGE_PR=1
log "huge PR: diff ${PR_DIFF_BYTES}B > ${HUGE_DIFF_BYTES}B — downshifting the fleet (advisory)"
MODELS="${GADFLY_HUGE_DIFF_MODELS:-${MODELS%%,*}}" # first model only by default
export GADFLY_SPECIALISTS="${GADFLY_HUGE_DIFF_SPECIALISTS:-security,correctness,error-handling}"
export GADFLY_MAX_STEPS="${GADFLY_HUGE_DIFF_MAX_STEPS:-12}"
export GADFLY_RECHECK_MAX_STEPS="${GADFLY_HUGE_DIFF_RECHECK_MAX_STEPS:-8}"
export GADFLY_RECHECK="${GADFLY_HUGE_DIFF_RECHECK:-0}" # skip recheck on huge PRs
# The Go-visible name directly (run.sh prefers GADFLY_MAX_DIFF_CHARS over its
# own MAX_DIFF_CHARS), so the cap is honored without relying on run.sh's alias.
export GADFLY_MAX_DIFF_CHARS="${GADFLY_HUGE_DIFF_MAX_DIFF_CHARS:-20000}"
# Surfaced on each posted comment so the shallower review is self-explaining.
export GADFLY_NOTICE="⚠️ Large PR (${PR_DIFF_BYTES} bytes): Gadfly downshifted to a focused, single-model review to stay within budget — coverage is intentionally shallower. Consider splitting the PR for a deeper review."
fi
fi
provider_of() { case "$1" in */*) echo "${1%%/*}";; *) echo "${GADFLY_PROVIDER:-ollama-cloud}";; esac; }
# Per-model status file path for the live board. The model id can contain '/'
# and ':' (e.g. m1/qwen3:14b), so sanitize to a flat filename; the JSON inside
# carries the real model/provider, so this just needs to be unique per model.
STATUS_DIR="${WORKDIR}/status"
status_file_for() { echo "${STATUS_DIR}/$(echo "$1" | tr -c '[:alnum:]._-' '_').json"; }
provider_cap() { # provider -> concurrency (override map "p=N,...", else default)
local p="$1" item k v
IFS=',' read -ra _caps <<< "${GADFLY_PROVIDER_CONCURRENCY:-}"
for item in "${_caps[@]}"; do
k="$(echo "${item%%=*}" | tr -d '[:space:]')"
v="$(echo "${item#*=}" | tr -d '[:space:]')"
if [ "$k" = "$p" ] && [ -n "$v" ]; then echo "$v"; return; fi
done
echo "$DEFAULT_CONC"
}
review_one() {
local sf="" ff=""
[ "${GADFLY_STATUS_BOARD:-1}" != "0" ] && sf="$(status_file_for "$1")"
[ "$CONSOLIDATE" = "1" ] && ff="$(findings_file_for "$1")"
PROVIDER=ollama MODEL="$1" GADFLY_BIN="/usr/local/bin/gadfly" GADFLY_REPO_DIR="$REPO_DIR" \
GADFLY_STATUS_FILE="$sf" GADFLY_FINDINGS_OUT="$ff" GADFLY_CONSOLIDATE="$CONSOLIDATE" \
bash "${SCRIPTS_DIR}/run.sh" || log "model $1 failed (continuing)"
# If the binary never wrote real status (run.sh skipped it: empty diff, no key,
# binary missing), the pre-seed stays {started:0, done:false} and the board
# would show this model "waiting to start" forever and never reach N/N. Mark
# such a never-started file done so the board can complete. The binary stamps a
# nonzero `started`, so that reliably distinguishes "ran" from "skipped".
if [ -n "$sf" ] && [ -f "$sf" ] && [ "$(jq -r '.started // 0' "$sf" 2>/dev/null)" = "0" ]; then
tmp="$(jq '.done = true' "$sf" 2>/dev/null)" && printf '%s' "$tmp" > "$sf"
fi
}
# Normalize the model list (trim, drop blanks) into MODEL_LIST.
IFS=',' read -ra _raw <<< "$MODELS" || true
MODEL_LIST=()
for raw in "${_raw[@]}"; do m="$(echo "$raw" | tr -d '[:space:]')"; [ -n "$m" ] && MODEL_LIST+=("$m"); done
# --- cross-model consolidation decision ------------------------------------
# With >=2 models, post ONE consensus comment (findings clustered + ranked by
# cross-model agreement) instead of N per-model walls of prose. Each model writes
# its findings to FINDINGS_DIR; a final pass (the binary in GADFLY_CONSOLIDATE_DIR
# mode) renders the consensus comment. GADFLY_CONSOLIDATE: "auto" (default; on for
# >=2 models), "1" force on, "0" force off (keep per-model comments).
FINDINGS_DIR="${WORKDIR}/findings"
CONSOLIDATE=0
case "${GADFLY_CONSOLIDATE:-auto}" in
1) CONSOLIDATE=1 ;;
0) CONSOLIDATE=0 ;;
*) [ "${#MODEL_LIST[@]}" -ge 2 ] && CONSOLIDATE=1 ;;
esac
# A model spec can contain '/' and ':' (e.g. claude-code/opus, qwen3:14b), so
# sanitize to a flat filename — but append a checksum of the raw spec so two
# specs that sanitize the same (foo:bar vs foo/bar -> foo_bar) don't collide onto
# one file and silently drop a model from the consensus.
findings_file_for() {
local safe sum
safe="$(echo "$1" | tr -c '[:alnum:]._-' '_')"
sum="$(printf '%s' "$1" | cksum | cut -d' ' -f1)"
echo "${FINDINGS_DIR}/${safe}-${sum}.json"
}
if [ "$CONSOLIDATE" = "1" ]; then
rm -rf "$FINDINGS_DIR"; mkdir -p "$FINDINGS_DIR"
log "consolidation ON: ${#MODEL_LIST[@]} models -> one consensus comment"
fi
# Distinct providers, in first-seen order (no associative arrays — portable).
PROVIDERS=""
for m in "${MODEL_LIST[@]}"; do
p="$(provider_of "$m")"
case " $PROVIDERS " in *" $p "*) ;; *) PROVIDERS="${PROVIDERS}${PROVIDERS:+ }$p" ;; esac
done
run_lane() { # $1=provider: run its models, at most `cap` at a time
local p="$1" cap inflight=0 m
cap="$(provider_cap "$p")"; [ "$cap" -ge 1 ] 2>/dev/null || cap=1
local mine=()
for m in "${MODEL_LIST[@]}"; do [ "$(provider_of "$m")" = "$p" ] && mine+=("$m"); done
log "lane ${p}: cap ${cap}; models: ${mine[*]}"
for m in "${mine[@]}"; do
review_one "$m" &
inflight=$((inflight+1))
if [ "$inflight" -ge "$cap" ]; then wait -n 2>/dev/null || wait; inflight=$((inflight-1)); fi
done
wait
}
# --- live status board (optional, default on) ------------------------------
# Each model process publishes per-lens progress to STATUS_DIR/<model>.json; a
# background renderer (status-board.sh) upserts ONE consolidated PR comment so
# progress across all models/lenses is visible at a glance — and a watcher can
# tell when the whole swarm is finished. Advisory/best-effort; the per-model
# findings still land in each model's own comment. Disable with
# GADFLY_STATUS_BOARD=0.
BOARD_PID=""
if [ "${GADFLY_STATUS_BOARD:-1}" != "0" ]; then
rm -rf "$STATUS_DIR"; mkdir -p "$STATUS_DIR"
# Pre-seed every model as queued so the board shows the full swarm from t=0,
# even models still waiting on their provider lane's concurrency cap. Each
# binary overwrites its own file with real per-lens detail once it starts.
for m in "${MODEL_LIST[@]}"; do
jq -n --arg model "$m" --arg provider "$(provider_of "$m")" \
'{model:$model, provider:$provider, started:0, updated:0, done:false, lenses:[]}' \
> "$(status_file_for "$m")" 2>/dev/null || true
done
GITEA_API="$GITEA_API" GITEA_TOKEN="$GITEA_TOKEN" PR="$PR" GADFLY_STATUS_DIR="$STATUS_DIR" \
bash "${SCRIPTS_DIR}/status-board.sh" &
BOARD_PID=$!
log "status board started (pid ${BOARD_PID})"
fi
# --- swarm-wide hard backstop ----------------------------------------------
# A wall-clock ceiling across the WHOLE fleet, so a pathological PR can never
# drain the usage block however the models behave. entrypoint is the only
# process spanning every model, so a single "never exceed X" guard lives here.
# On expiry it stops the review subtrees (the binary + run.sh); whatever partial
# findings were gathered are still posted and the job never fails (advisory).
# GADFLY_PR_BUDGET_SECS=0 (default) disables it.
KILLER_PID=""
rm -f "${WORKDIR}/.budget_killed" "${WORKDIR}/.disarmed" 2>/dev/null || true
if [ "${GADFLY_PR_BUDGET_SECS:-0}" -gt 0 ] 2>/dev/null; then
(
sleep "${GADFLY_PR_BUDGET_SECS}"
log "PR wall-clock budget (${GADFLY_PR_BUDGET_SECS}s) reached — stopping the review fleet (advisory; partial findings still posted)"
: > "${WORKDIR}/.budget_killed"
pkill -TERM -f '/usr/local/bin/gadfly' 2>/dev/null || true
pkill -TERM -f "${SCRIPTS_DIR}/run.sh" 2>/dev/null || true
sleep 5
# Guard the delayed SIGKILL on the disarm marker: once the lanes finished and
# the watchdog was disarmed, the consolidation gadfly pass runs next, and a
# name-based KILL here must NOT catch it.
[ -f "${WORKDIR}/.disarmed" ] || pkill -KILL -f '/usr/local/bin/gadfly' 2>/dev/null || true
) &
KILLER_PID=$!
log "PR budget watchdog armed (${GADFLY_PR_BUDGET_SECS}s, pid ${KILLER_PID})"
fi
log "providers: ${PROVIDERS:-none}"
# Each provider lane runs in parallel; cap is enforced within each lane. Track
# the lane PIDs so we wait ONLY for the review work — not the status board,
# which intentionally runs until we signal it below.
LANE_PIDS=()
for p in $PROVIDERS; do
run_lane "$p" &
LANE_PIDS+=("$!")
done
[ "${#LANE_PIDS[@]}" -gt 0 ] && wait "${LANE_PIDS[@]}"
# Reviews finished (or the watchdog killed them): disarm the watchdog so its
# delayed SIGKILL can't catch the consolidation pass that runs next. Drop the
# disarm marker FIRST so even a racing watchdog that already reached its KILL line
# skips it (the kill below also tears the watchdog subshell down during its sleep).
if [ -n "$KILLER_PID" ]; then
: > "${WORKDIR}/.disarmed"
kill "$KILLER_PID" 2>/dev/null || true
fi
# If the backstop fired, note it on the consensus comment (per-model comments
# were already posted during the run; a killed model surfaces as a failed lane).
if [ -f "${WORKDIR}/.budget_killed" ]; then
export GADFLY_NOTICE="${GADFLY_NOTICE:+${GADFLY_NOTICE} }⏱️ This review was stopped early by the per-PR time budget (GADFLY_PR_BUDGET_SECS); findings are partial."
fi
# Reviews are done: signal the board to render the final state once and exit.
if [ -n "$BOARD_PID" ]; then
touch "${STATUS_DIR}/.done" 2>/dev/null || true
wait "$BOARD_PID" 2>/dev/null || true
fi
# --- cross-model consensus comment -----------------------------------------
# Render ONE consensus comment from the per-model findings the swarm wrote. This
# is advisory and best-effort: if the consolidation pass produces nothing, fall
# back to posting each model's review as its own comment (the per-model comments
# were suppressed during the run), so a consolidation hiccup never loses output.
if [ "$CONSOLIDATE" = "1" ]; then
n_files="$(ls -1 "${FINDINGS_DIR}"/*.json 2>/dev/null | wc -l | tr -d '[:space:]')"
log "consolidating findings from ${n_files} model(s)"
# Fetch the PR diff so the consolidator can also post an inline PR review,
# anchoring findings to changed lines (GADFLY_DIFF_FILE). GITEA_API/GITEA_TOKEN/
# GADFLY_PR are already in the binary's environment. Best-effort: an empty diff
# just means no inline review.
DIFF_FILE="${WORKDIR}/pr.diff"
API "${GITEA_API}/pulls/${PR}.diff" > "$DIFF_FILE" 2>/dev/null || true
CONSENSUS="$(GADFLY_CONSOLIDATE_DIR="$FINDINGS_DIR" GADFLY_DIFF_FILE="$DIFF_FILE" \
/usr/local/bin/gadfly 2>"${WORKDIR}/consolidate.err" || true)"
if [ -n "$CONSENSUS" ]; then
NOTICE_BLOCK=""
[ -n "${GADFLY_NOTICE:-}" ] && NOTICE_BLOCK="> ${GADFLY_NOTICE}"$'\n\n'
BODY="$(printf '%s%s\n\n<sub>Automated adversarial review by Gadfly — consensus across the model swarm. Advisory only — does not block merge.</sub>' "$NOTICE_BLOCK" "$CONSENSUS")"
upsert_comment_body "<!-- gadfly-consensus -->" "$BODY"
log "consensus comment posted"
else
log "consolidation produced no output; falling back to per-model comments"
log "$(tail -c 500 "${WORKDIR}/consolidate.err" 2>/dev/null)"
for f in "${FINDINGS_DIR}"/*.json; do
[ -f "$f" ] || continue
m="$(jq -r '.model // ""' "$f" 2>/dev/null)"
[ -z "$m" ] && continue
prov="$(jq -r '.provider // ""' "$f" 2>/dev/null)"
md="$(jq -r '.markdown // ""' "$f" 2>/dev/null)"
marker="<!-- gadfly-review:ollama:${m} -->"
body="$(printf '%s\n### 🪰 Gadfly review — `%s` (%s)\n\n%s\n\n<sub>Automated adversarial review by Gadfly. Advisory only — does not block merge.</sub>' \
"$marker" "$m" "$prov" "$md")"
upsert_comment_body "$marker" "$body"
done
fi
fi
log "done"
+34
View File
@@ -0,0 +1,34 @@
# Optional per-repo Gadfly config. Place at the repo ROOT as `.gadfly.yml`
# (or point GADFLY_CONFIG at it). It's read from the PR's checked-out tree, so
# it's version-controlled and reviewed like any other file.
#
# Precedence for specialist DEFINITIONS: built-ins < this file < GADFLY_SPECIALIST_* env.
# Precedence for SELECTION: GADFLY_SPECIALISTS env > this file's `specialists:` > default suite.
# Which specialists run. Omit this key to use the default suite
# (security, correctness, maintainability, performance, error-handling).
# Built-in opt-ins you can add: tests, docs, conventions, improvements. Or "all".
specialists:
- security
- correctness
- maintainability
- tests
- migrations # a custom one, defined below
# Add new specialists, or override a built-in by reusing its name (e.g. give
# `security` a repo-specific focus). `focus` is appended to the base reviewer
# prompt as that lens's instruction.
define:
- name: migrations
title: "🗃️ DB migrations"
focus: >
Review database schema migrations for destructive operations (DROP/ALTER
that loses data), missing or unindexed foreign keys, non-idempotent steps,
and changes that would lock a large table on deploy. Check that any new
column added to a domain struct is wired through every storage layer.
- name: security
title: "🔒 Security (house rules)"
focus: >
In addition to the usual security review, this repo requires: all web
routes use the auth middleware, no secrets in logs, and all external HTTP
calls set a timeout. Flag any violation.
+27
View File
@@ -0,0 +1,27 @@
# Example consumer workflows
Each file here is a complete, copy-paste **stub workflow**. Pick the one that matches your
setup, copy it to `.gitea/workflows/adversarial-review.yml` in the repo you want reviewed, and
set the secrets/vars it references. Gadfly is advisory only — it never blocks a merge.
| File | Backend | Needs |
|------|---------|-------|
| [`reusable.yml`](reusable.yml) | **slimmest stub** — calls Gadfly's reusable workflow and inherits its **default swarm** (3 cloud + Claude Code, 5-lens suite), forwarding only the secrets it needs (least privilege, not `secrets: inherit`); the stub keeps a cloud-only `models:` override so it runs with just the Ollama key (drop it + add the Claude token to get the full default) | secret `OLLAMA_CLOUD_API_KEY` |
| [`adversarial-review.yml`](adversarial-review.yml) | **Ollama Cloud** (default) + inline notes for every provider; full self-contained stub | secret `OLLAMA_CLOUD_API_KEY` |
| [`local-ollama.yml`](local-ollama.yml) | a **local/LAN Ollama** daemon | nothing (or `GADFLY_BASE_URL` for a remote host) |
| [`openai-compatible.yml`](openai-compatible.yml) | any **OpenAI-compatible** endpoint (local Ollama `/v1`, gateway, vLLM, OpenRouter…) | `GADFLY_BASE_URL` (+ a key for most gateways) |
| [`endpoint-aliases.yml`](endpoint-aliases.yml) | **several named backends** at once (one comment each) | repo vars `GADFLY_ENDPOINT_<NAME>` |
| [`claude-code.yml`](claude-code.yml) | the bundled **Claude Code CLI** engine (`claude-code/<model>`) | secret `CLAUDE_CODE_OAUTH_TOKEN` (or `ANTHROPIC_API_KEY`) |
| [`.gadfly.yml`](.gadfly.yml) | **per-repo specialist config** (not a workflow — goes at your repo root) | — |
Common to all:
- **Triggers:** new/reopened/ready non-draft PR (auto), `@gadfly review` comment (allowed users),
or manual `workflow_dispatch` with a `pr_number`.
- `GITEA_TOKEN` is provided automatically; comments post as `gitea-actions`.
- Tested backends are the **Ollama** ones; OpenAI/Anthropic/Google are wired via majordomo but
untested. See the repo [README](../README.md#models--providers) for the full config reference
and the honest tested/untested status.
> **Gitea note:** repo `vars`/`secrets` are not auto-exposed as env — anything you reference
> via `${{ vars.X }}` / `${{ secrets.X }}` must appear in the step's `env:` block (already wired
> in these examples).
+59
View File
@@ -33,6 +33,13 @@ concurrency:
jobs:
review:
# Security: only trusted users may trigger a secret-bearing run via a PR
# comment (pull_request + workflow_dispatch are already trusted). Replace the
# username(s) below with your maintainers — keep them in sync with
# GADFLY_ALLOWED_USERS (the in-container belt-and-suspenders check).
if: >-
github.event_name != 'issue_comment'
|| github.actor == 'your-username'
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
@@ -43,6 +50,58 @@ jobs:
OLLAMA_CLOUD_API_KEY: ${{ secrets.OLLAMA_CLOUD_API_KEY }}
OLLAMA_REVIEW_MODELS: ${{ vars.OLLAMA_REVIEW_MODELS }}
GADFLY_ALLOWED_USERS: ${{ vars.GADFLY_ALLOWED_USERS }}
# Specialist suite (optional). Empty = default suite
# (security,correctness,maintainability,performance,error-handling).
# csv to choose; "all" for everything; or define custom ones via a repo
# .gadfly.yml / GADFLY_SPECIALIST_<NAME>. See README "Specialists".
GADFLY_SPECIALISTS: ${{ vars.GADFLY_SPECIALISTS }}
# Lens fan-out (optional; default 1 = lenses run sequentially within a
# model). Raise it to run a model's lenses concurrently so each model
# posts its comment sooner. Total in-flight requests = (models at once)
# × (lenses at once), so to fan out without oversubscribing a backend,
# keep its model cap low and raise its lens cap. Per-provider configurable
# via GADFLY_PROVIDER_LENS_CONCURRENCY (same lanes as the model map):
# GADFLY_PROVIDER_CONCURRENCY: "ollama-cloud=1,m1=1"
# GADFLY_PROVIDER_LENS_CONCURRENCY: "ollama-cloud=3,m1=1"
# GADFLY_LENS_CONCURRENCY: ${{ vars.GADFLY_LENS_CONCURRENCY }}
# GADFLY_PROVIDER_LENS_CONCURRENCY: ${{ vars.GADFLY_PROVIDER_LENS_CONCURRENCY }}
# Live status board (optional; ON by default): one consolidated comment
# showing every model's per-lens progress as it runs. Disable with
# GADFLY_STATUS_BOARD=0; tune the refresh with GADFLY_STATUS_POLL_SECS.
# GADFLY_STATUS_BOARD: ${{ vars.GADFLY_STATUS_BOARD }}
# --- Models & providers (optional; default = Ollama Cloud) ----------
# Gadfly is majordomo-powered, so it can target other backends. Set a
# provider for bare model ids; point at a different endpoint with a
# base URL; supply a key (or the provider's standard env var). Examples:
#
# Local Ollama daemon (no key):
# GADFLY_PROVIDER: ollama
# GADFLY_MODELS: qwen2.5-coder:7b
# # GADFLY_BASE_URL: http://my-ollama-host:11434 # if not localhost
#
# OpenAI-compatible endpoint (incl. local Ollama's /v1):
# GADFLY_PROVIDER: openai
# GADFLY_BASE_URL: http://localhost:11434/v1
# GADFLY_MODELS: qwen2.5-coder:7b
#
# OpenAI / Anthropic / Google (supported via majordomo, UNTESTED — see README):
# GADFLY_PROVIDER: openai # then set OPENAI_API_KEY below
# OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
#
# Named endpoint aliases (reference as name/model in GADFLY_MODELS).
# vars/secrets aren't auto-exposed, so map each one explicitly:
# GADFLY_ENDPOINT_BIGBOX: ${{ vars.GADFLY_ENDPOINT_BIGBOX }} # "ollama|http://192.168.1.50:11434"
# GADFLY_MODELS: bigbox/qwen2.5-coder:7b
GADFLY_PROVIDER: ${{ vars.GADFLY_PROVIDER }}
GADFLY_BASE_URL: ${{ vars.GADFLY_BASE_URL }}
GADFLY_MODELS: ${{ vars.GADFLY_MODELS }}
# --- Findings telemetry (optional; OFF by default) ------------------
# Set GADFLY_FINDINGS_URL to a gadfly-reports store base URL to POST each run +
# its findings for model-quality tracking. Advisory only: failures are
# logged to stderr and never affect the review. Add a bearer token if
# the store requires auth. (GADFLY_REPO / GADFLY_PR are derived for you.)
# GADFLY_FINDINGS_URL: ${{ vars.GADFLY_FINDINGS_URL }}
# GADFLY_FINDINGS_TOKEN: ${{ secrets.GADFLY_FINDINGS_TOKEN }}
EVENT_NAME: ${{ github.event_name }}
PR: ${{ github.event.pull_request.number || github.event.issue.number || github.event.inputs.pr_number }}
PR_BRANCH: ${{ github.head_ref }}
+78
View File
@@ -0,0 +1,78 @@
# Gadfly reviewing via the Claude Code CLI engine.
# Copy to .gitea/workflows/adversarial-review.yml in your repo.
#
# Instead of a majordomo model, each lens shells out to the bundled `claude` CLI
# inside the checked-out repo (it uses its own Read/Grep/Glob tools to verify
# findings), then Gadfly runs its usual verdict + recheck + consolidate pipeline.
#
# Auth: a Pro/Max subscription token from `claude setup-token` (no --bare),
# stored as the CLAUDE_CODE_OAUTH_TOKEN secret. Falls back to ANTHROPIC_API_KEY
# if you'd rather pay per-token — set only ONE.
#
# Heads-up: this engine is wired but not yet validated end-to-end here, and using
# subscription auth in automated CI is a gray area in Anthropic's terms — read
# the README's "Claude Code engine" note before relying on it.
name: Adversarial Review (Gadfly)
on:
pull_request:
types: [opened, reopened, ready_for_review]
issue_comment:
types: [created]
workflow_dispatch:
inputs:
pr_number: { description: "PR number to review", required: true }
permissions:
contents: read
issues: write
pull-requests: write
concurrency:
group: gadfly-${{ github.event.issue.number || github.event.pull_request.number || github.event.inputs.pr_number }}
cancel-in-progress: true
jobs:
review:
# Security: only trusted users may trigger a secret-bearing run via a PR
# comment. Replace the username(s) below with your maintainers — keep them in
# sync with GADFLY_ALLOWED_USERS (the in-container belt-and-suspenders check).
if: >-
github.event_name != 'issue_comment'
|| github.actor == 'your-username'
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: docker://gitea.stevedudenhoeffer.com/steve/gadfly:latest
env:
GITEA_API: ${{ github.server_url }}/api/v1/repos/${{ github.repository }}
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
# --- Claude Code engine ---
# Pro/Max subscription token (preferred). Or set ANTHROPIC_API_KEY
# instead for per-token billing — but never both.
CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
# ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
# bare "claude-code" uses the CLI default model; "claude-code/<model>"
# sets --model (sonnet/opus/haiku, or a full id). One comment per entry.
GADFLY_MODELS: "claude-code/sonnet"
# Optional CLI tuning (defaults are read-only-safe):
# GADFLY_CLAUDE_PERMISSION_MODE: plan # read-only; never edits
# GADFLY_CLAUDE_ALLOWED_TOOLS: "Read,Grep,Glob"
# GADFLY_CLAUDE_EXTRA_ARGS: "--max-turns 30"
# Alternate backend (EXAMPLE ONLY, not validated): point Claude Code at
# an Anthropic-API-compatible proxy (e.g. claude-code-router / LiteLLM in
# front of Ollama) to run Ollama models THROUGH the CC harness. The
# subprocess env forwards ANTHROPIC_*, so just set these instead of the
# token above. Tool-use support depends on the proxy/backend.
# ANTHROPIC_BASE_URL: ${{ vars.ANTHROPIC_BASE_URL }}
# ANTHROPIC_AUTH_TOKEN: ${{ secrets.ANTHROPIC_AUTH_TOKEN }}
GADFLY_ALLOWED_USERS: "your-username"
# --- event context (leave as-is) ---
EVENT_NAME: ${{ github.event_name }}
PR: ${{ github.event.pull_request.number || github.event.issue.number || github.event.inputs.pr_number }}
PR_BRANCH: ${{ github.head_ref }}
IS_DRAFT: ${{ github.event.pull_request.draft }}
COMMENT_BODY: ${{ github.event.comment.body }}
COMMENT_ID: ${{ github.event.comment.id }}
ACTOR: ${{ github.actor }}
+67
View File
@@ -0,0 +1,67 @@
# Gadfly with named ENDPOINT ALIASES — review with several backends at once,
# each posting its own comment. Copy to .gitea/workflows/adversarial-review.yml.
#
# GADFLY_ENDPOINT_<NAME>="<provider>|<base-url>[|<key>]" registers a provider you
# can then reference as "<name>/<model>" (NAME lowercases: BIGBOX -> bigbox).
# The base URL is used verbatim, so plaintext http LAN endpoints work.
#
# provider is ollama / foreman / openai / anthropic / google. "foreman" targets a
# foreman queue daemon (https://gitea.stevedudenhoeffer.com/steve/foreman) — native
# Ollama on the wire, so just give it the daemon's URL (and optional bearer token).
#
# Gitea note: vars/secrets aren't auto-exposed as env, so map each alias here.
# Suggested repo vars (and a secret when the value carries a token):
# GADFLY_ENDPOINT_BIGBOX = ollama|http://192.168.1.50:11434
# GADFLY_ENDPOINT_GPU = openai|http://gpu.lan:8000/v1
# GADFLY_ENDPOINT_M1 = foreman|http://foreman-m1:8080|<token> (use a secret)
name: Adversarial Review (Gadfly)
on:
pull_request:
types: [opened, reopened, ready_for_review]
issue_comment:
types: [created]
workflow_dispatch:
inputs:
pr_number: { description: "PR number to review", required: true }
permissions:
contents: read
issues: write
pull-requests: write
concurrency:
group: gadfly-${{ github.event.issue.number || github.event.pull_request.number || github.event.inputs.pr_number }}
cancel-in-progress: true
jobs:
review:
# Security: only trusted users may trigger a secret-bearing run via a PR
# comment (pull_request + workflow_dispatch are already trusted). Replace the
# username(s) below with your maintainers — keep them in sync with
# GADFLY_ALLOWED_USERS (the in-container belt-and-suspenders check).
if: >-
github.event_name != 'issue_comment'
|| github.actor == 'your-username'
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: docker://gitea.stevedudenhoeffer.com/steve/gadfly:latest
env:
GITEA_API: ${{ github.server_url }}/api/v1/repos/${{ github.repository }}
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
# --- named endpoints (mapped from repo vars) ---
GADFLY_ENDPOINT_BIGBOX: ${{ vars.GADFLY_ENDPOINT_BIGBOX }} # "ollama|http://192.168.1.50:11434"
GADFLY_ENDPOINT_GPU: ${{ vars.GADFLY_ENDPOINT_GPU }} # "openai|http://gpu.lan:8000/v1"
GADFLY_ENDPOINT_M1: ${{ secrets.GADFLY_ENDPOINT_M1 }} # "foreman|http://foreman-m1:8080|<token>"
# one reviewer (one comment) per model, across the aliased endpoints:
GADFLY_MODELS: "bigbox/qwen2.5-coder:7b,gpu/llama3.1,m1/qwen3:14b"
# --- event context (leave as-is) ---
EVENT_NAME: ${{ github.event_name }}
PR: ${{ github.event.pull_request.number || github.event.issue.number || github.event.inputs.pr_number }}
PR_BRANCH: ${{ github.head_ref }}
IS_DRAFT: ${{ github.event.pull_request.draft }}
COMMENT_BODY: ${{ github.event.comment.body }}
COMMENT_ID: ${{ github.event.comment.id }}
ACTOR: ${{ github.actor }}
+56
View File
@@ -0,0 +1,56 @@
# Gadfly using a LOCAL Ollama daemon (no API key needed).
# Copy to .gitea/workflows/adversarial-review.yml in your repo.
#
# The runner must be able to reach the Ollama host. For localhost on the runner,
# leave GADFLY_BASE_URL unset; for a LAN box set it to http://<host>:11434.
#
# Pick a model you've pulled into Ollama (e.g. `ollama pull qwen2.5-coder:7b`).
name: Adversarial Review (Gadfly)
on:
pull_request:
types: [opened, reopened, ready_for_review]
issue_comment:
types: [created]
workflow_dispatch:
inputs:
pr_number: { description: "PR number to review", required: true }
permissions:
contents: read
issues: write
pull-requests: write
concurrency:
group: gadfly-${{ github.event.issue.number || github.event.pull_request.number || github.event.inputs.pr_number }}
cancel-in-progress: true
jobs:
review:
# Security: only trusted users may trigger a secret-bearing run via a PR
# comment (pull_request + workflow_dispatch are already trusted). Replace the
# username(s) below with your maintainers — keep them in sync with
# GADFLY_ALLOWED_USERS (the in-container belt-and-suspenders check).
if: >-
github.event_name != 'issue_comment'
|| github.actor == 'your-username'
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: docker://gitea.stevedudenhoeffer.com/steve/gadfly:latest
env:
GITEA_API: ${{ github.server_url }}/api/v1/repos/${{ github.repository }}
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
# --- local Ollama ---
GADFLY_PROVIDER: ollama
GADFLY_MODELS: qwen2.5-coder:7b
# GADFLY_BASE_URL: http://192.168.1.50:11434 # uncomment for a remote/LAN daemon
# --- event context (leave as-is) ---
EVENT_NAME: ${{ github.event_name }}
PR: ${{ github.event.pull_request.number || github.event.issue.number || github.event.inputs.pr_number }}
PR_BRANCH: ${{ github.head_ref }}
IS_DRAFT: ${{ github.event.pull_request.draft }}
COMMENT_BODY: ${{ github.event.comment.body }}
COMMENT_ID: ${{ github.event.comment.id }}
ACTOR: ${{ github.actor }}
+59
View File
@@ -0,0 +1,59 @@
# Gadfly against an OpenAI-COMPATIBLE endpoint.
# Copy to .gitea/workflows/adversarial-review.yml in your repo.
#
# Works for: a local Ollama's OpenAI bridge (http://localhost:11434/v1), an
# in-house gateway, OpenRouter, vLLM, LM Studio, etc. This is the same code path
# the real OpenAI API uses, so it's a free way to exercise the OpenAI provider.
#
# Set GADFLY_API_KEY (or OPENAI_API_KEY) — Ollama ignores it, but most gateways
# require some value.
name: Adversarial Review (Gadfly)
on:
pull_request:
types: [opened, reopened, ready_for_review]
issue_comment:
types: [created]
workflow_dispatch:
inputs:
pr_number: { description: "PR number to review", required: true }
permissions:
contents: read
issues: write
pull-requests: write
concurrency:
group: gadfly-${{ github.event.issue.number || github.event.pull_request.number || github.event.inputs.pr_number }}
cancel-in-progress: true
jobs:
review:
# Security: only trusted users may trigger a secret-bearing run via a PR
# comment (pull_request + workflow_dispatch are already trusted). Replace the
# username(s) below with your maintainers — keep them in sync with
# GADFLY_ALLOWED_USERS (the in-container belt-and-suspenders check).
if: >-
github.event_name != 'issue_comment'
|| github.actor == 'your-username'
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: docker://gitea.stevedudenhoeffer.com/steve/gadfly:latest
env:
GITEA_API: ${{ github.server_url }}/api/v1/repos/${{ github.repository }}
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
# --- OpenAI-compatible endpoint ---
GADFLY_PROVIDER: openai
GADFLY_BASE_URL: http://localhost:11434/v1 # e.g. local Ollama /v1, or your gateway
GADFLY_API_KEY: ${{ secrets.OPENAI_API_KEY }} # any non-empty value for Ollama
GADFLY_MODELS: qwen2.5-coder:7b
# --- event context (leave as-is) ---
EVENT_NAME: ${{ github.event_name }}
PR: ${{ github.event.pull_request.number || github.event.issue.number || github.event.inputs.pr_number }}
PR_BRANCH: ${{ github.head_ref }}
IS_DRAFT: ${{ github.event.pull_request.draft }}
COMMENT_BODY: ${{ github.event.comment.body }}
COMMENT_ID: ${{ github.event.comment.id }}
ACTOR: ${{ github.actor }}
+71
View File
@@ -0,0 +1,71 @@
# Gadfly — SLIM consumer stub via the reusable workflow.
# Copy to .gitea/workflows/adversarial-review.yml in your repo.
#
# This is the shortest way to subscribe: it calls Gadfly's centralized reusable
# workflow, which holds the image pin + all the env plumbing. You only declare
# the triggers, the comment-trigger actor gate, and any overrides you want.
#
# The reusable ships a DEFAULT swarm: 3 cloud models + the Claude Code engine
# (sonnet/opus/opus:max), 5-lens suite. That default needs BOTH
# OLLAMA_CLOUD_API_KEY and CLAUDE_CODE_OAUTH_TOKEN. This example overrides
# `models:` to a cloud-only set so it works with just OLLAMA_CLOUD_API_KEY —
# delete that override (and forward the Claude token) to inherit the full default.
#
# Forward ONLY the secrets the reviewer uses (least privilege) — see the
# `secrets:` block below. GITEA_TOKEN is automatic. `secrets: inherit` also works
# but hands the reusable EVERY secret in your repo (registry/deploy/db creds the
# review never touches), so prefer the explicit form. Pin to an immutable
# @<sha>: long-lived act_runners CACHE the reusable by ref, so a moved tag (@v1)
# or @main is often not re-fetched and silently runs a stale copy. Bump the @<sha>
# to adopt a structural change; routine swarm tuning rides owner variables (see
# the gadfly README "Central config via variables") with no re-pin needed.
#
# For custom named endpoints (GADFLY_ENDPOINT_<NAME>) or a provider the reusable
# doesn't map, use the full stub in adversarial-review.yml instead.
name: Adversarial Review (Gadfly)
on:
pull_request:
types: [opened, reopened, ready_for_review]
issue_comment:
types: [created]
workflow_dispatch:
inputs:
pr_number: { description: "PR number to review", required: true }
permissions:
contents: read
issues: write
pull-requests: write
concurrency:
group: gadfly-${{ github.event.issue.number || github.event.pull_request.number || github.event.inputs.pr_number }}
cancel-in-progress: true
jobs:
review:
# Only let your maintainers re-trigger via a PR comment (keep in sync with
# the allowed_users override below).
if: >-
github.event_name != 'issue_comment'
|| (github.event.issue.pull_request && github.actor == 'your-username')
# Pin to an immutable @<sha> (runners cache the ref, so @v1/@main can run
# stale). Bump it for structural changes; tune the swarm via owner variables.
uses: steve/gadfly/.gitea/workflows/review-reusable.yml@v1
# Forward ONLY what the reviewer needs. Add provider keys you use
# (ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_API_KEY, GADFLY_API_KEY) and/or
# GADFLY_ENDPOINT_M1/M5; drop the findings ones if you don't run telemetry.
secrets:
OLLAMA_CLOUD_API_KEY: ${{ secrets.OLLAMA_CLOUD_API_KEY }}
# CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
# GADFLY_FINDINGS_URL: ${{ secrets.GADFLY_FINDINGS_URL }}
# GADFLY_FINDINGS_TOKEN: ${{ secrets.GADFLY_FINDINGS_TOKEN }}
with:
# Cloud-only override so this works with just OLLAMA_CLOUD_API_KEY. Delete
# this line (and forward CLAUDE_CODE_OAUTH_TOKEN above) to inherit the full
# default swarm (3 cloud + Claude Code sonnet/opus/opus:max, 5 lenses).
models: "minimax-m3:cloud,glm-5.2:cloud,deepseek-v4-pro:cloud"
# Other inputs inherit the default (5-lens suite, concurrency, 90-min cap);
# override any of them here (specialists, provider, base_url, timeout_secs…).
allowed_users: "your-username"
+35 -1
View File
@@ -2,4 +2,38 @@ module gitea.stevedudenhoeffer.com/steve/gadfly
go 1.26.2
require gitea.stevedudenhoeffer.com/steve/majordomo v0.0.0-20260610113006-0147a79d187b
require (
gitea.stevedudenhoeffer.com/steve/executus v0.1.4
gitea.stevedudenhoeffer.com/steve/majordomo v0.0.0-20260627225659-aa25b2c33462
gopkg.in/yaml.v3 v3.0.1
)
require (
cloud.google.com/go v0.123.0 // indirect
cloud.google.com/go/auth v0.20.0 // indirect
cloud.google.com/go/compute/metadata v0.9.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/felixge/httpsnoop v1.1.0 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/google/go-cmp v0.7.0 // indirect
github.com/google/s2a-go v0.1.9 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.17 // indirect
github.com/googleapis/gax-go/v2 v2.22.0 // indirect
github.com/gorilla/websocket v1.5.3 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.69.0 // indirect
go.opentelemetry.io/otel v1.44.0 // indirect
go.opentelemetry.io/otel/metric v1.44.0 // indirect
go.opentelemetry.io/otel/trace v1.44.0 // indirect
golang.org/x/crypto v0.53.0 // indirect
golang.org/x/net v0.56.0 // indirect
golang.org/x/sys v0.46.0 // indirect
golang.org/x/text v0.38.0 // indirect
google.golang.org/api v0.286.0 // indirect
google.golang.org/genai v1.62.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260622175928-b703f567277d // indirect
google.golang.org/grpc v1.81.1 // indirect
google.golang.org/protobuf v1.36.11 // indirect
)
+90 -2
View File
@@ -1,2 +1,90 @@
gitea.stevedudenhoeffer.com/steve/majordomo v0.0.0-20260610113006-0147a79d187b h1:/pglCqQW02kV2p9tKyQpIJoXZK2p7LKLeDCZL/V26MM=
gitea.stevedudenhoeffer.com/steve/majordomo v0.0.0-20260610113006-0147a79d187b/go.mod h1:UZLveG17SmENt4sne2RSLIbioix30RZbRIQUzBAnOyY=
cloud.google.com/go v0.123.0 h1:2NAUJwPR47q+E35uaJeYoNhuNEM9kM8SjgRgdeOJUSE=
cloud.google.com/go v0.123.0/go.mod h1:xBoMV08QcqUGuPW65Qfm1o9Y4zKZBpGS+7bImXLTAZU=
cloud.google.com/go/auth v0.20.0 h1:kXTssoVb4azsVDoUiF8KvxAqrsQcQtB53DcSgta74CA=
cloud.google.com/go/auth v0.20.0/go.mod h1:942/yi/itH1SsmpyrbnTMDgGfdy2BUqIKyd0cyYLc5Q=
cloud.google.com/go/compute/metadata v0.9.0 h1:pDUj4QMoPejqq20dK0Pg2N4yG9zIkYGdBtwLoEkH9Zs=
cloud.google.com/go/compute/metadata v0.9.0/go.mod h1:E0bWwX5wTnLPedCKqk3pJmVgCBSM6qQI1yTBdEb3C10=
gitea.stevedudenhoeffer.com/steve/executus v0.1.4 h1:4F99uCV3OVaE9ITFp0FjPiYxLUQO+WpE+wU2HCnpXNM=
gitea.stevedudenhoeffer.com/steve/executus v0.1.4/go.mod h1:WQP/lH+meU06OSNF0TQO/wQLcJCrMwpi0EMj5vSpVtk=
gitea.stevedudenhoeffer.com/steve/majordomo v0.0.0-20260627225659-aa25b2c33462 h1:1crjE1YkWHLZ91tUDOxN/Y5cuOnJ56e0U9UADoFfEPY=
gitea.stevedudenhoeffer.com/steve/majordomo v0.0.0-20260627225659-aa25b2c33462/go.mod h1:UZLveG17SmENt4sne2RSLIbioix30RZbRIQUzBAnOyY=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/felixge/httpsnoop v1.1.0 h1:3YtUj32ZZkqZtt3sZZsClsymw/QDuVfpNhoA31zeORc=
github.com/felixge/httpsnoop v1.1.0/go.mod h1:Zqxgdd+1Rkcz8euOqdr7lqgCRJztwr5hp9vDSi5UZCE=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/s2a-go v0.1.9 h1:LGD7gtMgezd8a/Xak7mEWL0PjoTQFvpRudN895yqKW0=
github.com/google/s2a-go v0.1.9/go.mod h1:YA0Ei2ZQL3acow2O62kdp9UlnvMmU7kA6Eutn0dXayM=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/googleapis/enterprise-certificate-proxy v0.3.17 h1:73NfMHdiqo9JFU9+7a5ExpVa10/R29pXfZIaW559nrg=
github.com/googleapis/enterprise-certificate-proxy v0.3.17/go.mod h1:rSEsBUemEBZEexP2y6jPp16LUmUbjmSbcPMQizR0o4k=
github.com/googleapis/gax-go/v2 v2.22.0 h1:PjIWBpgGIVKGoCXuiCoP64altEJCj3/Ei+kSU5vlZD4=
github.com/googleapis/gax-go/v2 v2.22.0/go.mod h1:irWBbALSr0Sk3qlqb9SyJ1h68WjgeFuiOzI4Rqw5+aY=
github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.69.0 h1:8tvICD4vSTOOsNrsI4Ljf6C+6UKvpTEH5XY3JMoyPoo=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.69.0/go.mod h1:z9+yiacE0IHRqM4qFfkbt/JYlmYXgss8GY/jXoNuPJI=
go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU=
go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc=
go.opentelemetry.io/otel/metric v1.44.0 h1:1w0gILTcHdr3YI+ixLyjemwrVnsMURbTZFrSYCdDdmc=
go.opentelemetry.io/otel/metric v1.44.0/go.mod h1:8O7hanEPBNgEMmybD3s2VBKcgWOCsA6tzHBPODAiquo=
go.opentelemetry.io/otel/sdk v1.44.0 h1:nHYwb9lK+fJPU/dnT6s7W7Z8itMWyqrnVfbheVYrZ58=
go.opentelemetry.io/otel/sdk v1.44.0/go.mod h1:Osuydd3Se74nqjAKxid74N5eC+jfEqfTegHRnq58oK0=
go.opentelemetry.io/otel/sdk/metric v1.44.0 h1:3LlKgI+VjbVsjNRFZJZAJ30WjXC5VkNRks6si09iEfI=
go.opentelemetry.io/otel/sdk/metric v1.44.0/go.mod h1:5B5pMARnXxKhltooO4xUuCBorl65a4EpnTalObqOigA=
go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk=
go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE=
golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto=
golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio=
golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw=
golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/text v0.38.0 h1:sXmwo9DwP3OK9EZ7PqAdaooSGozfl/3a6/xJcbzPRhE=
golang.org/x/text v0.38.0/go.mod h1:YXZt3QhHUKYT53r2lLKFIVi6Ao1jdzrTR/KQ09qyxF4=
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
google.golang.org/api v0.286.0 h1:TdTXMvzYKnWV1/lPbCdbXRqBrkDqjPto22H2xeZZ8LI=
google.golang.org/api v0.286.0/go.mod h1:NlOlUIr8MPoIhT9Bb/oUnRuHbJOLwxb6JSYJM8Yz+jQ=
google.golang.org/genai v1.62.0 h1:PaBju84orf4Vbcc6OfHe4vxhxhjwulKTgOpEc3iIc00=
google.golang.org/genai v1.62.0/go.mod h1:mDdPDFXo1Ats7f1WXVyZgWb/CkMzFWTWJruIMy7hGIU=
google.golang.org/genproto v0.0.0-20260319201613-d00831a3d3e7 h1:XzmzkmB14QhVhgnawEVsOn6OFsnpyxNPRY9QV01dNB0=
google.golang.org/genproto v0.0.0-20260319201613-d00831a3d3e7/go.mod h1:L43LFes82YgSonw6iTXTxXUX1OlULt4AQtkik4ULL/I=
google.golang.org/genproto/googleapis/api v0.0.0-20260319201613-d00831a3d3e7 h1:41r6JMbpzBMen0R/4TZeeAmGXSJC7DftGINUodzTkPI=
google.golang.org/genproto/googleapis/api v0.0.0-20260319201613-d00831a3d3e7/go.mod h1:EIQZ5bFCfRQDV4MhRle7+OgjNtZ6P1PiZBgAKuxXu/Y=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260622175928-b703f567277d h1:mpAgMyM9vQHxycBlDq50y1VHpfSfVwzXvrQKtYbXuUY=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260622175928-b703f567277d/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
google.golang.org/grpc v1.81.1 h1:VnnIIZ88UzOOKLukQi+ImGz8O1Wdp8nAGGnvOfEIWQQ=
google.golang.org/grpc v1.81.1/go.mod h1:xGH9GfzOyMTGIOXBJmXt+BX/V0kcdQbdcuwQ/zNw42I=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+104 -34
View File
@@ -23,8 +23,16 @@
# GADFLY_REPO_DIR (checked-out repo; default: this script's repo)
# antigravity: `agy` on PATH with credentials already seeded (~/.gemini)
#
# claude-code engine: when MODEL is "claude-code" or "claude-code/<model>" the
# binary shells out to the bundled `claude` CLI instead of a majordomo model.
# Its auth (CLAUDE_CODE_OAUTH_TOKEN, else ANTHROPIC_API_KEY) and GADFLY_CLAUDE_*
# tuning are read straight from the inherited environment — same as the other
# provider keys (OPENAI_API_KEY, …) — so no extra wiring is needed here.
#
# Optional:
# MAX_DIFF_CHARS diff truncation cap for the prompt (default 60000)
# MAX_DIFF_CHARS diff truncation cap for the prompt (default 60000)
# GADFLY_STATUS_FILE per-model JSON path for the live status board (set by
# entrypoint.sh; empty/unset disables status publishing)
#
# This script is advisory: it never fails the job for review content. It exits
# non-zero only on a usage/configuration error.
@@ -42,6 +50,21 @@ MAX_DIFF_CHARS="${MAX_DIFF_CHARS:-60000}"
MARKER="<!-- gadfly-review:${PROVIDER}:${MODEL} -->"
say() { echo "[gadfly-review:${PROVIDER}:${MODEL}] $*" >&2; }
# When the swarm is consolidating (GADFLY_CONSOLIDATE=1, set by entrypoint.sh for
# a multi-model run), this model does NOT post its own comment — it writes its
# findings to GADFLY_FINDINGS_OUT and a single cross-model consensus comment is
# posted after the whole swarm finishes. Live progress still shows on the status
# board. Default 0 (post a per-model comment, the standalone behavior).
CONSOLIDATE="${GADFLY_CONSOLIDATE:-0}"
# Display the model's ACTUAL backend: the provider segment of the spec
# ("m1pro/qwen3.6:35b-mlx" -> "m1pro"); a bare id uses GADFLY_PROVIDER (default
# ollama-cloud). This is what the comment header shows, not the run.sh lane.
case "$MODEL" in
*/*) MODEL_PROVIDER="${MODEL%%/*}" ;;
*) MODEL_PROVIDER="${GADFLY_PROVIDER:-ollama-cloud}" ;;
esac
# jq is required for payload building / response parsing; install if missing.
if ! command -v jq >/dev/null 2>&1; then
say "jq not found; attempting install"
@@ -55,6 +78,32 @@ fi
# binary / agy, not here.)
API_TIMEOUT="--connect-timeout 20 --max-time 30"
# upsert_comment BODY — create or update (by MARKER) this model's single comment.
upsert_comment() {
local body="$1" post_body existing_id page=1 cmts
post_body="$(jq -n --arg b "$body" '{body:$b}')"
existing_id=""
while [ "$page" -le 10 ]; do
cmts="$(curl $API_TIMEOUT -fsS -H "Authorization: token ${GITEA_TOKEN}" \
"${GITEA_API}/issues/${PR}/comments?limit=50&page=${page}" || echo '[]')"
[ "$(echo "$cmts" | jq 'length')" = "0" ] && break
existing_id="$(echo "$cmts" | jq -r --arg m "$MARKER" \
'.[] | select(.body != null and (.body | startswith($m))) | .id' | head -n1)"
[ -n "$existing_id" ] && break
page=$((page+1))
done
if [ -n "$existing_id" ]; then
curl $API_TIMEOUT -sS -X PATCH -H "Authorization: token ${GITEA_TOKEN}" -H "Content-Type: application/json" \
"${GITEA_API}/issues/comments/${existing_id}" -d "$post_body" >/dev/null
else
curl $API_TIMEOUT -sS -X POST -H "Authorization: token ${GITEA_TOKEN}" -H "Content-Type: application/json" \
"${GITEA_API}/issues/${PR}/comments" -d "$post_body" >/dev/null
fi
}
# fmt_duration SECONDS -> "1m 23s" / "45s"
fmt_duration() { if [ "$1" -ge 60 ]; then echo "$(($1/60))m $(($1%60))s"; else echo "$1s"; fi; }
# --- fetch PR context -------------------------------------------------------
say "fetching PR #${PR} context"
DIFF="$(curl $API_TIMEOUT -fsS -H "Authorization: token ${GITEA_TOKEN}" "${GITEA_API}/pulls/${PR}.diff" || true)"
@@ -81,17 +130,35 @@ SYS="$(cat "${SCRIPT_DIR}/system-prompt.txt")"
USR="$(printf 'PR #%s: %s\n\nDescription:\n%s\n\nUnified diff to review:\n```diff\n%s\n```%s' \
"$PR" "$TITLE" "$BODY" "$DIFF" "$TRUNC_NOTE")"
# --- announce start (placeholder comment) -----------------------------------
START_TS="$(date +%s)"
say "starting review with ${MODEL}"
# Skip the per-model placeholder when consolidating (the consensus comment is
# posted later; live progress is on the status board).
[ "$CONSOLIDATE" = "1" ] || upsert_comment "$(printf '%s\n### 🪰 Gadfly review — `%s` (%s)\n\n⏳ Reviewing… this comment will update with findings and run time.' \
"$MARKER" "$MODEL" "$MODEL_PROVIDER")"
# --- call the model ---------------------------------------------------------
REVIEW=""
case "$PROVIDER" in
ollama)
# Agentic lane: hand off to the cmd/gadfly binary, which runs a tool-using
# agent over the checked-out repo so it can verify findings instead of
# guessing from the diff. The workflow builds the binary and exports
# GADFLY_BIN + GADFLY_REPO_DIR; we fall back to sane defaults for a
# local run.
if [ -z "${OLLAMA_CLOUD_API_KEY:-}" ]; then
REVIEW="⚠️ \`OLLAMA_CLOUD_API_KEY\` is not configured; this reviewer was skipped."
# guessing from the diff. The reviewer is majordomo-powered, so GADFLY_PROVIDER
# selects the backend (default ollama-cloud); local Ollama, OpenAI, Anthropic,
# Google and OpenAI/Ollama-compatible endpoints all work — see the README.
# Back-compat: map the consumer's OLLAMA_CLOUD_API_KEY secret onto the
# OLLAMA_API_KEY env the ollama-cloud provider reads.
if [ -n "${OLLAMA_CLOUD_API_KEY:-}" ] && [ -z "${OLLAMA_API_KEY:-}" ]; then
export OLLAMA_API_KEY="$OLLAMA_CLOUD_API_KEY"
fi
GADFLY_PROVIDER_EFF="$MODEL_PROVIDER"
# Only the default cloud provider strictly needs a key up front; local Ollama
# and other providers either need none or read their own standard env var.
if [ "$GADFLY_PROVIDER_EFF" = "ollama-cloud" ] && [ -z "${OLLAMA_API_KEY:-}" ] && [ -z "${GADFLY_API_KEY:-}" ]; then
REVIEW="⚠️ No Ollama Cloud key configured (set \`OLLAMA_CLOUD_API_KEY\`) and \`GADFLY_PROVIDER\` is the default \`ollama-cloud\`; this reviewer was skipped."
else
BIN="${GADFLY_BIN:-gadfly}"
if ! command -v "$BIN" >/dev/null 2>&1 && [ ! -x "$BIN" ]; then
@@ -101,15 +168,18 @@ case "$PROVIDER" in
DIFF_FILE="$(mktemp)"
ERR_FILE="${DIFF_FILE}.err"
printf '%s' "$FULL_DIFF" > "$DIFF_FILE"
# GADFLY_PROVIDER / GADFLY_BASE_URL / GADFLY_API_KEY and provider key
# envs (OPENAI_API_KEY, …) are inherited from the process environment.
REVIEW="$(
OLLAMA_API_KEY="$OLLAMA_CLOUD_API_KEY" \
GADFLY_MODEL="$MODEL" \
GADFLY_REPO_DIR="$REPO_DIR" \
GADFLY_DIFF_FILE="$DIFF_FILE" \
GADFLY_SYSTEM_FILE="${SCRIPT_DIR}/system-prompt.txt" \
GADFLY_TITLE="$TITLE" \
GADFLY_BODY="$BODY" \
GADFLY_MAX_DIFF_CHARS="$MAX_DIFF_CHARS" \
GADFLY_MAX_DIFF_CHARS="${GADFLY_MAX_DIFF_CHARS:-$MAX_DIFF_CHARS}" \
GADFLY_STATUS_FILE="${GADFLY_STATUS_FILE:-}" \
GADFLY_FINDINGS_OUT="${GADFLY_FINDINGS_OUT:-}" \
"$BIN" 2>"$ERR_FILE"
)"
rc=$?
@@ -141,31 +211,31 @@ $(tail -c 1500 agy.err 2>/dev/null)
say "unknown provider: ${PROVIDER}"; exit 1 ;;
esac
# --- assemble comment -------------------------------------------------------
COMMENT="$(printf '%s\n### 🔭 Adversarial review — `%s` (%s)\n\n%s\n\n<sub>Automated adversarial review. Advisory only — does not block merge.</sub>' \
"$MARKER" "$MODEL" "$PROVIDER" "$REVIEW")"
POST_BODY="$(jq -n --arg b "$COMMENT" '{body:$b}')"
# --- assemble + post final comment (with run time) --------------------------
ELAPSED="$(( $(date +%s) - START_TS ))"
DUR="$(fmt_duration "$ELAPSED")"
# --- upsert by marker -------------------------------------------------------
EXISTING_ID=""
page=1
while [ "$page" -le 10 ]; do
CMTS="$(curl $API_TIMEOUT -fsS -H "Authorization: token ${GITEA_TOKEN}" \
"${GITEA_API}/issues/${PR}/comments?limit=50&page=${page}" || echo '[]')"
[ "$(echo "$CMTS" | jq 'length')" = "0" ] && break
EXISTING_ID="$(echo "$CMTS" | jq -r --arg m "$MARKER" \
'.[] | select(.body != null and (.body | startswith($m))) | .id' | head -n1)"
[ -n "$EXISTING_ID" ] && break
page=$((page+1))
done
if [ -n "$EXISTING_ID" ]; then
say "updating existing comment ${EXISTING_ID}"
curl $API_TIMEOUT -sS -X PATCH -H "Authorization: token ${GITEA_TOKEN}" -H "Content-Type: application/json" \
"${GITEA_API}/issues/comments/${EXISTING_ID}" -d "$POST_BODY" >/dev/null
else
say "creating new comment"
curl $API_TIMEOUT -sS -X POST -H "Authorization: token ${GITEA_TOKEN}" -H "Content-Type: application/json" \
"${GITEA_API}/issues/${PR}/comments" -d "$POST_BODY" >/dev/null
# Consolidating: the binary writes its findings file on success. If it failed or
# was skipped (no file, or an empty one), write a stub so this model still shows
# up in the consensus (as failed) and an all-models-fail run still posts a
# comment — never silently drop a model or the whole review.
if [ "$CONSOLIDATE" = "1" ] && [ -n "${GADFLY_FINDINGS_OUT:-}" ] && [ ! -s "${GADFLY_FINDINGS_OUT}" ]; then
jq -n --arg model "$MODEL" --arg provider "$MODEL_PROVIDER" --arg md "$REVIEW" \
'{model:$model, provider:$provider, verdict:"reviewer failed", errored:true, markdown:$md, findings:[]}' \
> "${GADFLY_FINDINGS_OUT}" 2>/dev/null || true
fi
# When consolidating, the binary has written this model's findings to
# GADFLY_FINDINGS_OUT; the consensus comment is posted by entrypoint.sh after the
# whole swarm finishes, so this model posts no comment of its own.
if [ "$CONSOLIDATE" = "1" ]; then
say "done in ${DUR} (consolidated; no per-model comment)"
else
# An optional one-line notice (e.g. entrypoint's huge-PR downshift advisory),
# shown under the header so a shallower review is self-explaining.
NOTICE_BLOCK=""
[ -n "${GADFLY_NOTICE:-}" ] && NOTICE_BLOCK="> ${GADFLY_NOTICE}"$'\n\n'
COMMENT="$(printf '%s\n### 🪰 Gadfly review — `%s` (%s)\n\n%s%s\n\n<sub>Automated adversarial review by Gadfly. Advisory only — does not block merge. · ⏱️ reviewed in %s</sub>' \
"$MARKER" "$MODEL" "$MODEL_PROVIDER" "$NOTICE_BLOCK" "$REVIEW" "$DUR")"
upsert_comment "$COMMENT"
say "done in ${DUR}"
fi
say "done"
+137
View File
@@ -0,0 +1,137 @@
#!/usr/bin/env bash
# Live status board for a Gadfly review.
#
# Each model process (the cmd/gadfly binary) publishes its per-lens progress to
# $GADFLY_STATUS_DIR/<model>.json as lenses go queued -> running -> finished.
# This script polls that directory and upserts ONE consolidated PR comment that
# aggregates every model's per-lens status, so a human (or an agent watching the
# PR) can see the whole swarm's progress at a glance and know when it's done —
# instead of staring at N separate "⏳ Reviewing…" placeholders.
#
# It is advisory and best-effort: a failed render/post is logged and retried on
# the next tick; nothing here can fail the review or block a merge. It runs in
# the background from entrypoint.sh and exits once the $GADFLY_STATUS_DIR/.done
# sentinel appears (the entrypoint touches it after all model lanes finish),
# after one final render.
#
# Required env:
# GITEA_API https://HOST/api/v1/repos/OWNER/REPO
# GITEA_TOKEN token with repo write access (posts the comment)
# PR pull request number
# GADFLY_STATUS_DIR directory holding the per-model <model>.json files
# Optional:
# GADFLY_STATUS_POLL_SECS render/upsert interval (default 12)
set -uo pipefail
: "${GITEA_API:?GITEA_API required}"
: "${GITEA_TOKEN:?GITEA_TOKEN required}"
: "${PR:?PR required}"
: "${GADFLY_STATUS_DIR:?GADFLY_STATUS_DIR required}"
POLL="${GADFLY_STATUS_POLL_SECS:-12}"
# Guard against a non-numeric poll interval: with `set -uo pipefail` (no set -e)
# a bad `sleep "$POLL"` would fail silently and the `while :` loop would spin,
# hammering the Gitea API. Coerce anything non-integer (or <1) back to 12.
case "$POLL" in ''|*[!0-9]*) POLL=12 ;; esac
[ "$POLL" -ge 1 ] 2>/dev/null || POLL=12
DONE_FILE="${GADFLY_STATUS_DIR}/.done"
MARKER="<!-- gadfly-status-board -->"
API_TIMEOUT="--connect-timeout 20 --max-time 30"
BOARD_ID="" # cached comment id, so we PATCH in place instead of re-searching
say() { echo "[gadfly-status-board] $*" >&2; }
command -v jq >/dev/null 2>&1 || { say "jq not found; status board disabled"; exit 0; }
# render_section FILE -> markdown for one model (its header + per-lens bullets).
# Reads the JSON the binary writes; tolerates a half-written/missing file by
# emitting nothing (jq exits non-zero -> caller skips it this tick).
render_section() {
jq -r '
def icon(state; errored):
if state == "finished" then (if errored then "⚠️" else "✅" end)
elif state == "running" then "🔄"
else "⏸️" end;
def lensline:
"- " + icon(.state; (.errored // false)) + " **" + .name + "** — " +
( if .state == "finished" then (if (.errored // false) then "could not complete" else (.verdict // "done") end)
elif .state == "running" then "running"
else "queued" end );
( [.lenses[] | select(.state == "finished")] | length ) as $fin
| ( .lenses | length ) as $tot
| ( if .done then "✅ done"
elif $tot == 0 then "⏳ waiting to start"
else "⏳ " + ($fin|tostring) + "/" + ($tot|tostring) + " lenses" end ) as $sum
| "#### `" + .model + "` · " + .provider + " — " + $sum + "\n"
+ ( if $tot == 0 then "- ⏸️ _no lenses reported yet_"
else ([.lenses[] | lensline] | join("\n")) end )
' "$1" 2>/dev/null
}
# render_body -> the full consolidated comment body (marker + header + sections).
render_body() {
local f sections="" total=0 done=0 ts
shopt -s nullglob
local files=("${GADFLY_STATUS_DIR}"/*.json)
shopt -u nullglob
for f in "${files[@]}"; do
local sec
sec="$(render_section "$f")" || continue
[ -z "$sec" ] && continue
total=$((total + 1))
if [ "$(jq -r 'if .done then 1 else 0 end' "$f" 2>/dev/null)" = "1" ]; then
done=$((done + 1))
fi
sections="${sections}${sec}"$'\n\n'
done
ts="$(date -u '+%Y-%m-%d %H:%M:%SZ')"
if [ "$total" -eq 0 ]; then
sections="_Waiting for reviewers to start…_"$'\n'
fi
printf '%s\n## 🪰 Gadfly — live review status\n\n%d/%d reviewers finished · updated %s\n\n%s\n<sub>Live status board. Findings are posted in each model'\''s own comment. Advisory only — does not block merge.</sub>' \
"$MARKER" "$done" "$total" "$ts" "$sections"
}
# find_existing -> id of the board comment if it already exists (paginate by
# marker). Used once, to recover the comment across container restarts.
find_existing() {
local page=1 cmts id
while [ "$page" -le 10 ]; do
cmts="$(curl $API_TIMEOUT -fsS -H "Authorization: token ${GITEA_TOKEN}" \
"${GITEA_API}/issues/${PR}/comments?limit=50&page=${page}" 2>/dev/null || echo '[]')"
[ "$(echo "$cmts" | jq 'length' 2>/dev/null || echo 0)" = "0" ] && break
id="$(echo "$cmts" | jq -r --arg m "$MARKER" \
'.[] | select(.body != null and (.body | startswith($m))) | .id' 2>/dev/null | head -n1)"
[ -n "$id" ] && { echo "$id"; return; }
page=$((page + 1))
done
echo ""
}
# upsert BODY — PATCH the cached/known board comment, else POST a new one and
# cache its id. A failed PATCH (e.g. comment deleted) clears the cache so the
# next tick re-discovers or re-creates it.
upsert() {
local body="$1" post_body resp
post_body="$(jq -n --arg b "$body" '{body:$b}')"
[ -z "$BOARD_ID" ] && BOARD_ID="$(find_existing)"
if [ -n "$BOARD_ID" ]; then
if ! curl $API_TIMEOUT -fsS -X PATCH -H "Authorization: token ${GITEA_TOKEN}" -H "Content-Type: application/json" \
"${GITEA_API}/issues/comments/${BOARD_ID}" -d "$post_body" >/dev/null 2>&1; then
say "patch of comment ${BOARD_ID} failed; will re-discover"
BOARD_ID=""
fi
else
resp="$(curl $API_TIMEOUT -fsS -X POST -H "Authorization: token ${GITEA_TOKEN}" -H "Content-Type: application/json" \
"${GITEA_API}/issues/${PR}/comments" -d "$post_body" 2>/dev/null || echo '{}')"
BOARD_ID="$(echo "$resp" | jq -r '.id // ""' 2>/dev/null)"
fi
}
say "starting (poll ${POLL}s, dir ${GADFLY_STATUS_DIR})"
while :; do
upsert "$(render_body)"
[ -f "$DONE_FILE" ] && break
sleep "$POLL"
done
say "done"
+34 -16
View File
@@ -1,13 +1,24 @@
You are Gadfly, an ADVERSARIAL code reviewer. Your job is to find real problems in the
pull request below — not to praise it. A gadfly does not let things slide.
You review through ONE assigned lens (named at the end of this prompt). Evaluate the change
THROUGH THAT LENS — that is your job. A separate reviewer independently covers each other
angle, so problems outside your lens WILL be caught without you. Do not restate a finding that
plainly belongs to another lens just to have something to report — that only creates noise. If
your lens turns up nothing material, say so plainly; an honest "nothing in my area" beats
re-reporting the obvious bug every other lens already sees. Only exception: if you spot a
SEVERE issue clearly outside your lens, you may add ONE line prefixed "Outside my lens:" — but
your actual findings must stay within your lens.
You are AGENTIC: you have read-only tools over the repository AT THIS PR's checked-out
state. USE THEM to verify before you report. Do not review the diff in isolation.
- read_file(path[, start_line, limit]) — read a file with line numbers.
- list_dir([path]) — list a directory.
- grep(pattern[, path, max_results]) — RE2 regex search across the repo.
- find_files(name[, max_results]) — locate a file by path substring.
- get_diff() — the full unified diff (the task message may truncate it).
- get_diff([path, start_line, limit]) — the unified diff as a paginated, numbered window;
pass `path` to fetch just one changed file's hunks (do this on a big PR instead of pulling
the whole diff at once).
Mandatory verification discipline — this is the whole point of giving you tools:
- Before claiming a missing/duplicate import, an undefined symbol, a wrong signature,
@@ -19,21 +30,10 @@ Mandatory verification discipline — this is the whole point of giving you tool
- If you cannot confirm a suspicion with the tools, either drop it or clearly label it
"unverified" — do NOT present an unchecked guess as a finding.
Be skeptical and concrete. Hunt specifically for:
- Correctness bugs and logic errors introduced by the change.
- SEMANTIC / domain correctness — the failure mode plausible-looking code hides best.
Do NOT trust a constant, conversion factor, formula, unit, or threshold just because
it looks reasonable. Independently RE-DERIVE the expected value from first principles
(units, dimensions, edge values) and compare. A magic number that "looks about right"
is exactly where real bugs hide (e.g. a linear factor used where it must be squared).
- Concurrency issues: data races, deadlocks, unsynchronized shared state, leaked tasks.
- Security problems: injection, missing authz/authn, secret leakage, unsafe input handling.
- Error handling gaps: ignored errors, swallowed exceptions, missing rollback/cleanup.
- Resource leaks: unclosed handles/bodies/files, context/lifetime misuse, unbounded growth.
- Missed edge cases: off-by-one, nil/null, empty collection, overflow, zero/negative.
- Violations of THIS repo's own conventions. Discover them — do not assume. Read any
README / CONTRIBUTING / CLAUDE.md / AGENTS.md / lint config the repo ships, and hold
the change to the patterns the surrounding code actually uses.
Be skeptical and concrete, and apply your assigned lens rigorously. (If your lens leads you to
a constant, conversion factor, formula, unit, or threshold, don't trust it because it looks
reasonable — re-derive the expected value from first principles and compare; plausible-looking
magic numbers hide real bugs. Pursue this when it's in your lane, not as a reason to leave it.)
Output rules:
- Output GitHub-flavored markdown, concise. No filler, no restating the diff.
@@ -45,3 +45,21 @@ Output rules:
- Only report issues you are reasonably confident are real after checking. If the diff
is clean, say so plainly rather than inventing nits.
- When you are done investigating, STOP calling tools and reply with the final review.
Machine-readable findings — AFTER the prose review, append ONE fenced code block,
tagged `gadfly-findings`, holding a JSON array of the SAME findings you described above
(this block is consumed by tooling and hidden from the rendered comment):
```gadfly-findings
[
{"file": "path/to/file.go", "line": 123, "severity": "high", "confidence": "high", "title": "one-line summary of the issue"}
]
```
- One object per real finding, in the same order as your prose. `file`/`line` must be a
concrete location you verified (the line the issue is at). `severity` is one of
`critical`, `high`, `medium`, `small`, `trivial`. `confidence` is your post-verification
confidence the issue is real: one of `high`, `medium`, `low`.
- Include ONLY genuine problems — never verification notes ("confirmed X is safe at f:line"),
and never an "Outside my lens:" aside. If your lens is clean, emit an empty array `[]`.
- This block is in ADDITION to the prose; do not drop the human-readable findings.