steveandClaude Opus 5
3af0f09387
Fourteen findings, and seven of them from all four models are the same one: endpointProvider was missing the no-cross-vendor-fallback guard I had just added to resolveModel. I fixed a credential leak on one path and left its sibling leaking, in the commit whose own message argued those two paths must move together. That is the third time in this PR. So it is no longer a rule written twice. openAICompatOptions owns it and both paths call it; builtinCompatProviders names the vendors that must never inherit OPENAI_API_KEY, replacing a `provider == "kimi" || provider == "qwen"` literal that was a fourth uncounted copy of the list. The test drives a real request at a local server and demands two things: that no request arrives carrying the OpenAI key, AND that the call fails closed naming the variable to set — the second half because my first draft pointed the provider at vendor.example, so the server saw nothing and the assertion held for a reason unrelated to the fix. Break-checked: removing the guard puts "Bearer sk-openai-must-not-travel" on the wire to the other vendor. The scrub check failed open. As a bare condition, a grep ERROR (exit >= 2) reads as "not found" and skips the guard — a credential check that passes precisely when it cannot see the filesystem it is searching. It now distinguishes 0/1/>=2 and refuses to continue on error. A bare "claude-code" spec has no "/", so the provider fell back to ollama-cloud and the pre-flight would skip a reviewer that authenticates with CLAUDE_CODE_OAUTH_TOKEN and needs no Ollama key. Engine specs are now exempt. preflight.sh's provider list duplicated its own case arms; both now read one table. And its comment claimed the Go cross-check fails if either list misses an entry from the other, when only one direction is checked — the reverse is not even desirable, since ollama-cloud and anthropic belong in that table and not in the Go one. The comment now says what is enforced. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
256 lines
12 KiB
Bash
256 lines
12 KiB
Bash
#!/usr/bin/env bash
|
|
# Adversarial PR review runner.
|
|
#
|
|
# Fetches a PR's unified diff + metadata from Gitea, asks ONE model to review it
|
|
# adversarially, then upserts the result as a single labeled PR comment (so
|
|
# re-runs on new commits update the comment in place instead of stacking dupes).
|
|
#
|
|
# The ollama lane is AGENTIC: it runs the cmd/gadfly Go binary, which drives a
|
|
# tool-using agent (majordomo + Ollama Cloud) over the PR's checked-out repo so
|
|
# the model can read_file/grep/etc. to VERIFY findings instead of guessing from
|
|
# the diff alone. The antigravity lane stays a one-shot `agy` call (agy has its
|
|
# own file tools).
|
|
#
|
|
# Required env:
|
|
# GITEA_API e.g. https://gitea.stevedudenhoeffer.com/api/v1/repos/steve/mort
|
|
# GITEA_TOKEN token with repo write access (posts the comment)
|
|
# PR pull request index/number
|
|
# PROVIDER "ollama" | "antigravity"
|
|
# MODEL model id (e.g. qwen3-coder:480b-cloud, gemini-3-pro)
|
|
#
|
|
# Provider-specific env:
|
|
# ollama: OLLAMA_CLOUD_API_KEY, GADFLY_BIN (path to the built reviewer),
|
|
# GADFLY_REPO_DIR (checked-out repo; default: this script's repo)
|
|
# antigravity: `agy` on PATH with credentials already seeded (~/.gemini)
|
|
#
|
|
# claude-code engine: when MODEL is "claude-code" or "claude-code/<model>" the
|
|
# binary shells out to the bundled `claude` CLI instead of a majordomo model.
|
|
# Its auth (CLAUDE_CODE_OAUTH_TOKEN, else ANTHROPIC_API_KEY) and GADFLY_CLAUDE_*
|
|
# tuning are read straight from the inherited environment — same as the other
|
|
# provider keys (OPENAI_API_KEY, …) — so no extra wiring is needed here.
|
|
#
|
|
# opencode engine: when MODEL is "opencode" or "opencode/<model>" the binary
|
|
# shells out to the bundled `opencode` CLI, driving an ollama-cloud model (for
|
|
# benchmarking against the majordomo path on the same model). Its auth reuses
|
|
# OLLAMA_CLOUD_API_KEY (mapped to OLLAMA_API_KEY below, same as the ollama-cloud
|
|
# path) and GADFLY_OPENCODE_* tuning is read from the inherited environment — so
|
|
# no extra wiring is needed here either.
|
|
#
|
|
# Optional:
|
|
# MAX_DIFF_CHARS diff truncation cap for the prompt (default 60000)
|
|
# GADFLY_STATUS_FILE per-model JSON path for the live status board (set by
|
|
# entrypoint.sh; empty/unset disables status publishing)
|
|
#
|
|
# This script is advisory: it never fails the job for review content. It exits
|
|
# non-zero only on a usage/configuration error.
|
|
set -uo pipefail
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
MAX_DIFF_CHARS="${MAX_DIFF_CHARS:-60000}"
|
|
|
|
# Credential pre-flight, shared verbatim with scripts/preflight_test.sh so the
|
|
# tested logic and the running logic are the same bytes.
|
|
# shellcheck source=preflight.sh
|
|
. "$SCRIPT_DIR/preflight.sh"
|
|
|
|
: "${GITEA_API:?GITEA_API required}"
|
|
: "${GITEA_TOKEN:?GITEA_TOKEN required}"
|
|
: "${PR:?PR required}"
|
|
: "${PROVIDER:?PROVIDER required}"
|
|
: "${MODEL:?MODEL required}"
|
|
|
|
MARKER="<!-- gadfly-review:${PROVIDER}:${MODEL} -->"
|
|
say() { echo "[gadfly-review:${PROVIDER}:${MODEL}] $*" >&2; }
|
|
|
|
# When the swarm is consolidating (GADFLY_CONSOLIDATE=1, set by entrypoint.sh for
|
|
# a multi-model run), this model does NOT post its own comment — it writes its
|
|
# findings to GADFLY_FINDINGS_OUT and a single cross-model consensus comment is
|
|
# posted after the whole swarm finishes. Live progress still shows on the status
|
|
# board. Default 0 (post a per-model comment, the standalone behavior).
|
|
CONSOLIDATE="${GADFLY_CONSOLIDATE:-0}"
|
|
|
|
# Display the model's ACTUAL backend: the provider segment of the spec
|
|
# ("m1pro/qwen3.6:35b-mlx" -> "m1pro"); a bare id uses GADFLY_PROVIDER (default
|
|
# ollama-cloud). This is what the comment header shows, not the run.sh lane.
|
|
case "$MODEL" in
|
|
*/*) MODEL_PROVIDER="${MODEL%%/*}" ;;
|
|
*) MODEL_PROVIDER="${GADFLY_PROVIDER:-ollama-cloud}" ;;
|
|
esac
|
|
|
|
# jq is required for payload building / response parsing; install if missing.
|
|
if ! command -v jq >/dev/null 2>&1; then
|
|
say "jq not found; attempting install"
|
|
{ apt-get update -qq && apt-get install -y -qq jq; } >/dev/null 2>&1 \
|
|
|| { sudo apt-get update -qq && sudo apt-get install -y -qq jq; } >/dev/null 2>&1 \
|
|
|| { say "could not install jq"; exit 1; }
|
|
fi
|
|
|
|
# curl timeouts: Gitea API calls are quick. Word-split on purpose so the flags
|
|
# expand as separate args. (The LLM call's own deadline lives in the reviewer
|
|
# binary / agy, not here.)
|
|
API_TIMEOUT="--connect-timeout 20 --max-time 30"
|
|
|
|
# upsert_comment BODY — create or update (by MARKER) this model's single comment.
|
|
upsert_comment() {
|
|
local body="$1" post_body existing_id page=1 cmts
|
|
post_body="$(jq -n --arg b "$body" '{body:$b}')"
|
|
existing_id=""
|
|
while [ "$page" -le 10 ]; do
|
|
cmts="$(curl $API_TIMEOUT -fsS -H "Authorization: token ${GITEA_TOKEN}" \
|
|
"${GITEA_API}/issues/${PR}/comments?limit=50&page=${page}" || echo '[]')"
|
|
[ "$(echo "$cmts" | jq 'length')" = "0" ] && break
|
|
existing_id="$(echo "$cmts" | jq -r --arg m "$MARKER" \
|
|
'.[] | select(.body != null and (.body | startswith($m))) | .id' | head -n1)"
|
|
[ -n "$existing_id" ] && break
|
|
page=$((page+1))
|
|
done
|
|
if [ -n "$existing_id" ]; then
|
|
curl $API_TIMEOUT -sS -X PATCH -H "Authorization: token ${GITEA_TOKEN}" -H "Content-Type: application/json" \
|
|
"${GITEA_API}/issues/comments/${existing_id}" -d "$post_body" >/dev/null
|
|
else
|
|
curl $API_TIMEOUT -sS -X POST -H "Authorization: token ${GITEA_TOKEN}" -H "Content-Type: application/json" \
|
|
"${GITEA_API}/issues/${PR}/comments" -d "$post_body" >/dev/null
|
|
fi
|
|
}
|
|
|
|
# fmt_duration SECONDS -> "1m 23s" / "45s"
|
|
fmt_duration() { if [ "$1" -ge 60 ]; then echo "$(($1/60))m $(($1%60))s"; else echo "$1s"; fi; }
|
|
|
|
# --- fetch PR context -------------------------------------------------------
|
|
say "fetching PR #${PR} context"
|
|
DIFF="$(curl $API_TIMEOUT -fsS -H "Authorization: token ${GITEA_TOKEN}" "${GITEA_API}/pulls/${PR}.diff" || true)"
|
|
META="$(curl $API_TIMEOUT -fsS -H "Authorization: token ${GITEA_TOKEN}" "${GITEA_API}/pulls/${PR}" || echo '{}')"
|
|
TITLE="$(echo "$META" | jq -r '.title // ""')"
|
|
BODY="$(echo "$META" | jq -r '.body // ""')"
|
|
|
|
if [ -z "$DIFF" ]; then
|
|
say "empty diff; nothing to review"
|
|
exit 0
|
|
fi
|
|
|
|
# Keep the FULL diff for the agentic (ollama) reviewer — it can pull the whole
|
|
# thing via the get_diff tool and embeds a truncated copy in the prompt itself.
|
|
# The truncated copy below is only for the one-shot antigravity prompt.
|
|
FULL_DIFF="$DIFF"
|
|
TRUNC_NOTE=""
|
|
if [ "${#DIFF}" -gt "$MAX_DIFF_CHARS" ]; then
|
|
DIFF="${DIFF:0:$MAX_DIFF_CHARS}"
|
|
TRUNC_NOTE=$'\n\n[NOTE: diff truncated to '"${MAX_DIFF_CHARS}"' chars for length; review the rest manually.]'
|
|
fi
|
|
|
|
SYS="$(cat "${SCRIPT_DIR}/system-prompt.txt")"
|
|
USR="$(printf 'PR #%s: %s\n\nDescription:\n%s\n\nUnified diff to review:\n```diff\n%s\n```%s' \
|
|
"$PR" "$TITLE" "$BODY" "$DIFF" "$TRUNC_NOTE")"
|
|
|
|
# --- announce start (placeholder comment) -----------------------------------
|
|
START_TS="$(date +%s)"
|
|
say "starting review with ${MODEL}"
|
|
# Skip the per-model placeholder when consolidating (the consensus comment is
|
|
# posted later; live progress is on the status board).
|
|
[ "$CONSOLIDATE" = "1" ] || upsert_comment "$(printf '%s\n### 🪰 Gadfly review — `%s` (%s)\n\n⏳ Reviewing… this comment will update with findings and run time.' \
|
|
"$MARKER" "$MODEL" "$MODEL_PROVIDER")"
|
|
|
|
# --- call the model ---------------------------------------------------------
|
|
REVIEW=""
|
|
case "$PROVIDER" in
|
|
ollama)
|
|
# Agentic lane: hand off to the cmd/gadfly binary, which runs a tool-using
|
|
# agent over the checked-out repo so it can verify findings instead of
|
|
# guessing from the diff. The reviewer is majordomo-powered, so GADFLY_PROVIDER
|
|
# selects the backend (default ollama-cloud); local Ollama, OpenAI, Anthropic,
|
|
# Google and OpenAI/Ollama-compatible endpoints all work — see the README.
|
|
|
|
# Back-compat: map the consumer's OLLAMA_CLOUD_API_KEY secret onto the
|
|
# OLLAMA_API_KEY env the ollama-cloud provider reads.
|
|
if [ -n "${OLLAMA_CLOUD_API_KEY:-}" ] && [ -z "${OLLAMA_API_KEY:-}" ]; then
|
|
export OLLAMA_API_KEY="$OLLAMA_CLOUD_API_KEY"
|
|
fi
|
|
GADFLY_PROVIDER_EFF="$MODEL_PROVIDER"
|
|
|
|
# Credential pre-flight — one definition, shared with preflight_test.sh.
|
|
# Pass the raw spec too: engine specs (claude-code/opencode) carry their
|
|
# own auth and must not be judged by the provider fallback.
|
|
MISSING_KEY="$(gadfly_preflight_key "$GADFLY_PROVIDER_EFF" "$MODEL")"
|
|
if [ -n "$MISSING_KEY" ]; then
|
|
REVIEW="⚠️ No API key configured for provider \`${GADFLY_PROVIDER_EFF}\` (set \`${MISSING_KEY}\`); this reviewer was skipped."
|
|
else
|
|
BIN="${GADFLY_BIN:-gadfly}"
|
|
if ! command -v "$BIN" >/dev/null 2>&1 && [ ! -x "$BIN" ]; then
|
|
REVIEW="⚠️ Agentic reviewer binary not found (\`GADFLY_BIN=${BIN}\`); the workflow build step may have failed."
|
|
else
|
|
REPO_DIR="${GADFLY_REPO_DIR:-$(cd "${SCRIPT_DIR}/../../.." && pwd)}"
|
|
DIFF_FILE="$(mktemp)"
|
|
ERR_FILE="${DIFF_FILE}.err"
|
|
printf '%s' "$FULL_DIFF" > "$DIFF_FILE"
|
|
# GADFLY_PROVIDER / GADFLY_BASE_URL / GADFLY_API_KEY and provider key
|
|
# envs (OPENAI_API_KEY, …) are inherited from the process environment.
|
|
REVIEW="$(
|
|
GADFLY_MODEL="$MODEL" \
|
|
GADFLY_REPO_DIR="$REPO_DIR" \
|
|
GADFLY_DIFF_FILE="$DIFF_FILE" \
|
|
GADFLY_SYSTEM_FILE="${SCRIPT_DIR}/system-prompt.txt" \
|
|
GADFLY_TITLE="$TITLE" \
|
|
GADFLY_BODY="$BODY" \
|
|
GADFLY_MAX_DIFF_CHARS="${GADFLY_MAX_DIFF_CHARS:-$MAX_DIFF_CHARS}" \
|
|
GADFLY_STATUS_FILE="${GADFLY_STATUS_FILE:-}" \
|
|
GADFLY_FINDINGS_OUT="${GADFLY_FINDINGS_OUT:-}" \
|
|
"$BIN" 2>"$ERR_FILE"
|
|
)"
|
|
rc=$?
|
|
if [ "$rc" -ne 0 ] || [ -z "$REVIEW" ]; then
|
|
REVIEW="⚠️ Agentic reviewer for \`${MODEL}\` failed (exit ${rc}):
|
|
\`\`\`
|
|
$(tail -c 1500 "$ERR_FILE" 2>/dev/null)
|
|
\`\`\`"
|
|
fi
|
|
rm -f "$DIFF_FILE" "$ERR_FILE"
|
|
fi
|
|
fi
|
|
;;
|
|
antigravity)
|
|
if ! command -v agy >/dev/null 2>&1; then
|
|
REVIEW="⚠️ Antigravity CLI (\`agy\`) not found on PATH."
|
|
else
|
|
FULL="$(printf '%s\n\n%s' "$SYS" "$USR")"
|
|
if ! REVIEW="$(agy -p "$FULL" --model "$MODEL" 2>agy.err)"; then
|
|
REVIEW="⚠️ Antigravity CLI failed:
|
|
\`\`\`
|
|
$(tail -c 1500 agy.err 2>/dev/null)
|
|
\`\`\`"
|
|
fi
|
|
[ -z "$REVIEW" ] && REVIEW="⚠️ Antigravity CLI returned no output (auth/quota?)."
|
|
fi
|
|
;;
|
|
*)
|
|
say "unknown provider: ${PROVIDER}"; exit 1 ;;
|
|
esac
|
|
|
|
# --- assemble + post final comment (with run time) --------------------------
|
|
ELAPSED="$(( $(date +%s) - START_TS ))"
|
|
DUR="$(fmt_duration "$ELAPSED")"
|
|
|
|
# Consolidating: the binary writes its findings file on success. If it failed or
|
|
# was skipped (no file, or an empty one), write a stub so this model still shows
|
|
# up in the consensus (as failed) and an all-models-fail run still posts a
|
|
# comment — never silently drop a model or the whole review.
|
|
if [ "$CONSOLIDATE" = "1" ] && [ -n "${GADFLY_FINDINGS_OUT:-}" ] && [ ! -s "${GADFLY_FINDINGS_OUT}" ]; then
|
|
jq -n --arg model "$MODEL" --arg provider "$MODEL_PROVIDER" --arg md "$REVIEW" \
|
|
'{model:$model, provider:$provider, verdict:"reviewer failed", errored:true, markdown:$md, findings:[]}' \
|
|
> "${GADFLY_FINDINGS_OUT}" 2>/dev/null || true
|
|
fi
|
|
# When consolidating, the binary has written this model's findings to
|
|
# GADFLY_FINDINGS_OUT; the consensus comment is posted by entrypoint.sh after the
|
|
# whole swarm finishes, so this model posts no comment of its own.
|
|
if [ "$CONSOLIDATE" = "1" ]; then
|
|
say "done in ${DUR} (consolidated; no per-model comment)"
|
|
else
|
|
# An optional one-line notice (e.g. entrypoint's huge-PR downshift advisory),
|
|
# shown under the header so a shallower review is self-explaining.
|
|
NOTICE_BLOCK=""
|
|
[ -n "${GADFLY_NOTICE:-}" ] && NOTICE_BLOCK="> ${GADFLY_NOTICE}"$'\n\n'
|
|
COMMENT="$(printf '%s\n### 🪰 Gadfly review — `%s` (%s)\n\n%s%s\n\n<sub>Automated adversarial review by Gadfly. Advisory only — does not block merge. · ⏱️ reviewed in %s</sub>' \
|
|
"$MARKER" "$MODEL" "$MODEL_PROVIDER" "$NOTICE_BLOCK" "$REVIEW" "$DUR")"
|
|
upsert_comment "$COMMENT"
|
|
say "done in ${DUR}"
|
|
fi
|