feat(concurrency): provider-wide lens budget, drop the model cap
Build & push image / build-and-push (pull_request) Successful in 4s
Gadfly review (reusable) / review (pull_request) Successful in 14m47s
Adversarial Review (Gadfly) / review (pull_request) Successful in 14m47s

Concurrency was two multiplicative gates in two processes: entrypoint.sh
capped MODELS-at-once per provider (GADFLY_PROVIDER_CONCURRENCY) while each
model's binary separately capped its own lenses (GADFLY_LENS_CONCURRENCY).
A model therefore held its whole model-slot until its LAST lens finished,
stalling the next model even with idle lens capacity.

Collapse to one throttle: a provider-wide lens budget shared across all of
that provider's models. entrypoint now runs every model in a lane at once and
seeds a single cross-process permit pool per lane (a dir of N flock files,
sized by GADFLY_PROVIDER_LENS_CONCURRENCY -> GADFLY_LENS_CONCURRENCY). Each
lens pass (review+recheck) acquires a permit before it runs and releases it
after, so a model winding down immediately yields its freed permits to
another model's queued lenses. flock auto-releases on process death, so a
killed/crashed model can't leak budget.

- cmd/gadfly/lenssem.go: the flock permit pool (+ lenssem_test.go).
- main.go: runSpecialists holds a shared permit per lens; fanout sized to the
  budget so a lone model can use all of it. Falls back to the in-process limit
  when no pool is set (local runs, tests).
- entrypoint.sh: drop provider_cap/DEFAULT_CONC; run_lane runs all models and
  seeds the per-lane pool.
- GADFLY_PROVIDER_CONCURRENCY / GADFLY_CONCURRENCY are now ignored; the
  reusable workflow marks provider_concurrency deprecated and stops forwarding
  it. Docs (README, CLAUDE.md, examples) updated per the maintenance rule.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
2026-07-18 12:22:11 -04:00
co-authored by Claude Opus 4.8
parent 0d51879450
commit 74831368ab
9 changed files with 333 additions and 95 deletions
+23 -31
View File
@@ -272,38 +272,32 @@ Unset = no delegation (current behavior).
### Concurrency (per-provider lanes)
With multiple models, each **provider** is its own lane and lanes run in **parallel**, so a fast
cloud provider isn't stuck behind a slow local box. Within a lane, at most `cap` models run at
once — `cap` comes from `GADFLY_PROVIDER_CONCURRENCY` (a `provider=N` map) else `GADFLY_CONCURRENCY`
(default `1`). The timeout is **per-lens** (`GADFLY_TIMEOUT_SECS`), so a slow model on one lens
can't starve the others.
cloud provider isn't stuck behind a slow local box. There is **one throttle**: a per-provider
**lens budget** — the max number of lens passes (a lens = one specialist's review+recheck) in
flight at once for that provider. Every model in the lane runs concurrently and its lenses draw
from that single shared budget, so nothing else caps how many models run. The budget comes from
`GADFLY_PROVIDER_LENS_CONCURRENCY` (a `provider=N` map) else the `GADFLY_LENS_CONCURRENCY` scalar
(default `1`). The timeout is **per-lens** (`GADFLY_TIMEOUT_SECS`), so a slow lens can't starve
the others.
```yaml
# One local box (serial — it serves one model at a time) + 3 cloud reviews at once,
# both lanes running concurrently:
GADFLY_PROVIDER_CONCURRENCY: "ollama-cloud=3,m1pro=1"
# The local box gets 1 lens at a time (serial); the cloud lane runs up to 3 lens passes at once,
# shared across ALL its models. Both lanes run concurrently.
GADFLY_PROVIDER_LENS_CONCURRENCY: "ollama-cloud=3,m1pro=1"
GADFLY_MODELS: "m1pro/qwen3:14b,qwen3-coder:480b-cloud,gpt-oss:120b-cloud"
```
A model's provider is the spec's first segment (`m1pro/…``m1pro`), or `GADFLY_PROVIDER`/
`ollama-cloud` for a bare id. Default (`cap 1`) keeps a single-provider pool fully sequential.
`ollama-cloud` for a bare id. The budget is **shared across the provider's models**: with a
budget of 3 and two cloud models, you get 3 lens passes in flight in any mix — as one model
finishes a lens, the freed slot immediately goes to another model's next lens, so a model
winding down to its last lens never stalls the others (the pre-2026-07 design capped *models*
separately and did stall — that `GADFLY_PROVIDER_CONCURRENCY`/`GADFLY_CONCURRENCY` model cap is
**gone**; those vars are now ignored). Default (budget `1`) keeps a provider fully sequential.
**Lens fan-out (within a model).** By default the specialist lenses run **sequentially** inside
each model (`GADFLY_LENS_CONCURRENCY=1`). Raise it to overlap the independent per-lens
review+recheck passes — the model then posts its consolidated comment as soon as its lenses
finish (so with sequential models, results stream in per model and per-model timings stay
clean). Like the model cap, it's **per-provider configurable**: `GADFLY_PROVIDER_LENS_CONCURRENCY`
takes a `provider=N` map keyed by the **same provider lanes** as `GADFLY_PROVIDER_CONCURRENCY`,
falling back to the `GADFLY_LENS_CONCURRENCY` scalar (default `1`). **It multiplies with the
model cap:** total in-flight requests ≈ *models-at-once × lenses-at-once*, so to fan lenses out
without oversubscribing a backend, keep its model cap low and raise its lens cap:
```yaml
# Per provider: cloud runs one model at a time but fans its 3 lenses out (3 concurrent requests);
# the slow local box stays fully serial. Both provider lanes still run in parallel.
GADFLY_PROVIDER_CONCURRENCY: "ollama-cloud=1,m1=1"
GADFLY_PROVIDER_LENS_CONCURRENCY: "ollama-cloud=3,m1=1"
GADFLY_SPECIALISTS: "security,correctness,error-handling"
```
> Under the hood the shared budget is a small cross-process permit pool (flock files, seeded per
> lane by `entrypoint.sh`); permits release automatically if a model process dies, so a crashed
> lens can't leak budget.
### Live status board
@@ -418,8 +412,7 @@ on its next review **without** a re-pin or a tag move:
|---|---|
| `GADFLY_DEFAULT_MODELS` | `GADFLY_MODELS` (csv) |
| `GADFLY_DEFAULT_SPECIALISTS` | the lens suite |
| `GADFLY_DEFAULT_PROVIDER_CONCURRENCY` | models-at-once per provider |
| `GADFLY_DEFAULT_PROVIDER_LENS_CONCURRENCY` | lenses-at-once per provider |
| `GADFLY_DEFAULT_PROVIDER_LENS_CONCURRENCY` | the per-provider lens budget (lens passes in flight per provider, shared across its models) |
| `GADFLY_ENDPOINT_RAGNAROS` | a named endpoint, e.g. `llamaswap\|https://host` |
Adding a *new* named endpoint still needs a one-line reusable edit (Gitea can't auto-expose arbitrary
@@ -441,10 +434,9 @@ The reviewer binary reads these (the stub/entrypoint set sane defaults):
| `GADFLY_SELECTOR_MODEL` | review model | model that picks lenses in `auto` mode |
| `GADFLY_WORKER_MODEL` | — | cheap model for `delegate_investigation`; unset = no delegation |
| `GADFLY_WORKER_MAX_STEPS` | 8 | tool-step cap for a delegated worker run |
| `GADFLY_CONCURRENCY` | 1 | default max models run at once **per provider** |
| `GADFLY_PROVIDER_CONCURRENCY` | — | per-provider overrides, e.g. `ollama-cloud=3,m1pro=1` |
| `GADFLY_LENS_CONCURRENCY` | 1 | specialist lenses run at once **within a model** (× model cap = total in-flight) |
| `GADFLY_PROVIDER_LENS_CONCURRENCY` | — | per-provider lens overrides, same lanes as `GADFLY_PROVIDER_CONCURRENCY`, e.g. `ollama-cloud=3,m1=1` |
| `GADFLY_LENS_CONCURRENCY` | 1 | **per-provider lens budget** — lens passes in flight per provider, shared across all its models (all a provider's models run at once; this is the only throttle) |
| `GADFLY_PROVIDER_LENS_CONCURRENCY` | — | per-provider lens-budget overrides, a `provider=N` map, e.g. `ollama-cloud=3,m1=1` |
| `GADFLY_CONCURRENCY` / `GADFLY_PROVIDER_CONCURRENCY` | | **removed** (was the per-provider models-at-once cap; now ignored — the lens budget is the single throttle) |
| `GADFLY_MAX_STEPS` | 24 | review-pass tool-step cap |
| `GADFLY_TIMEOUT_SECS` | 300 | deadline **per specialist lens** (review+recheck) |
| `GADFLY_RECHECK` | on | set `0`/`false` to skip the recheck pass |