feat(concurrency): provider-wide lens budget, drop the model cap
Concurrency was two multiplicative gates in two processes: entrypoint.sh capped MODELS-at-once per provider (GADFLY_PROVIDER_CONCURRENCY) while each model's binary separately capped its own lenses (GADFLY_LENS_CONCURRENCY). A model therefore held its whole model-slot until its LAST lens finished, stalling the next model even with idle lens capacity. Collapse to one throttle: a provider-wide lens budget shared across all of that provider's models. entrypoint now runs every model in a lane at once and seeds a single cross-process permit pool per lane (a dir of N flock files, sized by GADFLY_PROVIDER_LENS_CONCURRENCY -> GADFLY_LENS_CONCURRENCY). Each lens pass (review+recheck) acquires a permit before it runs and releases it after, so a model winding down immediately yields its freed permits to another model's queued lenses. flock auto-releases on process death, so a killed/crashed model can't leak budget. - cmd/gadfly/lenssem.go: the flock permit pool (+ lenssem_test.go). - main.go: runSpecialists holds a shared permit per lens; fanout sized to the budget so a lone model can use all of it. Falls back to the in-process limit when no pool is set (local runs, tests). - entrypoint.sh: drop provider_cap/DEFAULT_CONC; run_lane runs all models and seeds the per-lane pool. - GADFLY_PROVIDER_CONCURRENCY / GADFLY_CONCURRENCY are now ignored; the reusable workflow marks provider_concurrency deprecated and stops forwarding it. Docs (README, CLAUDE.md, examples) updated per the maintenance rule. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
@@ -272,38 +272,32 @@ Unset = no delegation (current behavior).
|
||||
### Concurrency (per-provider lanes)
|
||||
|
||||
With multiple models, each **provider** is its own lane and lanes run in **parallel**, so a fast
|
||||
cloud provider isn't stuck behind a slow local box. Within a lane, at most `cap` models run at
|
||||
once — `cap` comes from `GADFLY_PROVIDER_CONCURRENCY` (a `provider=N` map) else `GADFLY_CONCURRENCY`
|
||||
(default `1`). The timeout is **per-lens** (`GADFLY_TIMEOUT_SECS`), so a slow model on one lens
|
||||
can't starve the others.
|
||||
cloud provider isn't stuck behind a slow local box. There is **one throttle**: a per-provider
|
||||
**lens budget** — the max number of lens passes (a lens = one specialist's review+recheck) in
|
||||
flight at once for that provider. Every model in the lane runs concurrently and its lenses draw
|
||||
from that single shared budget, so nothing else caps how many models run. The budget comes from
|
||||
`GADFLY_PROVIDER_LENS_CONCURRENCY` (a `provider=N` map) else the `GADFLY_LENS_CONCURRENCY` scalar
|
||||
(default `1`). The timeout is **per-lens** (`GADFLY_TIMEOUT_SECS`), so a slow lens can't starve
|
||||
the others.
|
||||
|
||||
```yaml
|
||||
# One local box (serial — it serves one model at a time) + 3 cloud reviews at once,
|
||||
# both lanes running concurrently:
|
||||
GADFLY_PROVIDER_CONCURRENCY: "ollama-cloud=3,m1pro=1"
|
||||
# The local box gets 1 lens at a time (serial); the cloud lane runs up to 3 lens passes at once,
|
||||
# shared across ALL its models. Both lanes run concurrently.
|
||||
GADFLY_PROVIDER_LENS_CONCURRENCY: "ollama-cloud=3,m1pro=1"
|
||||
GADFLY_MODELS: "m1pro/qwen3:14b,qwen3-coder:480b-cloud,gpt-oss:120b-cloud"
|
||||
```
|
||||
|
||||
A model's provider is the spec's first segment (`m1pro/…` → `m1pro`), or `GADFLY_PROVIDER`/
|
||||
`ollama-cloud` for a bare id. Default (`cap 1`) keeps a single-provider pool fully sequential.
|
||||
`ollama-cloud` for a bare id. The budget is **shared across the provider's models**: with a
|
||||
budget of 3 and two cloud models, you get 3 lens passes in flight in any mix — as one model
|
||||
finishes a lens, the freed slot immediately goes to another model's next lens, so a model
|
||||
winding down to its last lens never stalls the others (the pre-2026-07 design capped *models*
|
||||
separately and did stall — that `GADFLY_PROVIDER_CONCURRENCY`/`GADFLY_CONCURRENCY` model cap is
|
||||
**gone**; those vars are now ignored). Default (budget `1`) keeps a provider fully sequential.
|
||||
|
||||
**Lens fan-out (within a model).** By default the specialist lenses run **sequentially** inside
|
||||
each model (`GADFLY_LENS_CONCURRENCY=1`). Raise it to overlap the independent per-lens
|
||||
review+recheck passes — the model then posts its consolidated comment as soon as its lenses
|
||||
finish (so with sequential models, results stream in per model and per-model timings stay
|
||||
clean). Like the model cap, it's **per-provider configurable**: `GADFLY_PROVIDER_LENS_CONCURRENCY`
|
||||
takes a `provider=N` map keyed by the **same provider lanes** as `GADFLY_PROVIDER_CONCURRENCY`,
|
||||
falling back to the `GADFLY_LENS_CONCURRENCY` scalar (default `1`). **It multiplies with the
|
||||
model cap:** total in-flight requests ≈ *models-at-once × lenses-at-once*, so to fan lenses out
|
||||
without oversubscribing a backend, keep its model cap low and raise its lens cap:
|
||||
|
||||
```yaml
|
||||
# Per provider: cloud runs one model at a time but fans its 3 lenses out (3 concurrent requests);
|
||||
# the slow local box stays fully serial. Both provider lanes still run in parallel.
|
||||
GADFLY_PROVIDER_CONCURRENCY: "ollama-cloud=1,m1=1"
|
||||
GADFLY_PROVIDER_LENS_CONCURRENCY: "ollama-cloud=3,m1=1"
|
||||
GADFLY_SPECIALISTS: "security,correctness,error-handling"
|
||||
```
|
||||
> Under the hood the shared budget is a small cross-process permit pool (flock files, seeded per
|
||||
> lane by `entrypoint.sh`); permits release automatically if a model process dies, so a crashed
|
||||
> lens can't leak budget.
|
||||
|
||||
### Live status board
|
||||
|
||||
@@ -418,8 +412,7 @@ on its next review **without** a re-pin or a tag move:
|
||||
|---|---|
|
||||
| `GADFLY_DEFAULT_MODELS` | `GADFLY_MODELS` (csv) |
|
||||
| `GADFLY_DEFAULT_SPECIALISTS` | the lens suite |
|
||||
| `GADFLY_DEFAULT_PROVIDER_CONCURRENCY` | models-at-once per provider |
|
||||
| `GADFLY_DEFAULT_PROVIDER_LENS_CONCURRENCY` | lenses-at-once per provider |
|
||||
| `GADFLY_DEFAULT_PROVIDER_LENS_CONCURRENCY` | the per-provider lens budget (lens passes in flight per provider, shared across its models) |
|
||||
| `GADFLY_ENDPOINT_RAGNAROS` | a named endpoint, e.g. `llamaswap\|https://host` |
|
||||
|
||||
Adding a *new* named endpoint still needs a one-line reusable edit (Gitea can't auto-expose arbitrary
|
||||
@@ -441,10 +434,9 @@ The reviewer binary reads these (the stub/entrypoint set sane defaults):
|
||||
| `GADFLY_SELECTOR_MODEL` | review model | model that picks lenses in `auto` mode |
|
||||
| `GADFLY_WORKER_MODEL` | — | cheap model for `delegate_investigation`; unset = no delegation |
|
||||
| `GADFLY_WORKER_MAX_STEPS` | 8 | tool-step cap for a delegated worker run |
|
||||
| `GADFLY_CONCURRENCY` | 1 | default max models run at once **per provider** |
|
||||
| `GADFLY_PROVIDER_CONCURRENCY` | — | per-provider overrides, e.g. `ollama-cloud=3,m1pro=1` |
|
||||
| `GADFLY_LENS_CONCURRENCY` | 1 | specialist lenses run at once **within a model** (× model cap = total in-flight) |
|
||||
| `GADFLY_PROVIDER_LENS_CONCURRENCY` | — | per-provider lens overrides, same lanes as `GADFLY_PROVIDER_CONCURRENCY`, e.g. `ollama-cloud=3,m1=1` |
|
||||
| `GADFLY_LENS_CONCURRENCY` | 1 | **per-provider lens budget** — lens passes in flight per provider, shared across all its models (all a provider's models run at once; this is the only throttle) |
|
||||
| `GADFLY_PROVIDER_LENS_CONCURRENCY` | — | per-provider lens-budget overrides, a `provider=N` map, e.g. `ollama-cloud=3,m1=1` |
|
||||
| `GADFLY_CONCURRENCY` / `GADFLY_PROVIDER_CONCURRENCY` | — | **removed** (was the per-provider models-at-once cap; now ignored — the lens budget is the single throttle) |
|
||||
| `GADFLY_MAX_STEPS` | 24 | review-pass tool-step cap |
|
||||
| `GADFLY_TIMEOUT_SECS` | 300 | deadline **per specialist lens** (review+recheck) |
|
||||
| `GADFLY_RECHECK` | on | set `0`/`false` to skip the recheck pass |
|
||||
|
||||
Reference in New Issue
Block a user