steve/llama-swap
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

internal/router: priority queues so batch jobs yield to interactive requests

Browse Source 
The GPU is a size-1 resource, so a single long job monopolises the box for its
whole duration and every interactive request queues behind it. Callers can now
declare intent with an X-LlamaSwap-Priority header and the serial scheduler
dispatches by score instead of by arrival.

- X-LlamaSwap-Priority: signed integer, 0 default, absent/unparseable means 0.
  interactive/normal/batch aliases resolve to +100/0/-100. Values are not
  clamped: the caller composes band and any per-user offset itself.
- serial dispatch score = priority + swap affinity + aging. Bands sit 100 apart
  so a small caller offset orders work inside a band without crossing one;
  aging is unbounded so low-priority work cannot starve.
- routing.scheduler.settings.serial.{agingDivisor,swapAffinityBonus}, defaulting
  to 60s/point and +10. swapAffinityBonus is capped at 99 so it can never
  promote a request into the next band.
- fifo adds the header to its per-model priority, so the header is not silently
  ignored under that scheduler.
- /metrics exports per-band queue depth, oldest wait and dispatch counts, plus
  counters for how often aging or swap affinity changed the pick. Each request
  records its priority, band, score and queue wait in the activity log.

Note swapAffinityBonus defaults to 10, so equal-priority requests for the
already-loaded model now run before older requests that need a swap. Set it to
0 for the previous strict arrival order.

fixes #9

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01WUyhZBgv8BBCC5MduX88gE
This commit is contained in:
steve
2026-08-07 02:12:08 -04:00
co-authored by Claude Opus 5
parent 2f0bb2556e
commit 0358fe321e
22 changed files with 1515 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/request-priority.md
+151
View File
@@ -0,0 +1,151 @@
# Request priority
A GPU is a size-1 resource: one job at a time. Without priorities, a single long
job monopolises the box for its whole duration and every interactive request
queues behind it. A 16-minute render blocking a chat completion is
uncomfortable; a 12-hour batch render blocking everything for a day is not
workable.
Callers declare intent with a header, and the scheduler dispatches by score.
## The header
```
X-LlamaSwap-Priority: 0 # normal — the default
X-LlamaSwap-Priority: 100 # interactive, a human is waiting
X-LlamaSwap-Priority: -100 # batch, nobody is watching
```
Priority is a **signed integer**; higher is more urgent. An absent or
unparseable header means `0`. Values are not clamped — llama-swap trusts the
caller.
Named aliases resolve to numbers for convenience, but **the number is the
interface**: callers may send any value.
| alias | value |
| ------------- | -----: |
| `interactive` | `100` |
| `normal` | `0` |
| `batch` | `-100` |
Priority is a property of the *caller's intent*, not of the model — the same
model serves both an interactive request and a batch job — which is why it
travels as a header rather than as per-model config. It also leaves the
OpenAI-compatible request body untouched.
```bash
curl http://localhost:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-H 'X-LlamaSwap-Priority: interactive' \
-d '{"model":"qwen3","messages":[{"role":"user","content":"hi"}]}'
```
## Bands and offsets
The three anchors sit 100 apart. That spacing is deliberate: it leaves room for
a caller to add a small per-user offset without ever crossing a band.
| tier | offset |
| ---- | -----: |
| free | `+0` |
| pro | `+1` |
| max | `+2` |
A max member's batch job is `-100 + 2 = -98`: ahead of pro's `-99` and free's
`-100`, and still **below** any normal request at `0`. A tier bonus breaks ties
*inside* a band and can never promote across one.
> **Invariant:** band spacing (100) must stay far larger than the largest offset
> a caller adds plus `swapAffinityBonus`. Otherwise a max member's batch work
> could outrank a free member's interactive request.
## How the serial scheduler dispatches
Scheduling is **non-preemptive**. A generation cannot be interrupted mid-sample
without discarding the work, so priority applies at *dispatch* time: when a job
finishes, the best-scoring waiting request goes next.
```
score = X-LlamaSwap-Priority + swap affinity + aging
```
| term | range | crosses bands? |
| --------------------- | ------- | ----------------------- |
| **request priority** | any | — it *is* the band |
| **swap affinity** | `0..99` | **never** |
| **aging** | `0..∞` | **yes, deliberately** |
Equal scores keep arrival order.
**Swap affinity** is a bounded bonus for a request that can run without a model
swap. Cold loads are expensive, so this keeps a run of same-model requests
together instead of forcing a reload on every dispatch. Because it is bounded
well below the band spacing, it can only break near-ties — it can never promote
batch work above interactive.
**Aging** is `waited_seconds / agingDivisor`, and is unbounded on purpose: a
batch job that has waited long enough *should* eventually beat normal traffic,
or it starves. At the default divisor of 60 (one point per minute), a batch job
at `-100` overtakes normal traffic after ~100 minutes of waiting.
That asymmetry is the design: **bounded offsets express policy, unbounded aging
prevents starvation.**
### What priority does not fix
Because dispatch is non-preemptive, an interactive request can still wait **up
to one full job duration**. The complementary lever is client-side: a batch
producer should submit **one unit at a time** and re-queue, so gaps are frequent.
For a long render that means one shot per submission, not one 12-hour chain.
Priority is also not fair-share. If one caller submits 200 batch jobs they all
sit at the same priority, and llama-swap will work through them in order. Per-user
fairness belongs in the client that submits the work.
## Configuration
```yaml
routing:
scheduler:
use: serial
settings:
serial:
# Seconds of waiting per priority point. 0 disables aging.
agingDivisor: 60
# Bonus for a request needing no model swap. 0..99; 0 disables it.
swapAffinityBonus: 10
```
Setting `swapAffinityBonus: 0` gives strict priority-then-arrival order — with
uniform priorities that is exact FIFO.
The `fifo` scheduler honours the header too, adding it to the model's configured
priority: a model at `10` with a `batch` header queues at `-90`.
## Observability
`GET /metrics` exports the queue in Prometheus text format, labelled by band
(`interactive`, `normal`, `batch`):
| metric | type | meaning |
| --------------------------------------------- | ------- | ---------------------------------------------------- |
| `llamaswap_scheduler_queue_depth` | gauge | requests waiting |
| `llamaswap_scheduler_queue_oldest_wait_seconds`| gauge | how long the longest-waiting request has waited |
| `llamaswap_scheduler_dispatched_total` | counter | requests dispatched |
| `llamaswap_scheduler_reorders_total` | counter | dispatches where `aging` or `swap_affinity` changed the pick |
The reorder counters are what make `agingDivisor` and `swapAffinityBonus`
tunable rather than guesswork: they say how often each term actually changed
which request went next.
Every dispatched request also records its decision in the activity log:
| key | meaning |
| ---------------------- | ------------------------------------------ |
| `serial_priority` | the caller's priority |
| `serial_band` | which band it fell in |
| `serial_score` | the score it won with |
| `serial_queue_wait_ms` | how long it waited before dispatch |
That is what answers "why did my request take 20 minutes" after the fact.