The GPU is a size-1 resource, so a single long job monopolises the box for its whole duration and every interactive request queues behind it. Callers can now declare intent with an X-LlamaSwap-Priority header and the serial scheduler dispatches by score instead of by arrival.
X-LlamaSwap-Priority: signed integer, 0 default, absent/unparseable means 0. interactive/normal/batch aliases resolve to +100/0/-100. Values are not clamped: the caller composes band and any per-user offset itself.
serial dispatch score = priority + swap affinity + aging. Bands sit 100 apart so a small caller offset orders work inside a band without crossing one; aging is unbounded so low-priority work cannot starve.
routing.scheduler.settings.serial.{agingDivisor,swapAffinityBonus}, defaulting to 60s/point and +10. swapAffinityBonus is capped at 99 so it can never promote a request into the next band.
fifo adds the header to its per-model priority, so the header is not silently ignored under that scheduler.
/metrics exports per-band queue depth, oldest wait and dispatch counts, plus counters for how often aging or swap affinity changed the pick. Each request records its priority, band, score and queue wait in the activity log.
Two behaviour changes worth calling out:
swapAffinityBonus defaults to 10, so equal-priority requests for the already-loaded model now run before older requests that need a swap — A B C A serves as A A B C. Set it to 0 for the previous strict arrival order.
/metrics returns 200 with just scheduler metrics when performance monitoring is off, where it previously returned 503 unconditionally.
Fairness is out of scope, as the issue describes: per-member fair-share stays in the client that submits the work.
The GPU is a size-1 resource, so a single long job monopolises the box for its whole duration and every interactive request queues behind it. Callers can now declare intent with an `X-LlamaSwap-Priority` header and the serial scheduler dispatches by score instead of by arrival.
- `X-LlamaSwap-Priority`: signed integer, 0 default, absent/unparseable means 0. `interactive`/`normal`/`batch` aliases resolve to +100/0/-100. Values are not clamped: the caller composes band and any per-user offset itself.
- serial dispatch score = priority + swap affinity + aging. Bands sit 100 apart so a small caller offset orders work inside a band without crossing one; aging is unbounded so low-priority work cannot starve.
- `routing.scheduler.settings.serial.{agingDivisor,swapAffinityBonus}`, defaulting to 60s/point and +10. `swapAffinityBonus` is capped at 99 so it can never promote a request into the next band.
- fifo adds the header to its per-model priority, so the header is not silently ignored under that scheduler.
- `/metrics` exports per-band queue depth, oldest wait and dispatch counts, plus counters for how often aging or swap affinity changed the pick. Each request records its priority, band, score and queue wait in the activity log.
Two behaviour changes worth calling out:
- `swapAffinityBonus` defaults to 10, so equal-priority requests for the already-loaded model now run before older requests that need a swap — `A B C A` serves as `A A B C`. Set it to 0 for the previous strict arrival order.
- `/metrics` returns 200 with just scheduler metrics when performance monitoring is off, where it previously returned 503 unconditionally.
Fairness is out of scope, as the issue describes: per-member fair-share stays in the client that submits the work.
fixes #9
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01WUyhZBgv8BBCC5MduX88gE
The GPU is a size-1 resource, so a single long job monopolises the box for its
whole duration and every interactive request queues behind it. Callers can now
declare intent with an X-LlamaSwap-Priority header and the serial scheduler
dispatches by score instead of by arrival.
- X-LlamaSwap-Priority: signed integer, 0 default, absent/unparseable means 0.
interactive/normal/batch aliases resolve to +100/0/-100. Values are not
clamped: the caller composes band and any per-user offset itself.
- serial dispatch score = priority + swap affinity + aging. Bands sit 100 apart
so a small caller offset orders work inside a band without crossing one;
aging is unbounded so low-priority work cannot starve.
- routing.scheduler.settings.serial.{agingDivisor,swapAffinityBonus}, defaulting
to 60s/point and +10. swapAffinityBonus is capped at 99 so it can never
promote a request into the next band.
- fifo adds the header to its per-model priority, so the header is not silently
ignored under that scheduler.
- /metrics exports per-band queue depth, oldest wait and dispatch counts, plus
counters for how often aging or swap affinity changed the pick. Each request
records its priority, band, score and queue wait in the activity log.
Note swapAffinityBonus defaults to 10, so equal-priority requests for the
already-loaded model now run before older requests that need a swap. Set it to
0 for the previous strict arrival order.
fixes#9
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01WUyhZBgv8BBCC5MduX88gE
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
The GPU is a size-1 resource, so a single long job monopolises the box for its whole duration and every interactive request queues behind it. Callers can now declare intent with an
X-LlamaSwap-Priorityheader and the serial scheduler dispatches by score instead of by arrival.X-LlamaSwap-Priority: signed integer, 0 default, absent/unparseable means 0.interactive/normal/batchaliases resolve to +100/0/-100. Values are not clamped: the caller composes band and any per-user offset itself.routing.scheduler.settings.serial.{agingDivisor,swapAffinityBonus}, defaulting to 60s/point and +10.swapAffinityBonusis capped at 99 so it can never promote a request into the next band./metricsexports per-band queue depth, oldest wait and dispatch counts, plus counters for how often aging or swap affinity changed the pick. Each request records its priority, band, score and queue wait in the activity log.Two behaviour changes worth calling out:
swapAffinityBonusdefaults to 10, so equal-priority requests for the already-loaded model now run before older requests that need a swap —A B C Aserves asA A B C. Set it to 0 for the previous strict arrival order./metricsreturns 200 with just scheduler metrics when performance monitoring is off, where it previously returned 503 unconditionally.Fairness is out of scope, as the issue describes: per-member fair-share stays in the client that submits the work.
fixes #9
🤖 Generated with Claude Code
https://claude.ai/code/session_01WUyhZBgv8BBCC5MduX88gE
The GPU is a size-1 resource, so a single long job monopolises the box for its whole duration and every interactive request queues behind it. Callers can now declare intent with an X-LlamaSwap-Priority header and the serial scheduler dispatches by score instead of by arrival. - X-LlamaSwap-Priority: signed integer, 0 default, absent/unparseable means 0. interactive/normal/batch aliases resolve to +100/0/-100. Values are not clamped: the caller composes band and any per-user offset itself. - serial dispatch score = priority + swap affinity + aging. Bands sit 100 apart so a small caller offset orders work inside a band without crossing one; aging is unbounded so low-priority work cannot starve. - routing.scheduler.settings.serial.{agingDivisor,swapAffinityBonus}, defaulting to 60s/point and +10. swapAffinityBonus is capped at 99 so it can never promote a request into the next band. - fifo adds the header to its per-model priority, so the header is not silently ignored under that scheduler. - /metrics exports per-band queue depth, oldest wait and dispatch counts, plus counters for how often aging or swap affinity changed the pick. Each request records its priority, band, score and queue wait in the activity log. Note swapAffinityBonus defaults to 10, so equal-priority requests for the already-loaded model now run before older requests that need a swap. Set it to 0 for the previous strict arrival order. fixes #9 Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01WUyhZBgv8BBCC5MduX88gE