6 Commits

Include renames

Author	SHA1	Message	Date
steveandClaude Opus 5	0358fe321e	internal/router: priority queues so batch jobs yield to interactive requests ... The GPU is a size-1 resource, so a single long job monopolises the box for its whole duration and every interactive request queues behind it. Callers can now declare intent with an X-LlamaSwap-Priority header and the serial scheduler dispatches by score instead of by arrival. - X-LlamaSwap-Priority: signed integer, 0 default, absent/unparseable means 0. interactive/normal/batch aliases resolve to +100/0/-100. Values are not clamped: the caller composes band and any per-user offset itself. - serial dispatch score = priority + swap affinity + aging. Bands sit 100 apart so a small caller offset orders work inside a band without crossing one; aging is unbounded so low-priority work cannot starve. - routing.scheduler.settings.serial.{agingDivisor,swapAffinityBonus}, defaulting to 60s/point and +10. swapAffinityBonus is capped at 99 so it can never promote a request into the next band. - fifo adds the header to its per-model priority, so the header is not silently ignored under that scheduler. - /metrics exports per-band queue depth, oldest wait and dispatch counts, plus counters for how often aging or swap affinity changed the pick. Each request records its priority, band, score and queue wait in the activity log. Note swapAffinityBonus defaults to 10, so equal-priority requests for the already-loaded model now run before older requests that need a swap. Set it to 0 for the previous strict arrival order. fixes #9 Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01WUyhZBgv8BBCC5MduX88gE	2026-08-07 02:12:08 -04:00
steveandClaude Opus 4.8	542b79dacf	internal/router/scheduler: add serial scheduler, default on this fork ... Add a strict one-model-at-a-time scheduler. Requests run in exact arrival order; at most one runs at a time; switching to a different model evicts every other running model first so a single model occupies memory at a time. Unlike fifo it never reorders or batches same-model requests, and it ignores group/matrix co-residency entirely, making the single-model guarantee a property of the scheduler rather than the config. - new Serial scheduler implementing the Scheduler interface - register "serial" in scheduler.New; default routing.scheduler.use to "serial" at config load (fifo still selectable for upstream behavior) - update config schema, example config, and config defaults tests Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>	2026-06-28 12:17:32 -04:00
Benson Wong	6cf1317341	schedule,shared: move concurrency 429 limits into scheduler code (#849) ... - make concurrency limiting the scheduler.Scheduler's responsibility - eliminate the separate concurrency limit middleware - move concurrencyLimit logic into scheduler.FIFO to maintain backwards compatibility - add HTTPError from #834 Updates #834	2026-06-15 22:35:12 -07:00
Benson Wong	ed77385d08	ui: improve manual model load and cancel (#847) ... - When a model is manually loaded show a cancel buttton and a queued status - Implement cancellation in scheduler.Scheduler interface and FIFO scheduler - Add cache bust query parameter to bypass browser cache Fixes #844	2026-06-14 13:38:10 -07:00
Benson Wong	62aea0e83d	internal/router,server,shared: refactor auth, libs (#839) ... - refactor shared http functionality into internal/shared/http.go - remove stripping of Authorization and x-api-key - add Request Context middleware to internal/server - add /ui and /metrics behind auth middleware, fixes #717 Fix #717 Updates: #834	2026-06-13 10:19:04 -07:00
Benson Wong	9b3a33d7b9	Implement new scheduler (#823) ... - introduce internal/router/scheduler to decouple routing, swapping and queuing into interface contracts. - introduce a new `routing` configuration section that supersedes `matrix` and `group` while maintaining backwards compatibility - add FIFO scheduler with prioritized queuing - add internal/router/design.md as developer documentation on implementing new schedulers and routers Fixes #797	2026-06-10 20:34:25 -07:00