internal/router: priority queues so batch jobs yield to interactive requests
Callers declare intent with X-LlamaSwap-Priority (signed int, 0 default,
interactive/normal/batch aliases at +100/0/-100). serial dispatches by
score = priority + swap affinity + aging; fifo adds the header to the
model's configured priority so it is not silently ignored there.
Closes#9. Consumer side: steve/mort#1576.
The GPU is a size-1 resource, so a single long job monopolises the box for its
whole duration and every interactive request queues behind it. Callers can now
declare intent with an X-LlamaSwap-Priority header and the serial scheduler
dispatches by score instead of by arrival.
- X-LlamaSwap-Priority: signed integer, 0 default, absent/unparseable means 0.
interactive/normal/batch aliases resolve to +100/0/-100. Values are not
clamped: the caller composes band and any per-user offset itself.
- serial dispatch score = priority + swap affinity + aging. Bands sit 100 apart
so a small caller offset orders work inside a band without crossing one;
aging is unbounded so low-priority work cannot starve.
- routing.scheduler.settings.serial.{agingDivisor,swapAffinityBonus}, defaulting
to 60s/point and +10. swapAffinityBonus is capped at 99 so it can never
promote a request into the next band.
- fifo adds the header to its per-model priority, so the header is not silently
ignored under that scheduler.
- /metrics exports per-band queue depth, oldest wait and dispatch counts, plus
counters for how often aging or swap affinity changed the pick. Each request
records its priority, band, score and queue wait in the activity log.
Note swapAffinityBonus defaults to 10, so equal-priority requests for the
already-loaded model now run before older requests that need a swap. Set it to
0 for the previous strict arrival order.
fixes#9
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01WUyhZBgv8BBCC5MduX88gE
Push-triggered builds always defaulted to the v230 label, silently
overwriting it on every main merge — a stack pinned to any other
version label (v231) kept pulling stale images. Every build now also
pushes sha-<short> (immutable pin) and main-cuda-<llamacpp> (explicit
latest-main), so deployments can choose mutable-tracking or exact-pin
deliberately.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01AXQxVhXBw8PwFAtsVrXSmj
New Video tab in the Playground: model selector filtered to
video_generation/image_to_video capabilities, prompt + negative prompt,
size/frames/fps/steps/guidance/seed knobs (0 = model default), optional
conditioning-image upload (image-to-video), elapsed-time spinner for
the minutes-long blocking call, inline <video> playback + download.
Calls POST /v1/videos/sync (multipart); resolution rides as both
width/height and size so either upstream convention honors it.
Config side: "video" joins the valid capability modalities, and
/v1/models maps text->video to video_generation and image->video to
image_to_video, mirroring the image mappings. Declare e.g.:
videogen-wan22-5b:
capabilities:
in: [text, image]
out: [video]
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01AXQxVhXBw8PwFAtsVrXSmj
The merge of the /v1/videos/sync routes produced no image because the
push trigger only watched the workflow file and Containerfile. Now any
Go/UI/deps change on main rebuilds; docs/config-example edits still
don't. (The video-routes image itself was built via manual dispatch,
tag prefix v231.)
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01AXQxVhXBw8PwFAtsVrXSmj
Review findings: the async POST /v1/videos leg is deliberately NOT
dispatched — its poll/download companions carry no model field to route
by, so registering creation alone would start unretrievable upstream
jobs (a stray OpenAI-videos client now gets a clean 404). And
/v1/videos/sync joins captureFieldsByPath (headers only, both
directions) so enabling captures doesn't buffer conditioning frames or
whole clips through cbor+zstd.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01AXQxVhXBw8PwFAtsVrXSmj
Registers the OpenAI/vLLM-Omni video generation endpoints as
model-dispatched routes so llama-swap can swap in video upstreams
(e.g. vllm serve <model> --omni, or a ComfyUI shim exposing the same
shape). Extraction is content-type driven, so both multipart form and
JSON bodies resolve the model field.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01AXQxVhXBw8PwFAtsVrXSmj
npm ci ran without .npmrc (legacy-peer-deps=true), failing on the
tailwind/vite peer dependency conflict. Copy .npmrc with the manifest.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Add a Gitea Actions workflow and multi-stage Containerfile that build
this fork's llama-swap (serial scheduler + embedded Svelte UI) from
source and layer it on a pinned llama.cpp CUDA server base, then push to
the Gitea container registry as v230-cuda-b9821.
- docker/fork-cuda.Containerfile: node UI -> go build -> cuda runtime,
runs as root to match the upstream non-suffixed image
- .gitea/workflows/build-cuda-image.yml: workflow_dispatch (version +
llama.cpp build inputs) and push-on-build-files; logs in with
REGISTRY_USER/REGISTRY_PASSWORD
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Add a strict one-model-at-a-time scheduler. Requests run in exact
arrival order; at most one runs at a time; switching to a different
model evicts every other running model first so a single model occupies
memory at a time. Unlike fifo it never reorders or batches same-model
requests, and it ignores group/matrix co-residency entirely, making the
single-model guarantee a property of the scheduler rather than the config.
- new Serial scheduler implementing the Scheduler interface
- register "serial" in scheduler.New; default routing.scheduler.use to
"serial" at config load (fifo still selectable for upstream behavior)
- update config schema, example config, and config defaults tests
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
steve
