Priority queues: let long-running batch jobs yield to interactive requests #9

Closed
opened 2026-08-07 03:13:16 +00:00 by steve · 0 comments
Owner

The GPU is a size-1 resource — one job at a time. Today llama-swap serves requests in arrival order, so a single long job monopolises the box for its entire duration and every interactive request queues behind it.

This is already uncomfortable (a 16-minute H3 render at 362 frames blocks everything), and it becomes untenable with the planned long-form video work — steve/mort#1567 describes a ~12-hour render made of ~174 sequential H3 shots. Without priorities that job means nobody generates anything for a day.

Priority is a signed integer, 0 = normal, higher = more urgent

X-LlamaSwap-Priority: 0      # normal — the default
X-LlamaSwap-Priority: 100    # interactive, a human is waiting
X-LlamaSwap-Priority: -100   # batch, nobody is watching

An absent or unparseable header means 0.

Named aliases may resolve to numbers for convenience, but the number is the interface — callers can use any value.

alias value
interactive +100
normal 0
batch -100

Composition: bands, offsets, and aging

The dispatcher picks the highest effective priority:

effective = band + tier_bonus + swap_affinity + aging
term range crosses bands?
band -100 / 0 / +100 — it is the band
tier_bonus 0…+2 never
swap_affinity 0…+10 never
aging 0…∞ yes, deliberately

Tier bonus — max always beats pro beats free, within a band

This is the point of using numbers rather than fixed classes. A consumer adds a small per-member offset:

tier bonus
free +0
pro +1
max +2

So a max member's batch job is -100 + 2 = -98, ahead of pro's -99 and free's -100 — while all three remain below any normal job at 0. Tier breaks ties inside a band and can never promote across one.

Invariant: band spacing (100) must stay far larger than the maximum tier + affinity offset (12). Otherwise a max member's batch work could outrank a free member's interactive request, which would be wrong.

Aging is the one term allowed to cross bands

aging = waited_seconds / aging_divisor, unbounded on purpose — a batch job that has waited long enough should eventually beat normal traffic, or it starves. At a divisor of 60 (1 point/minute), a max batch job at -98 overtakes a free interactive job at +100 after ~3.3 hours of waiting.

That asymmetry is the design: bounded offsets express policy, unbounded aging prevents starvation.

Swap affinity keeps the model cache from thrashing

Cold loads cost 2-3x on this host. A job needing the already-resident model gets a bounded bonus, so strict priority does not force a swap on every dispatch. Bounded, so it can never promote batch above interactive — it only breaks near-ties.

All four terms fold into one score. Three independent rules interacting produce behaviour nobody can predict.

The hard parts

Non-preemptive is probably the only honest option

A single H3 generation is 4-16 minutes and cannot be interrupted mid-sample without discarding the work. Priority applies at dispatch time — when a job finishes, pick the best-scoring waiting one.

Consequence worth stating plainly: an interactive request can still wait up to one full job duration. Priority alone does not fix that. The complementary lever is client-side — a batch producer submits one unit at a time and re-queues, so gaps are frequent. For long-form video that means one ~4-minute shot per submission, not a 12-hour chain (steve/mort#1567 Phase 2).

If real preemption is ever wanted, the only sane form is checkpoint-at-step-boundary and requeue — much larger, probably not worth it.

Fairness vs priority

Priority is not fair-share. If one member submits 200 batch shots, per-member fairness is still needed within a priority level, or one person's film starves another's. mort's pkg/lane already implements fair-share queueing — look at it before inventing a second policy.

mort also caps one in-flight GPU job per member (steve/mort#1572), which bounds concurrency client-side and reduces how much fairness the scheduler must enforce itself.

API surface

A header, since priority is a property of the caller's intent, not of the model — the same model serves both an interactive .video and a batch film shot, so per-model config in config.yaml would be too coarse. A header also leaves the OpenAI-compatible body untouched.

Open: whether clients may freely self-assign large positive values, or whether that needs constraining.

Observability

Per priority band: queue depth, current wait, jobs dispatched, and how often aging or swap-affinity changed the dispatch order. Without the last one, tuning the scoring function is guesswork — and "why did my image take 20 minutes" will be the first question asked the day a 12-hour render is running.

Consumer side (mort)

  • pkg/lane passes the number through to the llama-swap client.
  • Film shots at batch; ordinary .image / .video at interactive; general skill-tool calls at normal. Subscription tier adds its bonus on top.
  • generate_long_video is currently not lane-gated at all (deliberately — see the comment in generate_long_video.go), so mort's visibility into a running chain is already poor. Worth fixing at the same time.

Related

  • steve/mort#1567 — long-form video epic, blocked on this
  • steve/mort#1572 — subscription tiers; supplies the tier bonus, caps in-flight GPU jobs per member
  • mort ADR-0053 — size-1 GPU lane + reserved code_exec tier
The GPU is a **size-1 resource** — one job at a time. Today llama-swap serves requests in arrival order, so a single long job monopolises the box for its entire duration and every interactive request queues behind it. This is already uncomfortable (a 16-minute H3 render at 362 frames blocks everything), and it becomes untenable with the planned **long-form video** work — `steve/mort#1567` describes a ~12-hour render made of ~174 sequential H3 shots. Without priorities that job means *nobody generates anything for a day*. ## Priority is a signed integer, 0 = normal, higher = more urgent ``` X-LlamaSwap-Priority: 0 # normal — the default X-LlamaSwap-Priority: 100 # interactive, a human is waiting X-LlamaSwap-Priority: -100 # batch, nobody is watching ``` An absent or unparseable header means **0**. Named aliases may resolve to numbers for convenience, but **the number is the interface** — callers can use any value. | alias | value | |---|---:| | `interactive` | +100 | | `normal` | 0 | | `batch` | -100 | ## Composition: bands, offsets, and aging The dispatcher picks the highest **effective** priority: ``` effective = band + tier_bonus + swap_affinity + aging ``` | term | range | crosses bands? | |---|---|---| | **band** | -100 / 0 / +100 | — it *is* the band | | **tier_bonus** | 0…+2 | **never** | | **swap_affinity** | 0…+10 | **never** | | **aging** | 0…∞ | **yes, deliberately** | ### Tier bonus — max always beats pro beats free, within a band This is the point of using numbers rather than fixed classes. A consumer adds a small per-member offset: | tier | bonus | |---|---:| | free | +0 | | pro | +1 | | max | +2 | So a max member's batch job is `-100 + 2 = -98`, ahead of pro's `-99` and free's `-100` — while all three remain **below** any normal job at 0. Tier breaks ties *inside* a band and can never promote across one. **Invariant: band spacing (100) must stay far larger than the maximum tier + affinity offset (12).** Otherwise a max member's batch work could outrank a free member's interactive request, which would be wrong. ### Aging is the one term allowed to cross bands `aging = waited_seconds / aging_divisor`, unbounded on purpose — a batch job that has waited long enough *should* eventually beat normal traffic, or it starves. At a divisor of 60 (1 point/minute), a max batch job at -98 overtakes a free interactive job at +100 after ~3.3 hours of waiting. That asymmetry is the design: **bounded offsets express policy, unbounded aging prevents starvation.** ### Swap affinity keeps the model cache from thrashing Cold loads cost **2-3x** on this host. A job needing the already-resident model gets a bounded bonus, so strict priority does not force a swap on every dispatch. Bounded, so it can never promote batch above interactive — it only breaks near-ties. All four terms fold into **one score**. Three independent rules interacting produce behaviour nobody can predict. ## The hard parts ### Non-preemptive is probably the only honest option A single H3 generation is **4-16 minutes** and cannot be interrupted mid-sample without discarding the work. Priority applies **at dispatch time** — when a job finishes, pick the best-scoring waiting one. Consequence worth stating plainly: an interactive request can still wait **up to one full job duration**. Priority alone does not fix that. The complementary lever is client-side — a batch producer submits **one unit at a time** and re-queues, so gaps are frequent. For long-form video that means one ~4-minute shot per submission, not a 12-hour chain (`steve/mort#1567` Phase 2). If real preemption is ever wanted, the only sane form is checkpoint-at-step-boundary and requeue — much larger, probably not worth it. ### Fairness vs priority Priority is not fair-share. If one member submits 200 batch shots, per-member fairness is still needed *within* a priority level, or one person's film starves another's. mort's `pkg/lane` already implements fair-share queueing — look at it before inventing a second policy. mort also caps **one in-flight GPU job per member** (`steve/mort#1572`), which bounds concurrency client-side and reduces how much fairness the scheduler must enforce itself. ## API surface A header, since priority is a property of the *caller's intent*, not of the model — the same model serves both an interactive `.video` and a batch film shot, so per-model config in `config.yaml` would be too coarse. A header also leaves the OpenAI-compatible body untouched. Open: whether clients may freely self-assign large positive values, or whether that needs constraining. ## Observability Per priority band: queue depth, current wait, jobs dispatched, and **how often aging or swap-affinity changed the dispatch order**. Without the last one, tuning the scoring function is guesswork — and "why did my image take 20 minutes" will be the first question asked the day a 12-hour render is running. ## Consumer side (mort) - `pkg/lane` passes the number through to the llama-swap client. - Film shots at `batch`; ordinary `.image` / `.video` at `interactive`; general skill-tool calls at `normal`. Subscription tier adds its bonus on top. - `generate_long_video` is currently **not** lane-gated at all (deliberately — see the comment in `generate_long_video.go`), so mort's visibility into a running chain is already poor. Worth fixing at the same time. ## Related - `steve/mort#1567` — long-form video epic, blocked on this - `steve/mort#1572` — subscription tiers; supplies the tier bonus, caps in-flight GPU jobs per member - mort ADR-0053 — size-1 GPU lane + reserved code_exec tier
steve closed this issue 2026-08-08 01:23:14 +00:00
Sign in to join this conversation.
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: steve/llama-swap#9