The GPU is a size-1 resource — one job at a time. Today llama-swap serves requests in arrival order, so a single long job monopolises the box for its entire duration and every interactive request queues behind it.
This is already uncomfortable (a 16-minute H3 render at 362 frames blocks everything), and it becomes untenable with the planned long-form video work — steve/mort#1567 describes a ~12-hour render made of ~174 sequential H3 shots. Without priorities that job means nobody generates anything for a day.
Priority is a signed integer, 0 = normal, higher = more urgent
X-LlamaSwap-Priority: 0 # normal — the default
X-LlamaSwap-Priority: 100 # interactive, a human is waiting
X-LlamaSwap-Priority: -100 # batch, nobody is watching
An absent or unparseable header means 0.
Named aliases may resolve to numbers for convenience, but the number is the interface — callers can use any value.
alias
value
interactive
+100
normal
0
batch
-100
Composition: bands, offsets, and aging
The dispatcher picks the highest effective priority:
effective = band + tier_bonus + swap_affinity + aging
term
range
crosses bands?
band
-100 / 0 / +100
— it is the band
tier_bonus
0…+2
never
swap_affinity
0…+10
never
aging
0…∞
yes, deliberately
Tier bonus — max always beats pro beats free, within a band
This is the point of using numbers rather than fixed classes. A consumer adds a small per-member offset:
tier
bonus
free
+0
pro
+1
max
+2
So a max member's batch job is -100 + 2 = -98, ahead of pro's -99 and free's -100 — while all three remain below any normal job at 0. Tier breaks ties inside a band and can never promote across one.
Invariant: band spacing (100) must stay far larger than the maximum tier + affinity offset (12). Otherwise a max member's batch work could outrank a free member's interactive request, which would be wrong.
Aging is the one term allowed to cross bands
aging = waited_seconds / aging_divisor, unbounded on purpose — a batch job that has waited long enough should eventually beat normal traffic, or it starves. At a divisor of 60 (1 point/minute), a max batch job at -98 overtakes a free interactive job at +100 after ~3.3 hours of waiting.
That asymmetry is the design: bounded offsets express policy, unbounded aging prevents starvation.
Swap affinity keeps the model cache from thrashing
Cold loads cost 2-3x on this host. A job needing the already-resident model gets a bounded bonus, so strict priority does not force a swap on every dispatch. Bounded, so it can never promote batch above interactive — it only breaks near-ties.
All four terms fold into one score. Three independent rules interacting produce behaviour nobody can predict.
The hard parts
Non-preemptive is probably the only honest option
A single H3 generation is 4-16 minutes and cannot be interrupted mid-sample without discarding the work. Priority applies at dispatch time — when a job finishes, pick the best-scoring waiting one.
Consequence worth stating plainly: an interactive request can still wait up to one full job duration. Priority alone does not fix that. The complementary lever is client-side — a batch producer submits one unit at a time and re-queues, so gaps are frequent. For long-form video that means one ~4-minute shot per submission, not a 12-hour chain (steve/mort#1567 Phase 2).
If real preemption is ever wanted, the only sane form is checkpoint-at-step-boundary and requeue — much larger, probably not worth it.
Fairness vs priority
Priority is not fair-share. If one member submits 200 batch shots, per-member fairness is still needed within a priority level, or one person's film starves another's. mort's pkg/lane already implements fair-share queueing — look at it before inventing a second policy.
mort also caps one in-flight GPU job per member (steve/mort#1572), which bounds concurrency client-side and reduces how much fairness the scheduler must enforce itself.
API surface
A header, since priority is a property of the caller's intent, not of the model — the same model serves both an interactive .video and a batch film shot, so per-model config in config.yaml would be too coarse. A header also leaves the OpenAI-compatible body untouched.
Open: whether clients may freely self-assign large positive values, or whether that needs constraining.
Observability
Per priority band: queue depth, current wait, jobs dispatched, and how often aging or swap-affinity changed the dispatch order. Without the last one, tuning the scoring function is guesswork — and "why did my image take 20 minutes" will be the first question asked the day a 12-hour render is running.
Consumer side (mort)
pkg/lane passes the number through to the llama-swap client.
Film shots at batch; ordinary .image / .video at interactive; general skill-tool calls at normal. Subscription tier adds its bonus on top.
generate_long_video is currently not lane-gated at all (deliberately — see the comment in generate_long_video.go), so mort's visibility into a running chain is already poor. Worth fixing at the same time.
Related
steve/mort#1567 — long-form video epic, blocked on this
steve/mort#1572 — subscription tiers; supplies the tier bonus, caps in-flight GPU jobs per member
mort ADR-0053 — size-1 GPU lane + reserved code_exec tier
The GPU is a **size-1 resource** — one job at a time. Today llama-swap serves requests in arrival order, so a single long job monopolises the box for its entire duration and every interactive request queues behind it.
This is already uncomfortable (a 16-minute H3 render at 362 frames blocks everything), and it becomes untenable with the planned **long-form video** work — `steve/mort#1567` describes a ~12-hour render made of ~174 sequential H3 shots. Without priorities that job means *nobody generates anything for a day*.
## Priority is a signed integer, 0 = normal, higher = more urgent
```
X-LlamaSwap-Priority: 0 # normal — the default
X-LlamaSwap-Priority: 100 # interactive, a human is waiting
X-LlamaSwap-Priority: -100 # batch, nobody is watching
```
An absent or unparseable header means **0**.
Named aliases may resolve to numbers for convenience, but **the number is the interface** — callers can use any value.
| alias | value |
|---|---:|
| `interactive` | +100 |
| `normal` | 0 |
| `batch` | -100 |
## Composition: bands, offsets, and aging
The dispatcher picks the highest **effective** priority:
```
effective = band + tier_bonus + swap_affinity + aging
```
| term | range | crosses bands? |
|---|---|---|
| **band** | -100 / 0 / +100 | — it *is* the band |
| **tier_bonus** | 0…+2 | **never** |
| **swap_affinity** | 0…+10 | **never** |
| **aging** | 0…∞ | **yes, deliberately** |
### Tier bonus — max always beats pro beats free, within a band
This is the point of using numbers rather than fixed classes. A consumer adds a small per-member offset:
| tier | bonus |
|---|---:|
| free | +0 |
| pro | +1 |
| max | +2 |
So a max member's batch job is `-100 + 2 = -98`, ahead of pro's `-99` and free's `-100` — while all three remain **below** any normal job at 0. Tier breaks ties *inside* a band and can never promote across one.
**Invariant: band spacing (100) must stay far larger than the maximum tier + affinity offset (12).** Otherwise a max member's batch work could outrank a free member's interactive request, which would be wrong.
### Aging is the one term allowed to cross bands
`aging = waited_seconds / aging_divisor`, unbounded on purpose — a batch job that has waited long enough *should* eventually beat normal traffic, or it starves. At a divisor of 60 (1 point/minute), a max batch job at -98 overtakes a free interactive job at +100 after ~3.3 hours of waiting.
That asymmetry is the design: **bounded offsets express policy, unbounded aging prevents starvation.**
### Swap affinity keeps the model cache from thrashing
Cold loads cost **2-3x** on this host. A job needing the already-resident model gets a bounded bonus, so strict priority does not force a swap on every dispatch. Bounded, so it can never promote batch above interactive — it only breaks near-ties.
All four terms fold into **one score**. Three independent rules interacting produce behaviour nobody can predict.
## The hard parts
### Non-preemptive is probably the only honest option
A single H3 generation is **4-16 minutes** and cannot be interrupted mid-sample without discarding the work. Priority applies **at dispatch time** — when a job finishes, pick the best-scoring waiting one.
Consequence worth stating plainly: an interactive request can still wait **up to one full job duration**. Priority alone does not fix that. The complementary lever is client-side — a batch producer submits **one unit at a time** and re-queues, so gaps are frequent. For long-form video that means one ~4-minute shot per submission, not a 12-hour chain (`steve/mort#1567` Phase 2).
If real preemption is ever wanted, the only sane form is checkpoint-at-step-boundary and requeue — much larger, probably not worth it.
### Fairness vs priority
Priority is not fair-share. If one member submits 200 batch shots, per-member fairness is still needed *within* a priority level, or one person's film starves another's. mort's `pkg/lane` already implements fair-share queueing — look at it before inventing a second policy.
mort also caps **one in-flight GPU job per member** (`steve/mort#1572`), which bounds concurrency client-side and reduces how much fairness the scheduler must enforce itself.
## API surface
A header, since priority is a property of the *caller's intent*, not of the model — the same model serves both an interactive `.video` and a batch film shot, so per-model config in `config.yaml` would be too coarse. A header also leaves the OpenAI-compatible body untouched.
Open: whether clients may freely self-assign large positive values, or whether that needs constraining.
## Observability
Per priority band: queue depth, current wait, jobs dispatched, and **how often aging or swap-affinity changed the dispatch order**. Without the last one, tuning the scoring function is guesswork — and "why did my image take 20 minutes" will be the first question asked the day a 12-hour render is running.
## Consumer side (mort)
- `pkg/lane` passes the number through to the llama-swap client.
- Film shots at `batch`; ordinary `.image` / `.video` at `interactive`; general skill-tool calls at `normal`. Subscription tier adds its bonus on top.
- `generate_long_video` is currently **not** lane-gated at all (deliberately — see the comment in `generate_long_video.go`), so mort's visibility into a running chain is already poor. Worth fixing at the same time.
## Related
- `steve/mort#1567` — long-form video epic, blocked on this
- `steve/mort#1572` — subscription tiers; supplies the tier bonus, caps in-flight GPU jobs per member
- mort ADR-0053 — size-1 GPU lane + reserved code_exec tier
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
The GPU is a size-1 resource — one job at a time. Today llama-swap serves requests in arrival order, so a single long job monopolises the box for its entire duration and every interactive request queues behind it.
This is already uncomfortable (a 16-minute H3 render at 362 frames blocks everything), and it becomes untenable with the planned long-form video work —
steve/mort#1567describes a ~12-hour render made of ~174 sequential H3 shots. Without priorities that job means nobody generates anything for a day.Priority is a signed integer, 0 = normal, higher = more urgent
An absent or unparseable header means 0.
Named aliases may resolve to numbers for convenience, but the number is the interface — callers can use any value.
interactivenormalbatchComposition: bands, offsets, and aging
The dispatcher picks the highest effective priority:
Tier bonus — max always beats pro beats free, within a band
This is the point of using numbers rather than fixed classes. A consumer adds a small per-member offset:
So a max member's batch job is
-100 + 2 = -98, ahead of pro's-99and free's-100— while all three remain below any normal job at 0. Tier breaks ties inside a band and can never promote across one.Invariant: band spacing (100) must stay far larger than the maximum tier + affinity offset (12). Otherwise a max member's batch work could outrank a free member's interactive request, which would be wrong.
Aging is the one term allowed to cross bands
aging = waited_seconds / aging_divisor, unbounded on purpose — a batch job that has waited long enough should eventually beat normal traffic, or it starves. At a divisor of 60 (1 point/minute), a max batch job at -98 overtakes a free interactive job at +100 after ~3.3 hours of waiting.That asymmetry is the design: bounded offsets express policy, unbounded aging prevents starvation.
Swap affinity keeps the model cache from thrashing
Cold loads cost 2-3x on this host. A job needing the already-resident model gets a bounded bonus, so strict priority does not force a swap on every dispatch. Bounded, so it can never promote batch above interactive — it only breaks near-ties.
All four terms fold into one score. Three independent rules interacting produce behaviour nobody can predict.
The hard parts
Non-preemptive is probably the only honest option
A single H3 generation is 4-16 minutes and cannot be interrupted mid-sample without discarding the work. Priority applies at dispatch time — when a job finishes, pick the best-scoring waiting one.
Consequence worth stating plainly: an interactive request can still wait up to one full job duration. Priority alone does not fix that. The complementary lever is client-side — a batch producer submits one unit at a time and re-queues, so gaps are frequent. For long-form video that means one ~4-minute shot per submission, not a 12-hour chain (
steve/mort#1567Phase 2).If real preemption is ever wanted, the only sane form is checkpoint-at-step-boundary and requeue — much larger, probably not worth it.
Fairness vs priority
Priority is not fair-share. If one member submits 200 batch shots, per-member fairness is still needed within a priority level, or one person's film starves another's. mort's
pkg/lanealready implements fair-share queueing — look at it before inventing a second policy.mort also caps one in-flight GPU job per member (
steve/mort#1572), which bounds concurrency client-side and reduces how much fairness the scheduler must enforce itself.API surface
A header, since priority is a property of the caller's intent, not of the model — the same model serves both an interactive
.videoand a batch film shot, so per-model config inconfig.yamlwould be too coarse. A header also leaves the OpenAI-compatible body untouched.Open: whether clients may freely self-assign large positive values, or whether that needs constraining.
Observability
Per priority band: queue depth, current wait, jobs dispatched, and how often aging or swap-affinity changed the dispatch order. Without the last one, tuning the scoring function is guesswork — and "why did my image take 20 minutes" will be the first question asked the day a 12-hour render is running.
Consumer side (mort)
pkg/lanepasses the number through to the llama-swap client.batch; ordinary.image/.videoatinteractive; general skill-tool calls atnormal. Subscription tier adds its bonus on top.generate_long_videois currently not lane-gated at all (deliberately — see the comment ingenerate_long_video.go), so mort's visibility into a running chain is already poor. Worth fixing at the same time.Related
steve/mort#1567— long-form video epic, blocked on thissteve/mort#1572— subscription tiers; supplies the tier bonus, caps in-flight GPU jobs per member