Tracking issue for the ComfyUI migration investigated 2026-08-05. Phases are filed as separate issues; this one holds the reasoning so it does not get lost.
Motivation
The image and video pipelines currently in use — stable-diffusion.cpp's sd-server for the five imagegen-* models and vLLM-Omni for videogen-wan22-5b — are dead ends for tweaking and for new model support. ComfyUI is where new models land first and where the community tooling (LoRA stacks, ControlNet, IPAdapter, inpainting, regional prompting, in-graph upscale) actually exists.
We already proved the pattern: videogen-minimax-h3 is comfyui-h3:local — headless ComfyUI fronted by a shim.py that translates /v1/videos/sync into the FL2VA node graph. mort needed zero client changes because the engine is picked purely by model id. H3 exists because only ComfyUI could run it.
Verdict: do not recreate llama-swap inside ComfyUI
llama-swap stays the front door and the policy engine. ComfyUI becomes the engine for image + video, one tier down. The three load-bearing concepts stay exactly where they are:
1. The two embedding models stay off the queue, unmoved. embed-qwen3-0.6b and rerank-bge-v2-m3 are llama-server with -ngl 0, in the cpu-utility group (swap:false, persistent:true), preloaded via hooks.on_startup.preload. ComfyUI has no /v1/embeddings and no cross-encoder /v1/rerank. Two separate llama-server instances is itself load-bearing — --embeddings and --rerank on one server returns all-zero vectors (llama.cpp #20085). Any plan that moves the front door to ComfyUI deletes this capability. Under this plan they do not move at all.
2. The queue stays in llama-swap, and gains per-request priority. FIFO already has a priority queue — routing.scheduler.settings.fifo.priority is a map[modelID]int and enqueue() (internal/router/scheduler/fifo.go:318) inserts before the first strictly-lower-priority item, preserving arrival order within a tier. It is keyed on model, not request. Per-request priority is filed as its own phase.
ComfyUI's own queue has priority (number, lower = higher; front:true to prepend) but only sees ComfyUI jobs — it cannot rank a video render against a Whisper transcription. Under this plan ComfyUI's queue becomes a depth-1 queue behind llama-swap's.
3. Evict / load-on-demand stays the group planner + TTL.
An explicit exclusivity policy across heterogeneous backends beats a VRAM heuristic inside one process. ComfyUI adds a second, faster tier underneath: intra-process model swaps with a warm CUDA context instead of docker stop + docker run + engine boot.
The unlock: aliases
Verified in this fork's code. One comfyui-image entry with aliases: [imagegen-flux-dev, imagegen-flux-schnell, imagegen-flux-kontext, imagegen-qwen-image, imagegen-qwen-image-edit] and includeAliasesInList: true:
llama-swap resolves any alias to the one warm process (internal/config/config.go:190)
majordomo already sends model: imagegen-flux-dev in the /sdapi/v1/txt2img body — the shim reads the same field to pick the graph
/v1/models lists all five ids (internal/server/api.go:173), and mort's classifyModel works off id prefixes, so image_model_guide keeps working
mort needs no changes at all — not the tools, not the convars, not imageModelProfiles
Known cosmetic caveats: aliases inherit the parent's name/description/capabilities in the listing, and /running reports comfyui-image rather than the alias.
What it buys
66 GB of weights reused as-is./srv/models/sd/{diffusion_models,clip,vae,checkpoints} already is ComfyUI's folder taxonomy, and the five image models are GGUF — read by city96's ComfyUI-GGUF (UnetLoaderGGUF / CLIPLoaderGGUF). Nothing to re-download.
Actual tweakability. Today sampler/steps/cfg are baked into the docker cmd (--cfg-scale 1 --guidance 3.5 --steps 20 --sampling-method euler). /sdapi exposes no LoRA stacking, no ControlNet, no IPAdapter, no inpaint refinement, no in-graph upscale.
Warm swaps between image models. Measured from live logs: warm flux-dev is 10–33s, cold-with-swap is 45s–3m.
Wan 2.2 TI2V-5B is natively supported in ComfyUI with official templates — opens LightX2V speed LoRAs and VACE, and drops a 30.9 GB vLLM-Omni image.
What it costs
One process, one venv, one crash domain. Today a wedged sd-server is docker stopped by cmdStop. Mitigated by keeping image and video as separate ComfyUI images, not one mega-comfy.
RAM. 46 GB total, 41 in use with H3 resident. ComfyUI's default RAM model cache will thrash — needs --cache-none or the LRU cap plus --reserve-vram tuning.
The shim is the work. majordomo deliberately uses /sdapi/v1/txt2img + /sdapi/v1/img2img rather than the OpenAI route because sd-server drops seed there (majordomo provider/llamaswap/image.go:32). The shim must honour seed, batch_count, sample_method, and extra_images (Kontext / Qwen-Image-Edit ref-image conditioning).
Pre-existing breakage — do not attribute to the migration
On 2026-08-04 between 22:10 and 22:43 there were four consecutive 500/502s on /sdapi/v1/txt2img with 9m42s–10m24s durations, plus a dial tcp: lookup imagegen-flux-dev on 127.0.0.11:53: server misbehaving DNS race between docker run and the health check. Root-cause independently, or ComfyUI will get the blame for it.
Explicitly out of scope
The six LLMs, the embedding pair, and the reranker stay on llama-server. Whisper / WhisperX / Surya OCR / SadTalker / Hunyuan3D / Demucs all have ComfyUI nodes, but each drags a heavy pinned dependency tree into a shared venv for no capability gain.
31 model ids: 26 in the exclusive all group, 2 in cpu-utility (persistent), 3 in cpu-ondemand
Scheduler is fifo, explicitly overriding the fork's serial default (serial ignores groups and would evict the persistent CPU pair)
llama-swap shells out to the host docker CLI via a bind-mounted socket; cmdStop is mandatory on container-backed entries or VRAM never releases
Config lives in steve/steveternet at azeroth/outland/netherstorm/llama-swap/
Tracking issue for the ComfyUI migration investigated 2026-08-05. Phases are filed as separate issues; this one holds the reasoning so it does not get lost.
## Motivation
The image and video pipelines currently in use — `stable-diffusion.cpp`'s `sd-server` for the five `imagegen-*` models and vLLM-Omni for `videogen-wan22-5b` — are dead ends for tweaking and for new model support. ComfyUI is where new models land first and where the community tooling (LoRA stacks, ControlNet, IPAdapter, inpainting, regional prompting, in-graph upscale) actually exists.
We already proved the pattern: `videogen-minimax-h3` is `comfyui-h3:local` — headless ComfyUI fronted by a `shim.py` that translates `/v1/videos/sync` into the FL2VA node graph. mort needed **zero** client changes because the engine is picked purely by model id. H3 exists *because* only ComfyUI could run it.
## Verdict: do not recreate llama-swap inside ComfyUI
llama-swap stays the front door and the policy engine. ComfyUI becomes the *engine* for image + video, one tier down. The three load-bearing concepts stay exactly where they are:
**1. The two embedding models stay off the queue, unmoved.**
`embed-qwen3-0.6b` and `rerank-bge-v2-m3` are llama-server with `-ngl 0`, in the `cpu-utility` group (`swap:false, persistent:true`), preloaded via `hooks.on_startup.preload`. ComfyUI has no `/v1/embeddings` and no cross-encoder `/v1/rerank`. Two separate llama-server instances is itself load-bearing — `--embeddings` and `--rerank` on one server returns all-zero vectors (llama.cpp #20085). Any plan that moves the front door to ComfyUI deletes this capability. Under this plan they do not move at all.
**2. The queue stays in llama-swap, and gains per-request priority.**
`FIFO` already has a priority queue — `routing.scheduler.settings.fifo.priority` is a `map[modelID]int` and `enqueue()` (internal/router/scheduler/fifo.go:318) inserts before the first strictly-lower-priority item, preserving arrival order within a tier. It is keyed on *model*, not *request*. Per-request priority is filed as its own phase.
ComfyUI's own queue has priority (`number`, lower = higher; `front:true` to prepend) but only sees ComfyUI jobs — it cannot rank a video render against a Whisper transcription. Under this plan ComfyUI's queue becomes a depth-1 queue behind llama-swap's.
**3. Evict / load-on-demand stays the group planner + TTL.**
An explicit exclusivity policy across heterogeneous backends beats a VRAM heuristic inside one process. ComfyUI adds a *second, faster tier* underneath: intra-process model swaps with a warm CUDA context instead of `docker stop` + `docker run` + engine boot.
## The unlock: aliases
Verified in this fork's code. One `comfyui-image` entry with `aliases: [imagegen-flux-dev, imagegen-flux-schnell, imagegen-flux-kontext, imagegen-qwen-image, imagegen-qwen-image-edit]` and `includeAliasesInList: true`:
- llama-swap resolves any alias to the one warm process (`internal/config/config.go:190`)
- majordomo already sends `model: imagegen-flux-dev` in the `/sdapi/v1/txt2img` body — the shim reads the same field to pick the graph
- `/v1/models` lists all five ids (`internal/server/api.go:173`), and mort's `classifyModel` works off id prefixes, so `image_model_guide` keeps working
- **mort needs no changes at all** — not the tools, not the convars, not `imageModelProfiles`
Known cosmetic caveats: aliases inherit the parent's `name`/`description`/`capabilities` in the listing, and `/running` reports `comfyui-image` rather than the alias.
## What it buys
- **66 GB of weights reused as-is.** `/srv/models/sd/{diffusion_models,clip,vae,checkpoints}` already *is* ComfyUI's folder taxonomy, and the five image models are GGUF — read by city96's `ComfyUI-GGUF` (`UnetLoaderGGUF` / `CLIPLoaderGGUF`). Nothing to re-download.
- **Actual tweakability.** Today sampler/steps/cfg are baked into the docker `cmd` (`--cfg-scale 1 --guidance 3.5 --steps 20 --sampling-method euler`). `/sdapi` exposes no LoRA stacking, no ControlNet, no IPAdapter, no inpaint refinement, no in-graph upscale.
- **Warm swaps between image models.** Measured from live logs: warm flux-dev is 10–33s, cold-with-swap is 45s–3m.
- **Wan 2.2 TI2V-5B is natively supported in ComfyUI** with official templates — opens LightX2V speed LoRAs and VACE, and drops a 30.9 GB vLLM-Omni image.
## What it costs
- **One process, one venv, one crash domain.** Today a wedged `sd-server` is `docker stop`ped by `cmdStop`. Mitigated by keeping image and video as *separate* ComfyUI images, not one mega-comfy.
- **RAM.** 46 GB total, 41 in use with H3 resident. ComfyUI's default RAM model cache will thrash — needs `--cache-none` or the LRU cap plus `--reserve-vram` tuning.
- **The shim is the work.** majordomo deliberately uses `/sdapi/v1/txt2img` + `/sdapi/v1/img2img` rather than the OpenAI route *because sd-server drops `seed` there* (majordomo `provider/llamaswap/image.go:32`). The shim must honour `seed`, `batch_count`, `sample_method`, and `extra_images` (Kontext / Qwen-Image-Edit ref-image conditioning).
## Pre-existing breakage — do not attribute to the migration
On 2026-08-04 between 22:10 and 22:43 there were four consecutive `500`/`502`s on `/sdapi/v1/txt2img` with 9m42s–10m24s durations, plus a `dial tcp: lookup imagegen-flux-dev on 127.0.0.11:53: server misbehaving` DNS race between `docker run` and the health check. Root-cause independently, or ComfyUI will get the blame for it.
## Explicitly out of scope
The six LLMs, the embedding pair, and the reranker stay on llama-server. Whisper / WhisperX / Surya OCR / SadTalker / Hunyuan3D / Demucs all have ComfyUI nodes, but each drags a heavy pinned dependency tree into a shared venv for no capability gain.
## Host facts (netherstorm, 2026-08-05)
- RTX 4090 24 GB, driver 580.173.02; 46 GB RAM, 7 GB swap
- `/srv/models` = `/dev/sdb`, 984 GB, 615 GB used, 319 GB free
- 31 model ids: 26 in the exclusive `all` group, 2 in `cpu-utility` (persistent), 3 in `cpu-ondemand`
- Scheduler is `fifo`, explicitly overriding the fork's `serial` default (serial ignores groups and would evict the persistent CPU pair)
- llama-swap shells out to the host docker CLI via a bind-mounted socket; `cmdStop` is mandatory on container-backed entries or VRAM never releases
- Config lives in `steve/steveternet` at `azeroth/outland/netherstorm/llama-swap/`
Priority semantics settled 2026-08-05: no preemption. An in-progress low-priority job keeps the machine until it finishes; priority only decides which waiting request gets picked up next.
## Phase index
| # | Phase | Depends on | Touches mort? |
|---|---|---|---|
| #4 | Stand up `comfyui-image` beside sd-server and A/B it | — | no |
| #5 | Flip the `imagegen-*` aliases, delete the sd-server entries | #4 | **no** (alias trick) |
| #6 | Move Wan 2.2 TI2V-5B off vLLM-Omni onto ComfyUI | #4 | re-measure cost curve only |
| #7 | Per-request priority in the fifo scheduler (`X-Priority`) | — (orthogonal) | yes, majordomo sends the header |
| #8 | Fold `mediautils` / `rembg` / `segment-langsam` into the image comfy (optional) | #5 | only if the shim drops `/upstream/...` paths |
#7 depends on nothing and can land at any point.
Priority semantics settled 2026-08-05: **no preemption.** An in-progress low-priority job keeps the machine until it finishes; priority only decides which *waiting* request gets picked up next.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Tracking issue for the ComfyUI migration investigated 2026-08-05. Phases are filed as separate issues; this one holds the reasoning so it does not get lost.
Motivation
The image and video pipelines currently in use —
stable-diffusion.cpp'ssd-serverfor the fiveimagegen-*models and vLLM-Omni forvideogen-wan22-5b— are dead ends for tweaking and for new model support. ComfyUI is where new models land first and where the community tooling (LoRA stacks, ControlNet, IPAdapter, inpainting, regional prompting, in-graph upscale) actually exists.We already proved the pattern:
videogen-minimax-h3iscomfyui-h3:local— headless ComfyUI fronted by ashim.pythat translates/v1/videos/syncinto the FL2VA node graph. mort needed zero client changes because the engine is picked purely by model id. H3 exists because only ComfyUI could run it.Verdict: do not recreate llama-swap inside ComfyUI
llama-swap stays the front door and the policy engine. ComfyUI becomes the engine for image + video, one tier down. The three load-bearing concepts stay exactly where they are:
1. The two embedding models stay off the queue, unmoved.
embed-qwen3-0.6bandrerank-bge-v2-m3are llama-server with-ngl 0, in thecpu-utilitygroup (swap:false, persistent:true), preloaded viahooks.on_startup.preload. ComfyUI has no/v1/embeddingsand no cross-encoder/v1/rerank. Two separate llama-server instances is itself load-bearing —--embeddingsand--rerankon one server returns all-zero vectors (llama.cpp #20085). Any plan that moves the front door to ComfyUI deletes this capability. Under this plan they do not move at all.2. The queue stays in llama-swap, and gains per-request priority.
FIFOalready has a priority queue —routing.scheduler.settings.fifo.priorityis amap[modelID]intandenqueue()(internal/router/scheduler/fifo.go:318) inserts before the first strictly-lower-priority item, preserving arrival order within a tier. It is keyed on model, not request. Per-request priority is filed as its own phase.ComfyUI's own queue has priority (
number, lower = higher;front:trueto prepend) but only sees ComfyUI jobs — it cannot rank a video render against a Whisper transcription. Under this plan ComfyUI's queue becomes a depth-1 queue behind llama-swap's.3. Evict / load-on-demand stays the group planner + TTL.
An explicit exclusivity policy across heterogeneous backends beats a VRAM heuristic inside one process. ComfyUI adds a second, faster tier underneath: intra-process model swaps with a warm CUDA context instead of
docker stop+docker run+ engine boot.The unlock: aliases
Verified in this fork's code. One
comfyui-imageentry withaliases: [imagegen-flux-dev, imagegen-flux-schnell, imagegen-flux-kontext, imagegen-qwen-image, imagegen-qwen-image-edit]andincludeAliasesInList: true:internal/config/config.go:190)model: imagegen-flux-devin the/sdapi/v1/txt2imgbody — the shim reads the same field to pick the graph/v1/modelslists all five ids (internal/server/api.go:173), and mort'sclassifyModelworks off id prefixes, soimage_model_guidekeeps workingimageModelProfilesKnown cosmetic caveats: aliases inherit the parent's
name/description/capabilitiesin the listing, and/runningreportscomfyui-imagerather than the alias.What it buys
/srv/models/sd/{diffusion_models,clip,vae,checkpoints}already is ComfyUI's folder taxonomy, and the five image models are GGUF — read by city96'sComfyUI-GGUF(UnetLoaderGGUF/CLIPLoaderGGUF). Nothing to re-download.cmd(--cfg-scale 1 --guidance 3.5 --steps 20 --sampling-method euler)./sdapiexposes no LoRA stacking, no ControlNet, no IPAdapter, no inpaint refinement, no in-graph upscale.What it costs
sd-serverisdocker stopped bycmdStop. Mitigated by keeping image and video as separate ComfyUI images, not one mega-comfy.--cache-noneor the LRU cap plus--reserve-vramtuning./sdapi/v1/txt2img+/sdapi/v1/img2imgrather than the OpenAI route because sd-server dropsseedthere (majordomoprovider/llamaswap/image.go:32). The shim must honourseed,batch_count,sample_method, andextra_images(Kontext / Qwen-Image-Edit ref-image conditioning).Pre-existing breakage — do not attribute to the migration
On 2026-08-04 between 22:10 and 22:43 there were four consecutive
500/502s on/sdapi/v1/txt2imgwith 9m42s–10m24s durations, plus adial tcp: lookup imagegen-flux-dev on 127.0.0.11:53: server misbehavingDNS race betweendocker runand the health check. Root-cause independently, or ComfyUI will get the blame for it.Explicitly out of scope
The six LLMs, the embedding pair, and the reranker stay on llama-server. Whisper / WhisperX / Surya OCR / SadTalker / Hunyuan3D / Demucs all have ComfyUI nodes, but each drags a heavy pinned dependency tree into a shared venv for no capability gain.
Host facts (netherstorm, 2026-08-05)
/srv/models=/dev/sdb, 984 GB, 615 GB used, 319 GB freeallgroup, 2 incpu-utility(persistent), 3 incpu-ondemandfifo, explicitly overriding the fork'sserialdefault (serial ignores groups and would evict the persistent CPU pair)cmdStopis mandatory on container-backed entries or VRAM never releasessteve/steveternetatazeroth/outland/netherstorm/llama-swap/Phase index
comfyui-imagebeside sd-server and A/B itimagegen-*aliases, delete the sd-server entriesX-Priority)mediautils/rembg/segment-langsaminto the image comfy (optional)/upstream/...paths#7 depends on nothing and can land at any point.
Priority semantics settled 2026-08-05: no preemption. An in-progress low-priority job keeps the machine until it finishes; priority only decides which waiting request gets picked up next.