Epic: move the image + video engines onto ComfyUI, keep llama-swap as the front door #3

Open
opened 2026-08-06 00:50:53 +00:00 by steve · 1 comment
Owner

Tracking issue for the ComfyUI migration investigated 2026-08-05. Phases are filed as separate issues; this one holds the reasoning so it does not get lost.

Motivation

The image and video pipelines currently in use — stable-diffusion.cpp's sd-server for the five imagegen-* models and vLLM-Omni for videogen-wan22-5b — are dead ends for tweaking and for new model support. ComfyUI is where new models land first and where the community tooling (LoRA stacks, ControlNet, IPAdapter, inpainting, regional prompting, in-graph upscale) actually exists.

We already proved the pattern: videogen-minimax-h3 is comfyui-h3:local — headless ComfyUI fronted by a shim.py that translates /v1/videos/sync into the FL2VA node graph. mort needed zero client changes because the engine is picked purely by model id. H3 exists because only ComfyUI could run it.

Verdict: do not recreate llama-swap inside ComfyUI

llama-swap stays the front door and the policy engine. ComfyUI becomes the engine for image + video, one tier down. The three load-bearing concepts stay exactly where they are:

1. The two embedding models stay off the queue, unmoved.
embed-qwen3-0.6b and rerank-bge-v2-m3 are llama-server with -ngl 0, in the cpu-utility group (swap:false, persistent:true), preloaded via hooks.on_startup.preload. ComfyUI has no /v1/embeddings and no cross-encoder /v1/rerank. Two separate llama-server instances is itself load-bearing — --embeddings and --rerank on one server returns all-zero vectors (llama.cpp #20085). Any plan that moves the front door to ComfyUI deletes this capability. Under this plan they do not move at all.

2. The queue stays in llama-swap, and gains per-request priority.
FIFO already has a priority queue — routing.scheduler.settings.fifo.priority is a map[modelID]int and enqueue() (internal/router/scheduler/fifo.go:318) inserts before the first strictly-lower-priority item, preserving arrival order within a tier. It is keyed on model, not request. Per-request priority is filed as its own phase.

ComfyUI's own queue has priority (number, lower = higher; front:true to prepend) but only sees ComfyUI jobs — it cannot rank a video render against a Whisper transcription. Under this plan ComfyUI's queue becomes a depth-1 queue behind llama-swap's.

3. Evict / load-on-demand stays the group planner + TTL.
An explicit exclusivity policy across heterogeneous backends beats a VRAM heuristic inside one process. ComfyUI adds a second, faster tier underneath: intra-process model swaps with a warm CUDA context instead of docker stop + docker run + engine boot.

The unlock: aliases

Verified in this fork's code. One comfyui-image entry with aliases: [imagegen-flux-dev, imagegen-flux-schnell, imagegen-flux-kontext, imagegen-qwen-image, imagegen-qwen-image-edit] and includeAliasesInList: true:

  • llama-swap resolves any alias to the one warm process (internal/config/config.go:190)
  • majordomo already sends model: imagegen-flux-dev in the /sdapi/v1/txt2img body — the shim reads the same field to pick the graph
  • /v1/models lists all five ids (internal/server/api.go:173), and mort's classifyModel works off id prefixes, so image_model_guide keeps working
  • mort needs no changes at all — not the tools, not the convars, not imageModelProfiles

Known cosmetic caveats: aliases inherit the parent's name/description/capabilities in the listing, and /running reports comfyui-image rather than the alias.

What it buys

  • 66 GB of weights reused as-is. /srv/models/sd/{diffusion_models,clip,vae,checkpoints} already is ComfyUI's folder taxonomy, and the five image models are GGUF — read by city96's ComfyUI-GGUF (UnetLoaderGGUF / CLIPLoaderGGUF). Nothing to re-download.
  • Actual tweakability. Today sampler/steps/cfg are baked into the docker cmd (--cfg-scale 1 --guidance 3.5 --steps 20 --sampling-method euler). /sdapi exposes no LoRA stacking, no ControlNet, no IPAdapter, no inpaint refinement, no in-graph upscale.
  • Warm swaps between image models. Measured from live logs: warm flux-dev is 10–33s, cold-with-swap is 45s–3m.
  • Wan 2.2 TI2V-5B is natively supported in ComfyUI with official templates — opens LightX2V speed LoRAs and VACE, and drops a 30.9 GB vLLM-Omni image.

What it costs

  • One process, one venv, one crash domain. Today a wedged sd-server is docker stopped by cmdStop. Mitigated by keeping image and video as separate ComfyUI images, not one mega-comfy.
  • RAM. 46 GB total, 41 in use with H3 resident. ComfyUI's default RAM model cache will thrash — needs --cache-none or the LRU cap plus --reserve-vram tuning.
  • The shim is the work. majordomo deliberately uses /sdapi/v1/txt2img + /sdapi/v1/img2img rather than the OpenAI route because sd-server drops seed there (majordomo provider/llamaswap/image.go:32). The shim must honour seed, batch_count, sample_method, and extra_images (Kontext / Qwen-Image-Edit ref-image conditioning).

Pre-existing breakage — do not attribute to the migration

On 2026-08-04 between 22:10 and 22:43 there were four consecutive 500/502s on /sdapi/v1/txt2img with 9m42s–10m24s durations, plus a dial tcp: lookup imagegen-flux-dev on 127.0.0.11:53: server misbehaving DNS race between docker run and the health check. Root-cause independently, or ComfyUI will get the blame for it.

Explicitly out of scope

The six LLMs, the embedding pair, and the reranker stay on llama-server. Whisper / WhisperX / Surya OCR / SadTalker / Hunyuan3D / Demucs all have ComfyUI nodes, but each drags a heavy pinned dependency tree into a shared venv for no capability gain.

Host facts (netherstorm, 2026-08-05)

  • RTX 4090 24 GB, driver 580.173.02; 46 GB RAM, 7 GB swap
  • /srv/models = /dev/sdb, 984 GB, 615 GB used, 319 GB free
  • 31 model ids: 26 in the exclusive all group, 2 in cpu-utility (persistent), 3 in cpu-ondemand
  • Scheduler is fifo, explicitly overriding the fork's serial default (serial ignores groups and would evict the persistent CPU pair)
  • llama-swap shells out to the host docker CLI via a bind-mounted socket; cmdStop is mandatory on container-backed entries or VRAM never releases
  • Config lives in steve/steveternet at azeroth/outland/netherstorm/llama-swap/
Tracking issue for the ComfyUI migration investigated 2026-08-05. Phases are filed as separate issues; this one holds the reasoning so it does not get lost. ## Motivation The image and video pipelines currently in use — `stable-diffusion.cpp`'s `sd-server` for the five `imagegen-*` models and vLLM-Omni for `videogen-wan22-5b` — are dead ends for tweaking and for new model support. ComfyUI is where new models land first and where the community tooling (LoRA stacks, ControlNet, IPAdapter, inpainting, regional prompting, in-graph upscale) actually exists. We already proved the pattern: `videogen-minimax-h3` is `comfyui-h3:local` — headless ComfyUI fronted by a `shim.py` that translates `/v1/videos/sync` into the FL2VA node graph. mort needed **zero** client changes because the engine is picked purely by model id. H3 exists *because* only ComfyUI could run it. ## Verdict: do not recreate llama-swap inside ComfyUI llama-swap stays the front door and the policy engine. ComfyUI becomes the *engine* for image + video, one tier down. The three load-bearing concepts stay exactly where they are: **1. The two embedding models stay off the queue, unmoved.** `embed-qwen3-0.6b` and `rerank-bge-v2-m3` are llama-server with `-ngl 0`, in the `cpu-utility` group (`swap:false, persistent:true`), preloaded via `hooks.on_startup.preload`. ComfyUI has no `/v1/embeddings` and no cross-encoder `/v1/rerank`. Two separate llama-server instances is itself load-bearing — `--embeddings` and `--rerank` on one server returns all-zero vectors (llama.cpp #20085). Any plan that moves the front door to ComfyUI deletes this capability. Under this plan they do not move at all. **2. The queue stays in llama-swap, and gains per-request priority.** `FIFO` already has a priority queue — `routing.scheduler.settings.fifo.priority` is a `map[modelID]int` and `enqueue()` (internal/router/scheduler/fifo.go:318) inserts before the first strictly-lower-priority item, preserving arrival order within a tier. It is keyed on *model*, not *request*. Per-request priority is filed as its own phase. ComfyUI's own queue has priority (`number`, lower = higher; `front:true` to prepend) but only sees ComfyUI jobs — it cannot rank a video render against a Whisper transcription. Under this plan ComfyUI's queue becomes a depth-1 queue behind llama-swap's. **3. Evict / load-on-demand stays the group planner + TTL.** An explicit exclusivity policy across heterogeneous backends beats a VRAM heuristic inside one process. ComfyUI adds a *second, faster tier* underneath: intra-process model swaps with a warm CUDA context instead of `docker stop` + `docker run` + engine boot. ## The unlock: aliases Verified in this fork's code. One `comfyui-image` entry with `aliases: [imagegen-flux-dev, imagegen-flux-schnell, imagegen-flux-kontext, imagegen-qwen-image, imagegen-qwen-image-edit]` and `includeAliasesInList: true`: - llama-swap resolves any alias to the one warm process (`internal/config/config.go:190`) - majordomo already sends `model: imagegen-flux-dev` in the `/sdapi/v1/txt2img` body — the shim reads the same field to pick the graph - `/v1/models` lists all five ids (`internal/server/api.go:173`), and mort's `classifyModel` works off id prefixes, so `image_model_guide` keeps working - **mort needs no changes at all** — not the tools, not the convars, not `imageModelProfiles` Known cosmetic caveats: aliases inherit the parent's `name`/`description`/`capabilities` in the listing, and `/running` reports `comfyui-image` rather than the alias. ## What it buys - **66 GB of weights reused as-is.** `/srv/models/sd/{diffusion_models,clip,vae,checkpoints}` already *is* ComfyUI's folder taxonomy, and the five image models are GGUF — read by city96's `ComfyUI-GGUF` (`UnetLoaderGGUF` / `CLIPLoaderGGUF`). Nothing to re-download. - **Actual tweakability.** Today sampler/steps/cfg are baked into the docker `cmd` (`--cfg-scale 1 --guidance 3.5 --steps 20 --sampling-method euler`). `/sdapi` exposes no LoRA stacking, no ControlNet, no IPAdapter, no inpaint refinement, no in-graph upscale. - **Warm swaps between image models.** Measured from live logs: warm flux-dev is 10–33s, cold-with-swap is 45s–3m. - **Wan 2.2 TI2V-5B is natively supported in ComfyUI** with official templates — opens LightX2V speed LoRAs and VACE, and drops a 30.9 GB vLLM-Omni image. ## What it costs - **One process, one venv, one crash domain.** Today a wedged `sd-server` is `docker stop`ped by `cmdStop`. Mitigated by keeping image and video as *separate* ComfyUI images, not one mega-comfy. - **RAM.** 46 GB total, 41 in use with H3 resident. ComfyUI's default RAM model cache will thrash — needs `--cache-none` or the LRU cap plus `--reserve-vram` tuning. - **The shim is the work.** majordomo deliberately uses `/sdapi/v1/txt2img` + `/sdapi/v1/img2img` rather than the OpenAI route *because sd-server drops `seed` there* (majordomo `provider/llamaswap/image.go:32`). The shim must honour `seed`, `batch_count`, `sample_method`, and `extra_images` (Kontext / Qwen-Image-Edit ref-image conditioning). ## Pre-existing breakage — do not attribute to the migration On 2026-08-04 between 22:10 and 22:43 there were four consecutive `500`/`502`s on `/sdapi/v1/txt2img` with 9m42s–10m24s durations, plus a `dial tcp: lookup imagegen-flux-dev on 127.0.0.11:53: server misbehaving` DNS race between `docker run` and the health check. Root-cause independently, or ComfyUI will get the blame for it. ## Explicitly out of scope The six LLMs, the embedding pair, and the reranker stay on llama-server. Whisper / WhisperX / Surya OCR / SadTalker / Hunyuan3D / Demucs all have ComfyUI nodes, but each drags a heavy pinned dependency tree into a shared venv for no capability gain. ## Host facts (netherstorm, 2026-08-05) - RTX 4090 24 GB, driver 580.173.02; 46 GB RAM, 7 GB swap - `/srv/models` = `/dev/sdb`, 984 GB, 615 GB used, 319 GB free - 31 model ids: 26 in the exclusive `all` group, 2 in `cpu-utility` (persistent), 3 in `cpu-ondemand` - Scheduler is `fifo`, explicitly overriding the fork's `serial` default (serial ignores groups and would evict the persistent CPU pair) - llama-swap shells out to the host docker CLI via a bind-mounted socket; `cmdStop` is mandatory on container-backed entries or VRAM never releases - Config lives in `steve/steveternet` at `azeroth/outland/netherstorm/llama-swap/`
Author
Owner

Phase index

# Phase Depends on Touches mort?
#4 Stand up comfyui-image beside sd-server and A/B it no
#5 Flip the imagegen-* aliases, delete the sd-server entries #4 no (alias trick)
#6 Move Wan 2.2 TI2V-5B off vLLM-Omni onto ComfyUI #4 re-measure cost curve only
#7 Per-request priority in the fifo scheduler (X-Priority) — (orthogonal) yes, majordomo sends the header
#8 Fold mediautils / rembg / segment-langsam into the image comfy (optional) #5 only if the shim drops /upstream/... paths

#7 depends on nothing and can land at any point.

Priority semantics settled 2026-08-05: no preemption. An in-progress low-priority job keeps the machine until it finishes; priority only decides which waiting request gets picked up next.

## Phase index | # | Phase | Depends on | Touches mort? | |---|---|---|---| | #4 | Stand up `comfyui-image` beside sd-server and A/B it | — | no | | #5 | Flip the `imagegen-*` aliases, delete the sd-server entries | #4 | **no** (alias trick) | | #6 | Move Wan 2.2 TI2V-5B off vLLM-Omni onto ComfyUI | #4 | re-measure cost curve only | | #7 | Per-request priority in the fifo scheduler (`X-Priority`) | — (orthogonal) | yes, majordomo sends the header | | #8 | Fold `mediautils` / `rembg` / `segment-langsam` into the image comfy (optional) | #5 | only if the shim drops `/upstream/...` paths | #7 depends on nothing and can land at any point. Priority semantics settled 2026-08-05: **no preemption.** An in-progress low-priority job keeps the machine until it finishes; priority only decides which *waiting* request gets picked up next.
Sign in to join this conversation.
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: steve/llama-swap#3