majordomo/docs/adr/0020-upstream-passthrough-media-surfaces.md

ADR-0020: Upstream-passthrough media surfaces (mask, upscale, background removal, interpolation, diarization, meshgen)

Status: Accepted (2026-07-12)

Context

The llama-swap host is growing capabilities whose native HTTP APIs are not OpenAI-shaped and carry no routable model field: rembg (/api/remove), a Real-ESRGAN+RIFE shim (/v1/upscale, /v1/interpolate), a WhisperX diarization sidecar (/asr?diarize=true), and Hunyuan3D (/generate). llama-swap's fork already exposes a generic passthrough — /upstream/<model>/<any path> — that pins the model, runs the normal load/swap queue, and proxies the rest of the path verbatim.

Separately, sd-server's /sdapi/v1/img2img accepts an inpainting mask field that imagegen.EditRequest could not express.

Decision

1. Route odd-shaped upstreams through /upstream/<model>/<path> via a shared upstreamPath helper (model-id validation identical to Unload: reject /?#, never escape). No per-endpoint llama-swap routes; the passthrough is the contract.
2. Grow the existing leaf contracts instead of inventing parallel ones, keeping the ADR-0016→0019 conventions (functional options, zero value = backend default, bytes-only I/O, Raw escape hatch, provider mints model):
   • imagegen.EditRequest.Mask (white = repaint, black = keep; backends without mask support must reject, not ignore).
   • imagegen.Upscaler / UpscaleProvider — super-resolution is not a diffusion model, so it binds its own backend id.
   • imagegen.BackgroundRemover / BackgroundRemovalProvider — request Net selects the remover's internal network (rembg model= param); OnlyMask returns the segmentation mask (the natural mask source for inpainting).
   • videogen.Interpolator / InterpolationProvider — fps boost or slow-mo (SlowMo plays synthesized frames at the source rate; audio stripped by the backend in that mode).
   • audio.DiarizationModel / DiarizationProvider — speaker-labelled segments; per-file SPEAKER_NN labels are explicitly NOT stable identities.
3. New meshgen leaf package for image→3D (Mesh{Data, Format, MIME}, formats glb/stl/obj). Image-to-3D only: text-to-3D is the caller-owned composition imagegen → meshgen.
4. Binary success bodies are validated before wrapping: one-image results reject non-image payloads, interpolation reuses the videogen MIME rules and 512MB cap, meshes reject JSON-shaped "success" bodies (a queue-full detail page must never become "the mesh"), and diarization requires output=json because the vtt/srt shapes drop speaker labels.

Consequences

• provider/llamaswap gains five surfaces with no new wire machinery beyond upstreamPath + a shared one-file-multipart builder.
• The passthrough couples majordomo to the fork's /upstream route (upstream llama-swap has it too) and to each upstream's native API shape; those shapes are pinned by the netherstorm image builds, not by version negotiation — smoke tests on the host are the drift defence.
• upstream.ignorePaths (llama-swap config) can 409 asset-looking paths on cold models; none of the paths used here match the default pattern, but new surfaces must check.