# ADR-0020: Upstream-passthrough media surfaces (mask, upscale, background removal, interpolation, diarization, meshgen)

Status: Accepted (2026-07-12)

## Context

The llama-swap host is growing capabilities whose native HTTP APIs are not
OpenAI-shaped and carry no routable `model` field: rembg (`/api/remove`),
a Real-ESRGAN+RIFE shim (`/v1/upscale`, `/v1/interpolate`), a WhisperX
diarization sidecar (`/asr?diarize=true`), and Hunyuan3D (`/generate`).
llama-swap's fork already exposes a generic passthrough —
`/upstream/<model>/<any path>` — that pins the model, runs the normal
load/swap queue, and proxies the rest of the path verbatim.

Separately, sd-server's `/sdapi/v1/img2img` accepts an inpainting `mask`
field that `imagegen.EditRequest` could not express.

## Decision

1. **Route odd-shaped upstreams through `/upstream/<model>/<path>`** via a
   shared `upstreamPath` helper (model-id validation identical to `Unload`:
   reject `/?#`, never escape). No per-endpoint llama-swap routes; the
   passthrough is the contract.
2. **Grow the existing leaf contracts instead of inventing parallel ones**,
   keeping the ADR-0016→0019 conventions (functional options, zero value =
   backend default, bytes-only I/O, `Raw` escape hatch, provider mints
   model):
   - `imagegen.EditRequest.Mask` (white = repaint, black = keep; backends
     without mask support must reject, not ignore).
   - `imagegen.Upscaler` / `UpscaleProvider` — super-resolution is not a
     diffusion model, so it binds its own backend id.
   - `imagegen.BackgroundRemover` / `BackgroundRemovalProvider` — request
     `Net` selects the remover's internal network (rembg `model=` param);
     `OnlyMask` returns the segmentation mask (the natural mask source for
     inpainting).
   - `videogen.Interpolator` / `InterpolationProvider` — fps boost or
     slow-mo (`SlowMo` plays synthesized frames at the source rate; audio
     stripped by the backend in that mode).
   - `audio.DiarizationModel` / `DiarizationProvider` — speaker-labelled
     segments; per-file `SPEAKER_NN` labels are explicitly NOT stable
     identities.
3. **New `meshgen` leaf package** for image→3D (`Mesh{Data, Format, MIME}`,
   formats glb/stl/obj). Image-to-3D only: text-to-3D is the caller-owned
   composition imagegen → meshgen.
4. **Binary success bodies are validated before wrapping**: one-image
   results reject non-image payloads, interpolation reuses the videogen
   MIME rules and 512MB cap, meshes reject JSON-shaped "success" bodies
   (a queue-full detail page must never become "the mesh"), and
   diarization requires `output=json` because the vtt/srt shapes drop
   speaker labels.

## Consequences

- provider/llamaswap gains five surfaces with no new wire machinery beyond
  `upstreamPath` + a shared one-file-multipart builder.
- The passthrough couples majordomo to the fork's `/upstream` route
  (upstream llama-swap has it too) and to each upstream's native API shape;
  those shapes are pinned by the netherstorm image builds, not by version
  negotiation — smoke tests on the host are the drift defence.
- `upstream.ignorePaths` (llama-swap config) can 409 asset-looking paths on
  cold models; none of the paths used here match the default pattern, but
  new surfaces must check.