steveandClaude Fable 5
e98493bcfb
- imagegen.EditRequest.Mask -> sd-server img2img inpainting (white=repaint) - imagegen.Upscaler + BackgroundRemover, videogen.Interpolator, audio.DiarizationModel: new optional provider-minted surfaces - NEW meshgen leaf package (image->3D, glb/stl/obj) - provider/llamaswap: all five via the /upstream/<model>/<path> passthrough (upstreamPath helper, shared one-file multipart builder); binary success bodies validated (non-image, non-video, JSON-mesh rejection); diarization pins output=json (vtt/srt drop speaker labels) Co-Authored-By: Claude Fable 5 <[email protected]>
63 lines
3.2 KiB
Markdown
63 lines
3.2 KiB
Markdown
# ADR-0020: Upstream-passthrough media surfaces (mask, upscale, background removal, interpolation, diarization, meshgen)
|
|
|
|
Status: Accepted (2026-07-12)
|
|
|
|
## Context
|
|
|
|
The llama-swap host is growing capabilities whose native HTTP APIs are not
|
|
OpenAI-shaped and carry no routable `model` field: rembg (`/api/remove`),
|
|
a Real-ESRGAN+RIFE shim (`/v1/upscale`, `/v1/interpolate`), a WhisperX
|
|
diarization sidecar (`/asr?diarize=true`), and Hunyuan3D (`/generate`).
|
|
llama-swap's fork already exposes a generic passthrough —
|
|
`/upstream/<model>/<any path>` — that pins the model, runs the normal
|
|
load/swap queue, and proxies the rest of the path verbatim.
|
|
|
|
Separately, sd-server's `/sdapi/v1/img2img` accepts an inpainting `mask`
|
|
field that `imagegen.EditRequest` could not express.
|
|
|
|
## Decision
|
|
|
|
1. **Route odd-shaped upstreams through `/upstream/<model>/<path>`** via a
|
|
shared `upstreamPath` helper (model-id validation identical to `Unload`:
|
|
reject `/?#`, never escape). No per-endpoint llama-swap routes; the
|
|
passthrough is the contract.
|
|
2. **Grow the existing leaf contracts instead of inventing parallel ones**,
|
|
keeping the ADR-0016→0019 conventions (functional options, zero value =
|
|
backend default, bytes-only I/O, `Raw` escape hatch, provider mints
|
|
model):
|
|
- `imagegen.EditRequest.Mask` (white = repaint, black = keep; backends
|
|
without mask support must reject, not ignore).
|
|
- `imagegen.Upscaler` / `UpscaleProvider` — super-resolution is not a
|
|
diffusion model, so it binds its own backend id.
|
|
- `imagegen.BackgroundRemover` / `BackgroundRemovalProvider` — request
|
|
`Net` selects the remover's internal network (rembg `model=` param);
|
|
`OnlyMask` returns the segmentation mask (the natural mask source for
|
|
inpainting).
|
|
- `videogen.Interpolator` / `InterpolationProvider` — fps boost or
|
|
slow-mo (`SlowMo` plays synthesized frames at the source rate; audio
|
|
stripped by the backend in that mode).
|
|
- `audio.DiarizationModel` / `DiarizationProvider` — speaker-labelled
|
|
segments; per-file `SPEAKER_NN` labels are explicitly NOT stable
|
|
identities.
|
|
3. **New `meshgen` leaf package** for image→3D (`Mesh{Data, Format, MIME}`,
|
|
formats glb/stl/obj). Image-to-3D only: text-to-3D is the caller-owned
|
|
composition imagegen → meshgen.
|
|
4. **Binary success bodies are validated before wrapping**: one-image
|
|
results reject non-image payloads, interpolation reuses the videogen
|
|
MIME rules and 512MB cap, meshes reject JSON-shaped "success" bodies
|
|
(a queue-full detail page must never become "the mesh"), and
|
|
diarization requires `output=json` because the vtt/srt shapes drop
|
|
speaker labels.
|
|
|
|
## Consequences
|
|
|
|
- provider/llamaswap gains five surfaces with no new wire machinery beyond
|
|
`upstreamPath` + a shared one-file-multipart builder.
|
|
- The passthrough couples majordomo to the fork's `/upstream` route
|
|
(upstream llama-swap has it too) and to each upstream's native API shape;
|
|
those shapes are pinned by the netherstorm image builds, not by version
|
|
negotiation — smoke tests on the host are the drift defence.
|
|
- `upstream.ignorePaths` (llama-swap config) can 409 asset-looking paths on
|
|
cold models; none of the paths used here match the default pattern, but
|
|
new surfaces must check.
|