feat: media expansion surfaces — edit mask, upscale, background removal, interpolation, diarization, meshgen (ADR-0020)
- imagegen.EditRequest.Mask -> sd-server img2img inpainting (white=repaint) - imagegen.Upscaler + BackgroundRemover, videogen.Interpolator, audio.DiarizationModel: new optional provider-minted surfaces - NEW meshgen leaf package (image->3D, glb/stl/obj) - provider/llamaswap: all five via the /upstream/<model>/<path> passthrough (upstreamPath helper, shared one-file multipart builder); binary success bodies validated (non-image, non-video, JSON-mesh rejection); diarization pins output=json (vtt/srt drop speaker labels) Co-Authored-By: Claude Fable 5 <[email protected]>
This commit is contained in:
@@ -0,0 +1,62 @@
|
||||
# ADR-0020: Upstream-passthrough media surfaces (mask, upscale, background removal, interpolation, diarization, meshgen)
|
||||
|
||||
Status: Accepted (2026-07-12)
|
||||
|
||||
## Context
|
||||
|
||||
The llama-swap host is growing capabilities whose native HTTP APIs are not
|
||||
OpenAI-shaped and carry no routable `model` field: rembg (`/api/remove`),
|
||||
a Real-ESRGAN+RIFE shim (`/v1/upscale`, `/v1/interpolate`), a WhisperX
|
||||
diarization sidecar (`/asr?diarize=true`), and Hunyuan3D (`/generate`).
|
||||
llama-swap's fork already exposes a generic passthrough —
|
||||
`/upstream/<model>/<any path>` — that pins the model, runs the normal
|
||||
load/swap queue, and proxies the rest of the path verbatim.
|
||||
|
||||
Separately, sd-server's `/sdapi/v1/img2img` accepts an inpainting `mask`
|
||||
field that `imagegen.EditRequest` could not express.
|
||||
|
||||
## Decision
|
||||
|
||||
1. **Route odd-shaped upstreams through `/upstream/<model>/<path>`** via a
|
||||
shared `upstreamPath` helper (model-id validation identical to `Unload`:
|
||||
reject `/?#`, never escape). No per-endpoint llama-swap routes; the
|
||||
passthrough is the contract.
|
||||
2. **Grow the existing leaf contracts instead of inventing parallel ones**,
|
||||
keeping the ADR-0016→0019 conventions (functional options, zero value =
|
||||
backend default, bytes-only I/O, `Raw` escape hatch, provider mints
|
||||
model):
|
||||
- `imagegen.EditRequest.Mask` (white = repaint, black = keep; backends
|
||||
without mask support must reject, not ignore).
|
||||
- `imagegen.Upscaler` / `UpscaleProvider` — super-resolution is not a
|
||||
diffusion model, so it binds its own backend id.
|
||||
- `imagegen.BackgroundRemover` / `BackgroundRemovalProvider` — request
|
||||
`Net` selects the remover's internal network (rembg `model=` param);
|
||||
`OnlyMask` returns the segmentation mask (the natural mask source for
|
||||
inpainting).
|
||||
- `videogen.Interpolator` / `InterpolationProvider` — fps boost or
|
||||
slow-mo (`SlowMo` plays synthesized frames at the source rate; audio
|
||||
stripped by the backend in that mode).
|
||||
- `audio.DiarizationModel` / `DiarizationProvider` — speaker-labelled
|
||||
segments; per-file `SPEAKER_NN` labels are explicitly NOT stable
|
||||
identities.
|
||||
3. **New `meshgen` leaf package** for image→3D (`Mesh{Data, Format, MIME}`,
|
||||
formats glb/stl/obj). Image-to-3D only: text-to-3D is the caller-owned
|
||||
composition imagegen → meshgen.
|
||||
4. **Binary success bodies are validated before wrapping**: one-image
|
||||
results reject non-image payloads, interpolation reuses the videogen
|
||||
MIME rules and 512MB cap, meshes reject JSON-shaped "success" bodies
|
||||
(a queue-full detail page must never become "the mesh"), and
|
||||
diarization requires `output=json` because the vtt/srt shapes drop
|
||||
speaker labels.
|
||||
|
||||
## Consequences
|
||||
|
||||
- provider/llamaswap gains five surfaces with no new wire machinery beyond
|
||||
`upstreamPath` + a shared one-file-multipart builder.
|
||||
- The passthrough couples majordomo to the fork's `/upstream` route
|
||||
(upstream llama-swap has it too) and to each upstream's native API shape;
|
||||
those shapes are pinned by the netherstorm image builds, not by version
|
||||
negotiation — smoke tests on the host are the drift defence.
|
||||
- `upstream.ignorePaths` (llama-swap config) can 409 asset-looking paths on
|
||||
cold models; none of the paths used here match the default pattern, but
|
||||
new surfaces must check.
|
||||
Reference in New Issue
Block a user