steveandClaude Fable 5
e6987f54b2
- imagegen.Segmenter/SegmentationProvider: prompted mask via POST /upstream/<id>/v1/segment (file, prompt[, threshold], output=mask); white = prompted region, EditRequest.Mask polarity. - imagegen.Colorizer/ColorizeProvider: POST /upstream/<id>/v1/colorize. - imagegen.FaceRestorer/FaceRestoreProvider: POST /upstream/<id>/v1/restore_faces (upscale 1|2). - New ocr leaf package (Request/Page/Result, Recognize) + llamaswap OCRModel: POST /upstream/<id>/v1/ocr (file[, langs, max_pages]), tolerant per-page decode (join lines when page text absent), Raw escape hatch. - httptest contract tests per surface; ADR-0023; ADR index backfilled (0020-0022 rows were missing). Co-Authored-By: Claude Fable 5 <[email protected]>
60 lines
3.2 KiB
Markdown
60 lines
3.2 KiB
Markdown
# ADR-0023: Wave-3 image + document surfaces (segmentation, colorize, face restore, OCR)
|
|
|
|
Status: Accepted (2026-07-16)
|
|
|
|
## Context
|
|
|
|
The llama-swap host is gaining four wave-3 capabilities (spec:
|
|
mort docs/specs/2026-07-16-llamaswap-wave3.md): promptable segmentation
|
|
(GroundingDINO + SAM 2.1), photo colorization (DDColor), face restoration
|
|
(GFPGAN), and document OCR (Surya). All ride the `/upstream/<model>/<path>`
|
|
passthrough (ADR-0020); none of their native APIs are OpenAI-shaped.
|
|
|
|
## Decision
|
|
|
|
1. **Grow `imagegen` with three optional interfaces**, ADR-0016→0022
|
|
conventions (functional options, zero value = backend default, bytes-only
|
|
I/O, provider mints model):
|
|
- `imagegen.Segmenter` / `SegmentationProvider` — `SegmentationRequest
|
|
{Image, Prompt, Threshold}` → one grayscale mask image, WHITE = the
|
|
prompted region. Same polarity as `EditRequest.Mask` (white = repaint),
|
|
so the mask feeds inpainting directly; cutouts are derived client-side
|
|
(mask-as-alpha), one host call serving both. The client always sends
|
|
`output=mask` — the shim's `cutout`/`boxes` modes are not exposed.
|
|
- `imagegen.Colorizer` / `ColorizeProvider` — `ColorizeRequest{Image}`,
|
|
no knobs (the reference backend takes none; options reserved).
|
|
- `imagegen.FaceRestorer` / `FaceRestoreProvider` —
|
|
`FaceRestoreRequest{Image, Upscale}` (1 or 2, 0 = backend default).
|
|
2. **New `ocr` leaf package** rather than a method on an existing surface:
|
|
OCR is document-shaped (multi-page, PDFs), not image-generation-shaped.
|
|
`Request{Document, MIME, Filename, Languages, MaxPages}` →
|
|
`Result{Text, Pages []Page{Number, Text}, Raw}`. `Result.Text` is the
|
|
pages joined with a blank line; the per-line bbox/confidence/layout
|
|
detail stays in `Raw` (json.RawMessage) — exact text is the contract,
|
|
geometry is the escape hatch.
|
|
3. **provider/llamaswap wire shapes** (pinned by the netherstorm image
|
|
builds; host smoke tests are the drift defence):
|
|
- `POST /upstream/<id>/v1/segment` multipart `file`,`prompt`
|
|
[,`threshold`],`output=mask` → mask PNG.
|
|
- `POST /upstream/<id>/v1/colorize` multipart `file` → PNG.
|
|
- `POST /upstream/<id>/v1/restore_faces` multipart `file`[,`upscale`] → PNG.
|
|
- `POST /upstream/<id>/v1/ocr` multipart `file`[,`langs` (comma-joined),
|
|
`max_pages`] → JSON `{pages:[{number,text,lines,layout}]}`. The decode
|
|
is tolerant: a page without aggregate `text` joins its line texts; a
|
|
missing page `number` defaults to position. Zero pages is an error
|
|
(a blank page still arrives as a page), mirroring the "no transcript"
|
|
honesty rule.
|
|
4. **Binary success bodies are validated before wrapping** (ADR-0020 rule):
|
|
the three image surfaces reuse `singleImageResult` (positive evidence of
|
|
image-ness required), OCR requires decodable JSON.
|
|
|
|
## Consequences
|
|
|
|
- imagegen grows from five to eight optional surfaces; consumers
|
|
type-assert or use the provider methods directly, as before.
|
|
- `ocr` is the seventh leaf media package (imagegen, audio, videogen,
|
|
meshgen, musicgen, embeddings, ocr); the conventions have held across all
|
|
of them.
|
|
- PDF handling lives host-side (the shim rasterizes via pypdfium2);
|
|
majordomo ships bytes and never needs a PDF dependency.
|