feat: wave-3 image + document surfaces — segmentation, colorize, face restore, ocr (ADR-0023)
- imagegen.Segmenter/SegmentationProvider: prompted mask via POST /upstream/<id>/v1/segment (file, prompt[, threshold], output=mask); white = prompted region, EditRequest.Mask polarity. - imagegen.Colorizer/ColorizeProvider: POST /upstream/<id>/v1/colorize. - imagegen.FaceRestorer/FaceRestoreProvider: POST /upstream/<id>/v1/restore_faces (upscale 1|2). - New ocr leaf package (Request/Page/Result, Recognize) + llamaswap OCRModel: POST /upstream/<id>/v1/ocr (file[, langs, max_pages]), tolerant per-page decode (join lines when page text absent), Raw escape hatch. - httptest contract tests per surface; ADR-0023; ADR index backfilled (0020-0022 rows were missing). Co-Authored-By: Claude Fable 5 <[email protected]>
This commit is contained in:
@@ -0,0 +1,59 @@
|
||||
# ADR-0023: Wave-3 image + document surfaces (segmentation, colorize, face restore, OCR)
|
||||
|
||||
Status: Accepted (2026-07-16)
|
||||
|
||||
## Context
|
||||
|
||||
The llama-swap host is gaining four wave-3 capabilities (spec:
|
||||
mort docs/specs/2026-07-16-llamaswap-wave3.md): promptable segmentation
|
||||
(GroundingDINO + SAM 2.1), photo colorization (DDColor), face restoration
|
||||
(GFPGAN), and document OCR (Surya). All ride the `/upstream/<model>/<path>`
|
||||
passthrough (ADR-0020); none of their native APIs are OpenAI-shaped.
|
||||
|
||||
## Decision
|
||||
|
||||
1. **Grow `imagegen` with three optional interfaces**, ADR-0016→0022
|
||||
conventions (functional options, zero value = backend default, bytes-only
|
||||
I/O, provider mints model):
|
||||
- `imagegen.Segmenter` / `SegmentationProvider` — `SegmentationRequest
|
||||
{Image, Prompt, Threshold}` → one grayscale mask image, WHITE = the
|
||||
prompted region. Same polarity as `EditRequest.Mask` (white = repaint),
|
||||
so the mask feeds inpainting directly; cutouts are derived client-side
|
||||
(mask-as-alpha), one host call serving both. The client always sends
|
||||
`output=mask` — the shim's `cutout`/`boxes` modes are not exposed.
|
||||
- `imagegen.Colorizer` / `ColorizeProvider` — `ColorizeRequest{Image}`,
|
||||
no knobs (the reference backend takes none; options reserved).
|
||||
- `imagegen.FaceRestorer` / `FaceRestoreProvider` —
|
||||
`FaceRestoreRequest{Image, Upscale}` (1 or 2, 0 = backend default).
|
||||
2. **New `ocr` leaf package** rather than a method on an existing surface:
|
||||
OCR is document-shaped (multi-page, PDFs), not image-generation-shaped.
|
||||
`Request{Document, MIME, Filename, Languages, MaxPages}` →
|
||||
`Result{Text, Pages []Page{Number, Text}, Raw}`. `Result.Text` is the
|
||||
pages joined with a blank line; the per-line bbox/confidence/layout
|
||||
detail stays in `Raw` (json.RawMessage) — exact text is the contract,
|
||||
geometry is the escape hatch.
|
||||
3. **provider/llamaswap wire shapes** (pinned by the netherstorm image
|
||||
builds; host smoke tests are the drift defence):
|
||||
- `POST /upstream/<id>/v1/segment` multipart `file`,`prompt`
|
||||
[,`threshold`],`output=mask` → mask PNG.
|
||||
- `POST /upstream/<id>/v1/colorize` multipart `file` → PNG.
|
||||
- `POST /upstream/<id>/v1/restore_faces` multipart `file`[,`upscale`] → PNG.
|
||||
- `POST /upstream/<id>/v1/ocr` multipart `file`[,`langs` (comma-joined),
|
||||
`max_pages`] → JSON `{pages:[{number,text,lines,layout}]}`. The decode
|
||||
is tolerant: a page without aggregate `text` joins its line texts; a
|
||||
missing page `number` defaults to position. Zero pages is an error
|
||||
(a blank page still arrives as a page), mirroring the "no transcript"
|
||||
honesty rule.
|
||||
4. **Binary success bodies are validated before wrapping** (ADR-0020 rule):
|
||||
the three image surfaces reuse `singleImageResult` (positive evidence of
|
||||
image-ness required), OCR requires decodable JSON.
|
||||
|
||||
## Consequences
|
||||
|
||||
- imagegen grows from five to eight optional surfaces; consumers
|
||||
type-assert or use the provider methods directly, as before.
|
||||
- `ocr` is the seventh leaf media package (imagegen, audio, videogen,
|
||||
meshgen, musicgen, embeddings, ocr); the conventions have held across all
|
||||
of them.
|
||||
- PDF handling lives host-side (the shim rasterizes via pypdfium2);
|
||||
majordomo ships bytes and never needs a PDF dependency.
|
||||
@@ -23,3 +23,7 @@ One decision per file, append-only; supersede rather than rewrite.
|
||||
| [0017](0017-audio-interfaces.md) | audio — canonical speech synthesis + transcription interfaces | Accepted |
|
||||
| [0018](0018-imagegen-editor.md) | imagegen.Editor — image-to-image as a separate optional interface | Accepted |
|
||||
| [0019](0019-videogen-interface.md) | videogen — canonical video-generation surface | Accepted |
|
||||
| [0020](0020-upstream-passthrough-media-surfaces.md) | Upstream-passthrough media surfaces (mask, upscale, background removal, interpolation, diarization, meshgen) | Accepted |
|
||||
| [0021](0021-musicgen-interface.md) | musicgen — blocking Generate over an async job queue | Accepted |
|
||||
| [0022](0022-embeddings-rerank-interface.md) | embeddings + rerank interface | Accepted |
|
||||
| [0023](0023-image-doc-surfaces.md) | Wave-3 image + document surfaces (segmentation, colorize, face restore, OCR) | Accepted |
|
||||
|
||||
Reference in New Issue
Block a user