Image normalization on upload: decode HEIC/webp and re-encode to JPEG, without breaking CGO_ENABLED=0 #80

Closed
opened 2026-07-21 22:08:29 +00:00 by steve · 0 comments
Owner

Prerequisite for #TBD (seed-packet capture). Split out because it's a self-contained piece with a real constraint, and because getting it wrong breaks the feature for the exact person who asked for it.

Why this is needed

majordomo's media pipeline (media/media.go) normalizes images for the provider — it sniffs the real format from magic bytes, downscales, and re-encodes down a quality ladder to fit byte budgets. But it is stdlib-based, so it cannot decode HEIC or webp; both return ErrUnsupported.

HEIC is the iPhone camera default. The stated use case is "take a picture of a seed packet". Without this, the first thing that happens on the primary device is a failure.

The constraint

pansy is CGO_ENABLED=0, single static binary — that's why modernc.org/sqlite was chosen over the C one. Most HEIC decoders are libheif bindings and would break that. This is not negotiable for a convenience feature.

Verified solution

github.com/gen2brain/heic v0.7.1 runs libheif as WASM via wazero — pure Go, no cgo. Tested on a real sips-generated HEIC:

BUILD OK (CGO_ENABLED=0)
DecodeConfig format="heic" err=<nil>
decoded heic (400,300) in 4ms -> jpeg 7312 bytes
  • Registers with image.Decode, so normal format sniffing works
  • 4ms for a small image
  • Binary cost: ~5.2 MB (2.85 MB stdlib-only → 8.05 MB with heic+webp). Against pansy's current 32 MB binary that's ~15%. Acceptable; the alternative breaks the static build.

golang.org/x/image/webp covers webp (pure Go). Not verified locally — no webp encoder on the test machine to generate a sample — so verify during implementation rather than assuming.

Gotcha worth not rediscovering

Go's image registry is import-driven. My first build registered only heic and webp, and PNG failed with image: unknown format while HEIC decoded fine. The failure mode is backwards from intuition — the exotic format works and the common one breaks — and it's silent until someone uploads a PNG.

So the blank imports are load-bearing and want a test that exercises all four formats:

import (
    _ "image/jpeg"
    _ "image/png"
    _ "github.com/gen2brain/heic"
    _ "golang.org/x/image/webp"
)

Proposed shape

Normalize at the API boundary on upload, not deep in the agent path:

  1. Read the upload with a hard size cap (see below).
  2. image.Decode — accepts jpeg, png, heic, webp.
  3. Downscale to fit 2048px on the long edge (ollama-cloud's MaxDim; provider/ollama/ollama.go:43-50).
  4. Re-encode as JPEG.
  5. Hand only JPEG bytes to majordomo.

This keeps the vision code path single-format, means majordomo's media.Normalize only ever sees what it can handle, and puts the size limit in one place.

Client-side downscale too. A modern phone photo is 3–5 MB and gets scaled to 2048px anyway. Re-encoding via canvas before upload saves the round trip and most of the bytes. Not a substitute for server-side normalization — never trust the client — but worth doing for the phone case that motivates the feature.

Limits to set deliberately

  • Upload size cap with a clear error, not a truncated read. ollama-cloud's ceiling is 20 MiB, but the cap should be well under that since we re-encode anyway.
  • ReadTimeout: 15s (cmd/pansy/main.go:63) applies to reading the request body. A multi-megabyte photo over a slow phone connection will exceed it. This route needs a longer read deadline via http.NewResponseController(...).SetReadDeadline(...) — the same mechanism #78 needs for writes, and worth doing consistently.
  • Decoding untrusted images is a memory-amplification surface: a small file can decode to a huge bitmap. Check image.DecodeConfig dimensions before image.Decode and refuse anything absurd.

Tests

  • Round-trip each of jpeg/png/heic/webp → JPEG out, asserting dimensions and that the format was actually recognized.
  • A PNG specifically, to catch the missing-blank-import failure above.
  • An oversized-dimensions header that must be refused before full decode.
  • A truncated/corrupt file → clean error, no panic.
Prerequisite for #TBD (seed-packet capture). Split out because it's a self-contained piece with a real constraint, and because getting it wrong breaks the feature for the exact person who asked for it. ## Why this is needed majordomo's media pipeline (`media/media.go`) normalizes images for the provider — it sniffs the real format from magic bytes, downscales, and re-encodes down a quality ladder to fit byte budgets. But it is stdlib-based, so it **cannot decode HEIC or webp**; both return `ErrUnsupported`. **HEIC is the iPhone camera default.** The stated use case is "take a picture of a seed packet". Without this, the first thing that happens on the primary device is a failure. ## The constraint pansy is `CGO_ENABLED=0`, single static binary — that's why `modernc.org/sqlite` was chosen over the C one. Most HEIC decoders are libheif bindings and would break that. This is not negotiable for a convenience feature. ## Verified solution `github.com/gen2brain/heic` v0.7.1 runs libheif as **WASM via wazero** — pure Go, no cgo. Tested on a real `sips`-generated HEIC: ``` BUILD OK (CGO_ENABLED=0) DecodeConfig format="heic" err=<nil> decoded heic (400,300) in 4ms -> jpeg 7312 bytes ``` - Registers with `image.Decode`, so normal format sniffing works - 4ms for a small image - **Binary cost: ~5.2 MB** (2.85 MB stdlib-only → 8.05 MB with heic+webp). Against pansy's current 32 MB binary that's ~15%. Acceptable; the alternative breaks the static build. `golang.org/x/image/webp` covers webp (pure Go). Not verified locally — no webp encoder on the test machine to generate a sample — so **verify during implementation** rather than assuming. ## Gotcha worth not rediscovering Go's image registry is **import-driven**. My first build registered only heic and webp, and PNG failed with `image: unknown format` while HEIC decoded fine. The failure mode is backwards from intuition — the exotic format works and the common one breaks — and it's silent until someone uploads a PNG. So the blank imports are load-bearing and want a test that exercises **all four** formats: ```go import ( _ "image/jpeg" _ "image/png" _ "github.com/gen2brain/heic" _ "golang.org/x/image/webp" ) ``` ## Proposed shape Normalize **at the API boundary on upload**, not deep in the agent path: 1. Read the upload with a hard size cap (see below). 2. `image.Decode` — accepts jpeg, png, heic, webp. 3. Downscale to fit 2048px on the long edge (ollama-cloud's `MaxDim`; `provider/ollama/ollama.go:43-50`). 4. Re-encode as JPEG. 5. Hand only JPEG bytes to majordomo. This keeps the vision code path single-format, means majordomo's `media.Normalize` only ever sees what it can handle, and puts the size limit in one place. **Client-side downscale too.** A modern phone photo is 3–5 MB and gets scaled to 2048px anyway. Re-encoding via `canvas` before upload saves the round trip and most of the bytes. Not a substitute for server-side normalization — never trust the client — but worth doing for the phone case that motivates the feature. ## Limits to set deliberately - **Upload size cap** with a clear error, not a truncated read. ollama-cloud's ceiling is 20 MiB, but the cap should be well under that since we re-encode anyway. - **`ReadTimeout: 15s`** (`cmd/pansy/main.go:63`) applies to reading the request body. A multi-megabyte photo over a slow phone connection will exceed it. This route needs a longer read deadline via `http.NewResponseController(...).SetReadDeadline(...)` — the same mechanism #78 needs for writes, and worth doing consistently. - Decoding untrusted images is a memory-amplification surface: a small file can decode to a huge bitmap. Check `image.DecodeConfig` dimensions **before** `image.Decode` and refuse anything absurd. ## Tests - Round-trip each of jpeg/png/heic/webp → JPEG out, asserting dimensions and that the format was actually recognized. - A PNG specifically, to catch the missing-blank-import failure above. - An oversized-dimensions header that must be refused before full decode. - A truncated/corrupt file → clean error, no panic.
steve closed this issue 2026-07-22 03:27:32 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: steve/pansy#80