Seed-packet capture: vision model, extraction, catalog match, create (backend) (#81)
Build image / build-and-push (push) Successful in 7s
Gadfly review (reusable) / review (pull_request) Successful in 9m44s
Adversarial Review (Gadfly) / review (pull_request) Successful in 9m44s

Photograph a seed packet → it fills in the plant and the purchase. This is the
backend; the scan UI is a follow-up PR.

Vision model config (mirrors the agent model from #79):
- Migration 0011 adds instance_settings.vision_model; PANSY_VISION_MODEL is the
  env default. Precedence Settings → env → empty; the KEY stays in the env.
- EffectiveVision resolves it; /capabilities advertises "vision" only when a
  model + key are configured, so the UI offers the scan button only when it works.

Extraction is one-shot, NOT an agent loop (internal/vision):
- majordomo.Generate[SeedPacket] derives a JSON schema from the struct tags and
  hands the image to the vision model; it can't call a tool, so it can't touch
  the garden — it only reads a picture and returns data. Numeric fields are
  pointers, so a field the packet doesn't print comes back nil, not a made-up 0.
- Hermetic test: majordomo's fake provider returns canned packet JSON and
  Generate unmarshals it, image + derived schema included. No live model.

The image is normalized to JPEG at the upload boundary (imagenorm from #80),
which is where an iPhone HEIC becomes readable — majordomo's media path can't
decode HEIC. imagenorm now links into the binary (~7 MB, the cost #80 deferred).

The hard part is catalog matching, not OCR (internal/service/seed_packet.go):
- A wrong auto-match splits a variety's seed-lot history across duplicate rows,
  so the service NEVER auto-creates. matchPlants surfaces RANKED candidates
  (exact name → variety-in-name → same species, conservative and name-based),
  the user confirms, and CreateFromPacket makes the plant (new or existing) + the
  lot. Exactly one of plantId/newPlant, refused otherwise.
- Plants/lots aren't in the undo history (catalog/inventory), so no change set.
- The extractor is injectable (service.WithPacketExtractor) so ExtractSeedPacket
  and the /scan endpoint test end to end against a fake, no live model.

Endpoints: POST /seed-lots/scan (multipart image → proposal, reads only; extends
the read deadline for a slow phone upload, caps the body, maps too-large/unreadable
to clear statuses) and POST /seed-lots/from-packet (confirmed proposal → 201).

Docs: README (PANSY_VISION_MODEL), DESIGN (routes + the decision and why the
model can't touch the garden).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01H3zbym8Doka2d7D48maSgZ
This commit is contained in:
2026-07-21 23:50:50 -04:00
co-authored by Claude Opus 4.8
parent e3d8e01e5b
commit 156b3fd14c
16 changed files with 1085 additions and 24 deletions
+12 -1
View File
@@ -185,6 +185,10 @@ func New(cfg *config.Config, svc *service.Service) *gin.Engine {
seedLots.GET("/:id", h.getSeedLot)
seedLots.PATCH("/:id", h.updateSeedLot)
seedLots.DELETE("/:id", h.deleteSeedLot)
// Seed-packet capture (#81): scan a photo into a proposal, then create the
// plant + lot from the confirmed proposal. scan reads only.
seedLots.POST("/scan", h.scanSeedPacket)
seedLots.POST("/from-packet", h.createFromPacket)
// Public, unauthenticated read of a garden by its share token. Deliberately
// NOT behind requireAuth: the token is the capability, so a logged-out visitor
@@ -204,7 +208,14 @@ func New(cfg *config.Config, svc *service.Service) *gin.Engine {
// so offering the tab must track the live Runner. Reading agent.get() (an atomic
// load) means this reflects a settings-driven swap on the very next poll.
func (h *handlers) capabilities(c *gin.Context) {
c.JSON(http.StatusOK, gin.H{"agent": h.agent.get() != nil})
// vision advertises whether seed-packet scanning (#81) can be offered — a
// configured, resolvable vision model + a key. Read per-request so a settings
// change is reflected on the next poll, same as agent.
vision := false
if vis, err := h.svc.EffectiveVision(c.Request.Context()); err == nil {
vision = vis.Ready()
}
c.JSON(http.StatusOK, gin.H{"agent": h.agent.get() != nil, "vision": vision})
}
// healthz is a liveness probe: always returns {"ok": true} when the server is up.