Seed-packet capture: vision model, extraction, catalog match, create (backend) (#81)
Build image / build-and-push (push) Successful in 7s
Gadfly review (reusable) / review (pull_request) Successful in 9m44s
Adversarial Review (Gadfly) / review (pull_request) Successful in 9m44s

Photograph a seed packet → it fills in the plant and the purchase. This is the
backend; the scan UI is a follow-up PR.

Vision model config (mirrors the agent model from #79):
- Migration 0011 adds instance_settings.vision_model; PANSY_VISION_MODEL is the
  env default. Precedence Settings → env → empty; the KEY stays in the env.
- EffectiveVision resolves it; /capabilities advertises "vision" only when a
  model + key are configured, so the UI offers the scan button only when it works.

Extraction is one-shot, NOT an agent loop (internal/vision):
- majordomo.Generate[SeedPacket] derives a JSON schema from the struct tags and
  hands the image to the vision model; it can't call a tool, so it can't touch
  the garden — it only reads a picture and returns data. Numeric fields are
  pointers, so a field the packet doesn't print comes back nil, not a made-up 0.
- Hermetic test: majordomo's fake provider returns canned packet JSON and
  Generate unmarshals it, image + derived schema included. No live model.

The image is normalized to JPEG at the upload boundary (imagenorm from #80),
which is where an iPhone HEIC becomes readable — majordomo's media path can't
decode HEIC. imagenorm now links into the binary (~7 MB, the cost #80 deferred).

The hard part is catalog matching, not OCR (internal/service/seed_packet.go):
- A wrong auto-match splits a variety's seed-lot history across duplicate rows,
  so the service NEVER auto-creates. matchPlants surfaces RANKED candidates
  (exact name → variety-in-name → same species, conservative and name-based),
  the user confirms, and CreateFromPacket makes the plant (new or existing) + the
  lot. Exactly one of plantId/newPlant, refused otherwise.
- Plants/lots aren't in the undo history (catalog/inventory), so no change set.
- The extractor is injectable (service.WithPacketExtractor) so ExtractSeedPacket
  and the /scan endpoint test end to end against a fake, no live model.

Endpoints: POST /seed-lots/scan (multipart image → proposal, reads only; extends
the read deadline for a slow phone upload, caps the body, maps too-large/unreadable
to clear statuses) and POST /seed-lots/from-packet (confirmed proposal → 201).

Docs: README (PANSY_VISION_MODEL), DESIGN (routes + the decision and why the
model can't touch the garden).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01H3zbym8Doka2d7D48maSgZ
This commit is contained in:
2026-07-21 23:50:50 -04:00
co-authored by Claude Opus 4.8
parent e3d8e01e5b
commit 156b3fd14c
16 changed files with 1085 additions and 24 deletions
+4 -4
View File
@@ -12,7 +12,7 @@ import (
// The instance_settings row is seeded by migration 0010 and there is exactly one
// (CHECK id = 1), so reads never branch on existence and writes never insert.
const instanceSettingsColumns = `agent_model, agent_enabled, version, updated_at`
const instanceSettingsColumns = `agent_model, agent_enabled, vision_model, version, updated_at`
// scanInstanceSettings reads the single settings row. agent_enabled is a nullable
// INTEGER (NULL = inherit env), so it is scanned through sql.NullInt64.
@@ -21,7 +21,7 @@ func scanInstanceSettings(s scanner) (*domain.InstanceSettings, error) {
out domain.InstanceSettings
enabled sql.NullInt64
)
if err := s.Scan(&out.AgentModel, &enabled, &out.Version, &out.UpdatedAt); err != nil {
if err := s.Scan(&out.AgentModel, &enabled, &out.VisionModel, &out.Version, &out.UpdatedAt); err != nil {
return nil, err
}
if enabled.Valid {
@@ -60,12 +60,12 @@ func (d *DB) UpdateInstanceSettings(ctx context.Context, s *domain.InstanceSetti
}
updated, err := scanInstanceSettings(d.sql.QueryRowContext(ctx,
`UPDATE instance_settings
SET agent_model = ?, agent_enabled = ?,
SET agent_model = ?, agent_enabled = ?, vision_model = ?,
version = version + 1,
updated_at = strftime('%Y-%m-%dT%H:%M:%SZ', 'now')
WHERE id = 1 AND version = ?
RETURNING `+instanceSettingsColumns,
s.AgentModel, enabled, s.Version,
s.AgentModel, enabled, s.VisionModel, s.Version,
))
if errors.Is(err, sql.ErrNoRows) {
current, gerr := d.GetInstanceSettings(ctx)
@@ -0,0 +1,10 @@
-- Vision model setting (#81): the model that reads a photographed seed packet.
--
-- Separate from agent_model because it's a different capability — extracting
-- structured fields from an image needs a vision-capable model, which the chat
-- model may not be. Same "inherit from env unless set" contract as agent_model:
-- '' falls back to PANSY_VISION_MODEL, then the feature is simply not offered.
--
-- Like agent_model, the API KEY is NOT stored here — the vision model runs
-- against the same OLLAMA_CLOUD_API_KEY from the environment.
ALTER TABLE instance_settings ADD COLUMN vision_model TEXT NOT NULL DEFAULT '';