Seed-packet capture: vision model, extraction, catalog match, create (backend) (#81)
Build image / build-and-push (push) Successful in 7s
Gadfly review (reusable) / review (pull_request) Successful in 9m44s
Adversarial Review (Gadfly) / review (pull_request) Successful in 9m44s

Photograph a seed packet → it fills in the plant and the purchase. This is the
backend; the scan UI is a follow-up PR.

Vision model config (mirrors the agent model from #79):
- Migration 0011 adds instance_settings.vision_model; PANSY_VISION_MODEL is the
  env default. Precedence Settings → env → empty; the KEY stays in the env.
- EffectiveVision resolves it; /capabilities advertises "vision" only when a
  model + key are configured, so the UI offers the scan button only when it works.

Extraction is one-shot, NOT an agent loop (internal/vision):
- majordomo.Generate[SeedPacket] derives a JSON schema from the struct tags and
  hands the image to the vision model; it can't call a tool, so it can't touch
  the garden — it only reads a picture and returns data. Numeric fields are
  pointers, so a field the packet doesn't print comes back nil, not a made-up 0.
- Hermetic test: majordomo's fake provider returns canned packet JSON and
  Generate unmarshals it, image + derived schema included. No live model.

The image is normalized to JPEG at the upload boundary (imagenorm from #80),
which is where an iPhone HEIC becomes readable — majordomo's media path can't
decode HEIC. imagenorm now links into the binary (~7 MB, the cost #80 deferred).

The hard part is catalog matching, not OCR (internal/service/seed_packet.go):
- A wrong auto-match splits a variety's seed-lot history across duplicate rows,
  so the service NEVER auto-creates. matchPlants surfaces RANKED candidates
  (exact name → variety-in-name → same species, conservative and name-based),
  the user confirms, and CreateFromPacket makes the plant (new or existing) + the
  lot. Exactly one of plantId/newPlant, refused otherwise.
- Plants/lots aren't in the undo history (catalog/inventory), so no change set.
- The extractor is injectable (service.WithPacketExtractor) so ExtractSeedPacket
  and the /scan endpoint test end to end against a fake, no live model.

Endpoints: POST /seed-lots/scan (multipart image → proposal, reads only; extends
the read deadline for a slow phone upload, caps the body, maps too-large/unreadable
to clear statuses) and POST /seed-lots/from-packet (confirmed proposal → 201).

Docs: README (PANSY_VISION_MODEL), DESIGN (routes + the decision and why the
model can't touch the garden).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01H3zbym8Doka2d7D48maSgZ
This commit is contained in:
2026-07-21 23:50:50 -04:00
co-authored by Claude Opus 4.8
parent e3d8e01e5b
commit 156b3fd14c
16 changed files with 1085 additions and 24 deletions
+38 -4
View File
@@ -43,6 +43,7 @@ func (s *Service) GetInstanceSettings(ctx context.Context, actorID int64) (*doma
type InstanceSettingsPatch struct {
AgentModel string
AgentEnabled *bool
VisionModel string
Version int64
}
@@ -58,16 +59,20 @@ func (s *Service) UpdateInstanceSettings(ctx context.Context, actorID int64, pat
return nil, err
}
model := strings.TrimSpace(patch.AgentModel)
// Validate a non-empty spec up front. An empty one is the "inherit env"
vision := strings.TrimSpace(patch.VisionModel)
// Validate non-empty specs up front. An empty one is the "inherit env"
// sentinel and needs no check — the env value was validated at boot.
if model != "" {
if err := agentmodel.Validate(s.cfg.Agent.OllamaCloudAPIKey, model); err != nil {
return nil, domain.ErrInvalidInput
for _, spec := range []string{model, vision} {
if spec != "" {
if err := agentmodel.Validate(s.cfg.Agent.OllamaCloudAPIKey, spec); err != nil {
return nil, domain.ErrInvalidInput
}
}
}
return s.store.UpdateInstanceSettings(ctx, &domain.InstanceSettings{
AgentModel: model,
AgentEnabled: patch.AgentEnabled,
VisionModel: vision,
Version: patch.Version,
})
}
@@ -108,3 +113,32 @@ func (s *Service) EffectiveAgent(ctx context.Context) (EffectiveAgent, error) {
}
return eff, nil
}
// EffectiveVision resolves the vision configuration in force for seed-packet
// capture (#81): the model from DB-over-env, the key always from env.
type EffectiveVision struct {
Model string
APIKey string
}
// Ready reports whether packet capture can be offered: a key and a vision model.
// There's no separate enabled flag — configuring a vision model IS enabling it.
func (e EffectiveVision) Ready() bool {
return e.APIKey != "" && e.Model != ""
}
// EffectiveVision reads the settings row and layers it over the environment.
func (s *Service) EffectiveVision(ctx context.Context) (EffectiveVision, error) {
st, err := s.store.GetInstanceSettings(ctx)
if err != nil {
return EffectiveVision{}, err
}
eff := EffectiveVision{
Model: s.cfg.Agent.VisionModel,
APIKey: s.cfg.Agent.OllamaCloudAPIKey,
}
if st.VisionModel != "" {
eff.Model = st.VisionModel
}
return eff, nil
}
+218
View File
@@ -0,0 +1,218 @@
package service
import (
"context"
"sort"
"strings"
"gitea.stevedudenhoeffer.com/steve/pansy/internal/domain"
"gitea.stevedudenhoeffer.com/steve/pansy/internal/vision"
)
// Seed-packet capture (#81): read a photographed packet, propose a plant + lot,
// let the user confirm. The two halves are deliberately separate operations:
// extraction only READS (a picture in, a proposal out — it can't touch the
// garden), and creation happens later, from the confirmed proposal, so a
// misread never writes anything on its own.
// PacketPlantMatch is a candidate existing plant the packet might be, with why it
// matched, so the UI can pre-select the likely one and let the user override.
type PacketPlantMatch struct {
Plant domain.Plant `json:"plant"`
// Reason is a short human tag: "exact name", "variety in name", "same species".
Reason string `json:"reason"`
}
// PacketProposal is what a scan returns: the fields read off the packet, plus
// candidate existing plants it might already be. Nothing is created yet.
type PacketProposal struct {
Packet vision.SeedPacket `json:"packet"`
// Candidates are existing plants the packet may match, best first. Empty means
// "probably a new variety" — the UI then offers to create one.
Candidates []PacketPlantMatch `json:"candidates"`
// SuggestedName is the variety (or species) to prefill a new-plant name with.
SuggestedName string `json:"suggestedName"`
// SuggestedCategory is the packet's category if it's a valid one, for prefill.
SuggestedCategory string `json:"suggestedCategory"`
}
// ExtractSeedPacket reads a (JPEG) packet photo and proposes a plant + lot for
// the actor to confirm. It needs a configured vision model; with none it returns
// ErrInvalidInput (the API layer turns "not configured" into a clear message and
// never offers the feature in the first place).
//
// The extraction runs as the actor only in the sense that the catalog match is
// scoped to what they can see; the model call itself has no ACL — it just reads
// a picture the actor uploaded.
func (s *Service) ExtractSeedPacket(ctx context.Context, actorID int64, jpeg []byte) (*PacketProposal, error) {
if len(jpeg) == 0 {
return nil, domain.ErrInvalidInput
}
vis, err := s.EffectiveVision(ctx)
if err != nil {
return nil, err
}
if !vis.Ready() {
// No vision model configured — the feature isn't available.
return nil, domain.ErrInvalidInput
}
packet, err := s.extractPacket(ctx, vis.APIKey, vis.Model, jpeg)
if err != nil {
return nil, err
}
plants, err := s.store.ListPlantsForActor(ctx, actorID)
if err != nil {
return nil, err
}
return &PacketProposal{
Packet: packet,
Candidates: matchPlants(packet, plants),
SuggestedName: suggestedName(packet),
SuggestedCategory: validCategory(packet.Category),
}, nil
}
// suggestedName is the variety if the packet named one, else the species — what
// to prefill a new plant's name with. "Music" beats "garlic" when both are read.
func suggestedName(p vision.SeedPacket) string {
if v := strings.TrimSpace(p.Variety); v != "" {
return v
}
return strings.TrimSpace(p.Species)
}
// validCategory returns the packet's category if it's one pansy knows, else "".
func validCategory(c string) string {
switch c {
case domain.CategoryVegetable, domain.CategoryHerb, domain.CategoryFlower,
domain.CategoryFruit, domain.CategoryTreeShrub, domain.CategoryCover:
return c
default:
return ""
}
}
// matchPlants ranks existing plants the packet might already be, best first.
//
// This is the crux of "create both, linked" (#81): getting it wrong makes a
// duplicate catalog entry that then splits a variety's seed-lot history across
// two rows. So it NEVER decides — it only surfaces candidates for the user to
// confirm. Matching is deliberately conservative and name-based (no fuzzy
// scoring that could confidently mis-rank): exact variety name, variety appearing
// within a plant's name, then same species word. Case-insensitive.
func matchPlants(p vision.SeedPacket, plants []domain.Plant) []PacketPlantMatch {
variety := strings.ToLower(strings.TrimSpace(p.Variety))
species := strings.ToLower(strings.TrimSpace(p.Species))
// rank: lower is better; keep only matched plants.
type scored struct {
match PacketPlantMatch
rank int
}
var out []scored
seen := map[int64]bool{}
add := func(pl domain.Plant, rank int, reason string) {
if seen[pl.ID] {
return
}
seen[pl.ID] = true
out = append(out, scored{PacketPlantMatch{Plant: pl, Reason: reason}, rank})
}
for _, pl := range plants {
name := strings.ToLower(pl.Name)
switch {
case variety != "" && name == variety:
add(pl, 0, "exact name")
case variety != "" && strings.Contains(name, variety):
add(pl, 1, "variety in name")
case species != "" && wordIn(name, species):
add(pl, 2, "same species")
}
}
sort.SliceStable(out, func(i, j int) bool { return out[i].rank < out[j].rank })
matches := make([]PacketPlantMatch, len(out))
for i, s := range out {
matches[i] = s.match
}
return matches
}
// wordIn reports whether word appears as a whole space-delimited token in name,
// so "garlic" matches "German Garlic" but not "garlicky-thing".
func wordIn(name, word string) bool {
for _, tok := range strings.Fields(name) {
if tok == word {
return true
}
}
return false
}
// PacketConfirm is a user-confirmed proposal to turn into rows.
//
// Plant selection is explicit: either PlantID names an existing plant to attach
// the lot to, or NewPlant carries the fields to create one. Exactly one — the
// service refuses both or neither, so an ambiguous confirm can't silently pick.
type PacketConfirm struct {
// PlantID attaches the lot to an existing plant. Set this XOR NewPlant.
PlantID *int64
// NewPlant creates a variety. Set this XOR PlantID.
NewPlant *PlantInput
// Lot is the purchase to record against whichever plant results.
Lot SeedLotInput
}
// PacketResult is what a confirm produced.
type PacketResult struct {
Plant *domain.Plant `json:"plant"`
Lot *domain.SeedLot `json:"lot"`
PlantIsNew bool `json:"plantIsNew"`
}
// CreateFromPacket turns a confirmed proposal into a plant (new or existing) plus
// a seed lot attributed to it. Unlike garden edits these rows aren't in the undo
// history — plants and lots are catalog/inventory, created directly — so there is
// no change set to wrap; the two creations just happen in sequence.
//
// If a new plant is created but the lot then fails, the plant is left behind
// rather than rolled back: a stray catalog entry is harmless and editable,
// whereas silently discarding a variety the user just confirmed is worse. The
// caller sees the error and can retry the lot against the now-existing plant.
func (s *Service) CreateFromPacket(ctx context.Context, actorID int64, in PacketConfirm) (*PacketResult, error) {
hasID, hasNew := in.PlantID != nil, in.NewPlant != nil
if hasID == hasNew {
return nil, domain.ErrInvalidInput // exactly one of existing / new
}
res := &PacketResult{}
if hasNew {
plant, err := s.CreatePlant(ctx, actorID, *in.NewPlant)
if err != nil {
return nil, err
}
res.Plant = plant
res.PlantIsNew = true
} else {
// Attach to an existing plant the actor can see. visiblePlant enforces the
// ACL (built-ins + their own); a plant they can't see is ErrNotFound.
plant, err := s.visiblePlant(ctx, actorID, *in.PlantID)
if err != nil {
return nil, err
}
res.Plant = plant
}
lotIn := in.Lot
lotIn.PlantID = res.Plant.ID
lot, err := s.CreateSeedLot(ctx, actorID, lotIn)
if err != nil {
return nil, err
}
res.Lot = lot
return res, nil
}
+197
View File
@@ -0,0 +1,197 @@
package service
import (
"context"
"errors"
"testing"
"gitea.stevedudenhoeffer.com/steve/pansy/internal/domain"
"gitea.stevedudenhoeffer.com/steve/pansy/internal/vision"
)
func packet(species, variety, category string) vision.SeedPacket {
return vision.SeedPacket{Species: species, Variety: variety, Category: category}
}
func plant(id int64, name string) domain.Plant {
return domain.Plant{ID: id, Name: name, Category: domain.CategoryVegetable}
}
// TestMatchPlants pins the catalog-matching heuristic: it surfaces candidates,
// best first, and never invents a match — the whole point, since a wrong auto-
// match would fragment a variety's seed-lot history across duplicate rows.
func TestMatchPlants(t *testing.T) {
catalog := []domain.Plant{
plant(1, "Garlic"),
plant(2, "Music Garlic"),
plant(3, "Cherokee Purple"),
plant(4, "Basil"),
}
t.Run("exact variety wins, ranked above looser matches", func(t *testing.T) {
got := matchPlants(packet("tomato", "Cherokee Purple", "vegetable"), catalog)
if len(got) == 0 || got[0].Plant.ID != 3 || got[0].Reason != "exact name" {
t.Fatalf("want Cherokee Purple exact first, got %+v", got)
}
})
t.Run("variety within a name, plus same-species, ordered", func(t *testing.T) {
// "Music" garlic: "Music Garlic" contains the variety (rank 1); "Garlic"
// shares the species word (rank 2).
got := matchPlants(packet("garlic", "Music", "vegetable"), catalog)
if len(got) != 2 {
t.Fatalf("got %d candidates, want 2: %+v", len(got), got)
}
if got[0].Plant.ID != 2 || got[0].Reason != "variety in name" {
t.Errorf("first = %+v, want Music Garlic / variety in name", got[0])
}
if got[1].Plant.ID != 1 || got[1].Reason != "same species" {
t.Errorf("second = %+v, want Garlic / same species", got[1])
}
})
t.Run("case-insensitive", func(t *testing.T) {
got := matchPlants(packet("", "cherokee purple", ""), catalog)
if len(got) == 0 || got[0].Plant.ID != 3 {
t.Errorf("case-insensitive exact match failed: %+v", got)
}
})
t.Run("no match → empty (a new variety)", func(t *testing.T) {
if got := matchPlants(packet("okra", "Clemson Spineless", "vegetable"), catalog); len(got) != 0 {
t.Errorf("want no candidates, got %+v", got)
}
})
t.Run("species word boundary, not substring", func(t *testing.T) {
// "garlic" should not match a hypothetical "garlicky" — wordIn is token-based.
got := matchPlants(packet("garlic", "", ""), []domain.Plant{plant(9, "Garlicky Mustard")})
if len(got) != 0 {
t.Errorf("substring shouldn't match on species: %+v", got)
}
})
}
// visionTestService builds a service with a configured vision model and a canned
// extractor, so ExtractSeedPacket can run with no live model.
func visionTestService(t *testing.T, out vision.SeedPacket, extractErr error) (*Service, int64) {
t.Helper()
cfg := openConfig()
cfg.Agent.OllamaCloudAPIKey = "k"
cfg.Agent.VisionModel = "ollama-cloud/vision:cloud"
s := newTestService(t, cfg)
s.extractPacket = func(ctx context.Context, apiKey, model string, jpeg []byte) (vision.SeedPacket, error) {
return out, extractErr
}
owner := seedUser(t, s, "[email protected]")
return s, owner
}
// TestExtractSeedPacket exercises the orchestration: canned packet → proposal
// with catalog candidates + prefill suggestions.
func TestExtractSeedPacket(t *testing.T) {
ctx := context.Background()
s, owner := visionTestService(t, packet("garlic", "Music", "vegetable"), nil)
// Seed a matching plant.
if _, err := s.CreatePlant(ctx, owner, PlantInput{
Name: "Music Garlic", Category: domain.CategoryVegetable, SpacingCM: 15, Color: "#4a7c3f", Icon: "🧄",
}); err != nil {
t.Fatalf("seed plant: %v", err)
}
prop, err := s.ExtractSeedPacket(ctx, owner, []byte("jpeg-bytes"))
if err != nil {
t.Fatalf("extract: %v", err)
}
if prop.Packet.Variety != "Music" {
t.Errorf("packet variety = %q", prop.Packet.Variety)
}
if len(prop.Candidates) == 0 || prop.Candidates[0].Plant.Name != "Music Garlic" {
t.Errorf("expected Music Garlic candidate, got %+v", prop.Candidates)
}
if prop.SuggestedName != "Music" || prop.SuggestedCategory != domain.CategoryVegetable {
t.Errorf("suggestions = %q/%q", prop.SuggestedName, prop.SuggestedCategory)
}
}
// TestExtractSeedPacketNeedsVisionModel: with no vision model configured, the
// feature is unavailable (ErrInvalidInput), and the extractor is never called.
func TestExtractSeedPacketNeedsVisionModel(t *testing.T) {
s := newTestService(t, openConfig()) // no vision model, no key
called := false
s.extractPacket = func(ctx context.Context, _, _ string, _ []byte) (vision.SeedPacket, error) {
called = true
return vision.SeedPacket{}, nil
}
owner := seedUser(t, s, "[email protected]")
if _, err := s.ExtractSeedPacket(context.Background(), owner, []byte("x")); !errors.Is(err, domain.ErrInvalidInput) {
t.Errorf("err = %v, want ErrInvalidInput", err)
}
if called {
t.Error("extractor was called despite no configured vision model")
}
}
// TestCreateFromPacketNewPlant: a confirm with NewPlant creates the plant and a
// lot attributed to it, in one call.
func TestCreateFromPacketNewPlant(t *testing.T) {
ctx := context.Background()
s, owner := visionTestService(t, vision.SeedPacket{}, nil)
res, err := s.CreateFromPacket(ctx, owner, PacketConfirm{
NewPlant: &PlantInput{Name: "Music Garlic", Category: domain.CategoryVegetable, SpacingCM: 15, Color: "#4a7c3f", Icon: "🧄"},
Lot: SeedLotInput{Vendor: "Johnny's", Quantity: 8, Unit: domain.UnitBulbs},
})
if err != nil {
t.Fatalf("create: %v", err)
}
if !res.PlantIsNew || res.Plant.Name != "Music Garlic" {
t.Errorf("plant = %+v, isNew=%v", res.Plant, res.PlantIsNew)
}
if res.Lot == nil || res.Lot.PlantID != res.Plant.ID {
t.Errorf("lot not attributed to the new plant: %+v", res.Lot)
}
}
// TestCreateFromPacketExistingPlant: a confirm with PlantID attaches the lot to
// the existing plant and creates nothing new.
func TestCreateFromPacketExistingPlant(t *testing.T) {
ctx := context.Background()
s, owner := visionTestService(t, vision.SeedPacket{}, nil)
existing, err := s.CreatePlant(ctx, owner, PlantInput{
Name: "Garlic", Category: domain.CategoryVegetable, SpacingCM: 15, Color: "#4a7c3f", Icon: "🧄",
})
if err != nil {
t.Fatalf("seed plant: %v", err)
}
res, err := s.CreateFromPacket(ctx, owner, PacketConfirm{
PlantID: &existing.ID,
Lot: SeedLotInput{Vendor: "Fedco", Quantity: 10, Unit: domain.UnitBulbs},
})
if err != nil {
t.Fatalf("create: %v", err)
}
if res.PlantIsNew || res.Plant.ID != existing.ID {
t.Errorf("should attach to existing plant, got %+v isNew=%v", res.Plant, res.PlantIsNew)
}
if res.Lot.PlantID != existing.ID {
t.Errorf("lot plantId = %d, want %d", res.Lot.PlantID, existing.ID)
}
}
// TestCreateFromPacketExactlyOne: both or neither of PlantID/NewPlant is refused,
// so an ambiguous confirm can't silently pick.
func TestCreateFromPacketExactlyOne(t *testing.T) {
ctx := context.Background()
s, owner := visionTestService(t, vision.SeedPacket{}, nil)
id := int64(1)
for _, in := range []PacketConfirm{
{}, // neither
{PlantID: &id, NewPlant: &PlantInput{Name: "X"}}, // both
} {
if _, err := s.CreateFromPacket(ctx, owner, in); !errors.Is(err, domain.ErrInvalidInput) {
t.Errorf("CreateFromPacket(%+v) err = %v, want ErrInvalidInput", in, err)
}
}
}
+27 -6
View File
@@ -9,6 +9,7 @@
package service
import (
"context"
"crypto/rand"
"crypto/sha256"
"encoding/base64"
@@ -18,6 +19,7 @@ import (
"gitea.stevedudenhoeffer.com/steve/pansy/internal/config"
"gitea.stevedudenhoeffer.com/steve/pansy/internal/store"
"gitea.stevedudenhoeffer.com/steve/pansy/internal/vision"
)
// timeLayout is the ISO-8601 UTC format used for every full timestamp pansy
@@ -41,16 +43,35 @@ type Service struct {
// produced by timingHash (fixed salt, no RNG) so it is always present — an
// empty one would silently re-open account enumeration.
dummyHash string
// extractPacket reads a photographed seed packet (#81). Injectable so tests
// can supply a canned packet instead of calling a live vision model — the
// same reason `now` is injectable. Defaults to vision.Extract.
extractPacket func(ctx context.Context, apiKey, modelSpec string, jpeg []byte) (vision.SeedPacket, error)
}
// Option customizes a Service at construction. The only current use is injecting
// a seed-packet extractor in tests so they don't call a live vision model.
type Option func(*Service)
// WithPacketExtractor overrides how a photographed seed packet is read (#81).
// Production uses vision.Extract; a test supplies a canned reader.
func WithPacketExtractor(fn func(ctx context.Context, apiKey, modelSpec string, jpeg []byte) (vision.SeedPacket, error)) Option {
return func(s *Service) { s.extractPacket = fn }
}
// New constructs a Service.
func New(st *store.DB, cfg *config.Config) *Service {
return &Service{
store: st,
cfg: cfg,
now: time.Now,
dummyHash: timingHash(),
func New(st *store.DB, cfg *config.Config, opts ...Option) *Service {
s := &Service{
store: st,
cfg: cfg,
now: time.Now,
dummyHash: timingHash(),
extractPacket: vision.Extract,
}
for _, o := range opts {
o(s)
}
return s
}
// formatTime renders a time as pansy's canonical UTC string.