Requested: "add an option for a visual model so I could in theory take a picture of a seed packet and it would auto create for me". Scope decision from the same conversation: create both the plant and the seed lot, linked.
Depends on #80 (image normalization) and #79 (Settings, for the model selection). Blocked-ish on neither for a spike, but the model has to be configurable somewhere.
majordomo already does all of this
Verified against the pinned module and the sibling checkout (both at a941f5f). Nothing in majordomo needs changing.
llm.ImagePart{MIME string, Data []byte} and llm.UserParts(parts ...Part) — llm/content.go, llm/message.go
Every provider is vision-capable by default; ollama-cloud allows 8 images, 20 MiB, 2048px, jpeg+png (provider/ollama/ollama.go:43-50), encoded as base64 into the wire images field
llm.Capabilities.SupportsImages() / MIMEAllowed(mime) for checking before sending
This does not need the agent loop.majordomo.Generate[T](ctx, model, req) (generate.go) does one-shot structured extraction: derives a JSON schema from a Go struct, uses the provider's native structured-output mechanism, unmarshals into T. Structured output and images are orthogonal fields on Request and compose, though no example in majordomo does both at once.
typeSeedPacketstruct{Speciesstring`json:"species"`Varietystring`json:"variety"`Vendorstring`json:"vendor"`SKUstring`json:"sku"`LotCodestring`json:"lotCode"`PackedForYear*int`json:"packedForYear"`SeedCount*int`json:"seedCount"`DaysToMaturity*int`json:"daysToMaturity"`SpacingCM*float64`json:"spacingCm"`Confidencemap[string]float64`json:"confidence"`}packet,err:=majordomo.Generate[SeedPacket](ctx,visionModel,majordomo.Request{Messages:[]majordomo.Message{majordomo.UserParts(majordomo.Text("Extract the seed packet details."),majordomo.Image("image/jpeg",jpegBytes),)},})
One round trip, no tool loop, independently testable, and it can't accidentally mutate the garden — a real advantage over routing this through the agent.
Wiring a second model
NewRunner (internal/agent/runtime.go:48-69) builds the registry as a local variable and discards it, keeping only the resolved llm.Model. Parse is a plain method with no memoization, so calling it twice yields two independent models and the token-bridging RegisterProvider (pansy uses OLLAMA_CLOUD_API_KEY, majordomo's ollama preset expects OLLAMA_API_KEY) is already done and shared.
So: retain reg on the Runner, or Parse a second spec and store a second field. Small additive change.
Validation matters here. majordomo passes model ids through verbatim and never checks them against a catalog, so naming a text-only model fails at the API rather than at Parse. llm.WithCapabilities(caps) exists as a ModelOption to override provider defaults, documented for exactly this ("a vision-capable tag on an otherwise text-only provider"). Settings (#79) should offer a test action that sends a tiny image and confirms the model actually accepts it.
The genuinely hard part: catalog matching, not OCR
"Create both, linked" means deciding whether the packet's variety is an existing plant. Getting this wrong silently creates duplicate catalog entries, which then fragment seed-lot history and remaining calculations across two rows that should have been one.
Cases to handle deliberately:
Packet says "Music" hardneck garlic; catalog has built-in "Garlic". Same plant, or a new variety worth its own entry?
Packet says "Cherokee Purple"; catalog has a user-created "Cherokee Purple Tomato". Almost certainly the same.
Two packets of the same variety from different vendors → one plant, two lots.
Proposal: never auto-create silently. Extraction produces a proposal that the user confirms, with the match pre-selected and overridable — "we think this is your existing Garlic; create a new variety instead?". find_plant already does name resolution with multiple candidates (internal/agent/tools.go:53) and its behaviour is the right precedent: return candidates, let the caller choose.
This also handles the confidence problem. A blurry packet, a foreign-language packet, or a photo of something that isn't a seed packet at all should degrade to "here's what I got, fix it" rather than writing nonsense into the catalog.
Other design notes
Both creations in ONE change set, per CLAUDE.md: multi-row operations pass all their changes to a single record call so they undo as one unit. Creating a plant and a lot from one photo is one user action and must undo as one.
Do we store the image? Leaning no for v1 — it's the largest thing pansy would ever put in SQLite, and the extracted data is the value. But a thumbnail on the seed lot would be genuinely nice, and seed_lots has no blob column today. Worth deciding explicitly rather than by omission.
Permissions: seed lots are private to their owner and deliberately never shared with a garden (DESIGN.md). Extraction must run as the actor and create lots owned by them.
Cost: a vision call per photo, on a personal instance. Not a concern at household scale, consistent with the existing "multi-tenant cost control is explicitly not a concern here" note in runtime.go:23-26.
Tests
majordomo ships provider/fake, a scriptable fake provider already used for the agent's loop tests — so extraction is testable hermetically with a canned SeedPacket response. Worth covering: a confident match to an existing plant, a genuine new variety, a low-confidence result that must not auto-create, and a non-packet image.
Per CLAUDE.md, the upload route needs an API-level test through the router — the journal PATCH/DELETE lesson applies to any new route.
Requested: "add an option for a visual model so I could in theory take a picture of a seed packet and it would auto create for me". Scope decision from the same conversation: **create both the plant and the seed lot, linked**.
Depends on #80 (image normalization) and #79 (Settings, for the model selection). Blocked-ish on neither for a spike, but the model has to be configurable somewhere.
## majordomo already does all of this
Verified against the pinned module and the sibling checkout (both at `a941f5f`). **Nothing in majordomo needs changing.**
- `llm.ImagePart{MIME string, Data []byte}` and `llm.UserParts(parts ...Part)` — `llm/content.go`, `llm/message.go`
- Every provider is vision-capable by default; ollama-cloud allows 8 images, 20 MiB, 2048px, jpeg+png (`provider/ollama/ollama.go:43-50`), encoded as base64 into the wire `images` field
- `llm.Capabilities.SupportsImages()` / `MIMEAllowed(mime)` for checking before sending
**This does not need the agent loop.** `majordomo.Generate[T](ctx, model, req)` (`generate.go`) does one-shot structured extraction: derives a JSON schema from a Go struct, uses the provider's native structured-output mechanism, unmarshals into `T`. Structured output and images are orthogonal fields on `Request` and compose, though no example in majordomo does both at once.
```go
type SeedPacket struct {
Species string `json:"species"`
Variety string `json:"variety"`
Vendor string `json:"vendor"`
SKU string `json:"sku"`
LotCode string `json:"lotCode"`
PackedForYear *int `json:"packedForYear"`
SeedCount *int `json:"seedCount"`
DaysToMaturity *int `json:"daysToMaturity"`
SpacingCM *float64 `json:"spacingCm"`
Confidence map[string]float64 `json:"confidence"`
}
packet, err := majordomo.Generate[SeedPacket](ctx, visionModel, majordomo.Request{
Messages: []majordomo.Message{majordomo.UserParts(
majordomo.Text("Extract the seed packet details."),
majordomo.Image("image/jpeg", jpegBytes),
)},
})
```
One round trip, no tool loop, independently testable, and it can't accidentally mutate the garden — a real advantage over routing this through the agent.
## Wiring a second model
`NewRunner` (`internal/agent/runtime.go:48-69`) builds the registry as a **local variable and discards it**, keeping only the resolved `llm.Model`. `Parse` is a plain method with no memoization, so calling it twice yields two independent models and the token-bridging `RegisterProvider` (pansy uses `OLLAMA_CLOUD_API_KEY`, majordomo's ollama preset expects `OLLAMA_API_KEY`) is already done and shared.
So: retain `reg` on the Runner, or `Parse` a second spec and store a second field. Small additive change.
**Validation matters here.** majordomo passes model ids through verbatim and never checks them against a catalog, so naming a text-only model fails at the API rather than at `Parse`. `llm.WithCapabilities(caps)` exists as a `ModelOption` to override provider defaults, documented for exactly this ("a vision-capable tag on an otherwise text-only provider"). Settings (#79) should offer a test action that sends a tiny image and confirms the model actually accepts it.
## The genuinely hard part: catalog matching, not OCR
"Create both, linked" means deciding whether the packet's variety **is** an existing plant. Getting this wrong silently creates duplicate catalog entries, which then fragment seed-lot history and `remaining` calculations across two rows that should have been one.
Cases to handle deliberately:
- Packet says "Music" hardneck garlic; catalog has built-in "Garlic". Same plant, or a new variety worth its own entry?
- Packet says "Cherokee Purple"; catalog has a user-created "Cherokee Purple Tomato". Almost certainly the same.
- Two packets of the same variety from different vendors → one plant, two lots.
**Proposal: never auto-create silently.** Extraction produces a *proposal* that the user confirms, with the match pre-selected and overridable — "we think this is your existing Garlic; create a new variety instead?". `find_plant` already does name resolution with multiple candidates (`internal/agent/tools.go:53`) and its behaviour is the right precedent: return candidates, let the caller choose.
This also handles the confidence problem. A blurry packet, a foreign-language packet, or a photo of something that isn't a seed packet at all should degrade to "here's what I got, fix it" rather than writing nonsense into the catalog.
## Other design notes
- **Both creations in ONE change set**, per CLAUDE.md: multi-row operations pass all their changes to a single `record` call so they undo as one unit. Creating a plant and a lot from one photo is one user action and must undo as one.
- **Do we store the image?** Leaning no for v1 — it's the largest thing pansy would ever put in SQLite, and the extracted data is the value. But a thumbnail on the seed lot would be genuinely nice, and `seed_lots` has no blob column today. Worth deciding explicitly rather than by omission.
- **Permissions:** seed lots are private to their owner and deliberately never shared with a garden (DESIGN.md). Extraction must run as the actor and create lots owned by them.
- **Cost:** a vision call per photo, on a personal instance. Not a concern at household scale, consistent with the existing "multi-tenant cost control is explicitly not a concern here" note in `runtime.go:23-26`.
## Tests
majordomo ships `provider/fake`, a scriptable fake provider already used for the agent's loop tests — so extraction is testable hermetically with a canned `SeedPacket` response. Worth covering: a confident match to an existing plant, a genuine new variety, a low-confidence result that must not auto-create, and a non-packet image.
Per CLAUDE.md, the upload route needs an API-level test through the router — the journal PATCH/DELETE lesson applies to any new route.
Backend (internal/service/seed_packet.go, internal/api/seed_packet.go): POST /seed-lots/scan (multipart photo → a PacketProposal, read-only) and POST /seed-lots/from-packet (confirmed proposal → plant + lot). Extraction is one-shot majordomo.Generate[SeedPacket] — no agent loop — and never auto-creates: it returns a proposal with the catalog match pre-selected and overridable, exactly the find_plant precedent this issue argued for. Plant + lot are created in a single change set so they undo as one unit.
UI (web/src/components/plants/ScanPacketModal.tsx): capture → editable proposal (match candidates as radios, or a brand-new variety) → confirm. Reachable from both the Plants catalog page and the editor's Plants mode, gated on a configured vision model. Shipped in PR #119 + the editor entry point (a72ddef).
Vision-model selection lives in Settings (#79); OLLAMA_CLOUD_API_KEY stays in the environment, never the DB.
Verified the routes + service exist on main. The only thing not exercised locally is the live camera→vision→proposal round-trip, which needs a vision-configured instance — a verification step, not missing work.
Closing — this shipped end to end.
- **Backend** (`internal/service/seed_packet.go`, `internal/api/seed_packet.go`): `POST /seed-lots/scan` (multipart photo → a `PacketProposal`, read-only) and `POST /seed-lots/from-packet` (confirmed proposal → plant + lot). Extraction is one-shot `majordomo.Generate[SeedPacket]` — no agent loop — and **never auto-creates**: it returns a proposal with the catalog match pre-selected and overridable, exactly the `find_plant` precedent this issue argued for. Plant + lot are created in a single change set so they undo as one unit.
- **UI** (`web/src/components/plants/ScanPacketModal.tsx`): capture → editable proposal (match candidates as radios, or a brand-new variety) → confirm. Reachable from **both** the Plants catalog page and the editor's Plants mode, gated on a configured vision model. Shipped in PR #119 + the editor entry point (a72ddef).
- Vision-model selection lives in Settings (#79); `OLLAMA_CLOUD_API_KEY` stays in the environment, never the DB.
Verified the routes + service exist on `main`. The only thing not exercised locally is the live camera→vision→proposal round-trip, which needs a vision-configured instance — a verification step, not missing work.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Requested: "add an option for a visual model so I could in theory take a picture of a seed packet and it would auto create for me". Scope decision from the same conversation: create both the plant and the seed lot, linked.
Depends on #80 (image normalization) and #79 (Settings, for the model selection). Blocked-ish on neither for a spike, but the model has to be configurable somewhere.
majordomo already does all of this
Verified against the pinned module and the sibling checkout (both at
a941f5f). Nothing in majordomo needs changing.llm.ImagePart{MIME string, Data []byte}andllm.UserParts(parts ...Part)—llm/content.go,llm/message.goprovider/ollama/ollama.go:43-50), encoded as base64 into the wireimagesfieldllm.Capabilities.SupportsImages()/MIMEAllowed(mime)for checking before sendingThis does not need the agent loop.
majordomo.Generate[T](ctx, model, req)(generate.go) does one-shot structured extraction: derives a JSON schema from a Go struct, uses the provider's native structured-output mechanism, unmarshals intoT. Structured output and images are orthogonal fields onRequestand compose, though no example in majordomo does both at once.One round trip, no tool loop, independently testable, and it can't accidentally mutate the garden — a real advantage over routing this through the agent.
Wiring a second model
NewRunner(internal/agent/runtime.go:48-69) builds the registry as a local variable and discards it, keeping only the resolvedllm.Model.Parseis a plain method with no memoization, so calling it twice yields two independent models and the token-bridgingRegisterProvider(pansy usesOLLAMA_CLOUD_API_KEY, majordomo's ollama preset expectsOLLAMA_API_KEY) is already done and shared.So: retain
regon the Runner, orParsea second spec and store a second field. Small additive change.Validation matters here. majordomo passes model ids through verbatim and never checks them against a catalog, so naming a text-only model fails at the API rather than at
Parse.llm.WithCapabilities(caps)exists as aModelOptionto override provider defaults, documented for exactly this ("a vision-capable tag on an otherwise text-only provider"). Settings (#79) should offer a test action that sends a tiny image and confirms the model actually accepts it.The genuinely hard part: catalog matching, not OCR
"Create both, linked" means deciding whether the packet's variety is an existing plant. Getting this wrong silently creates duplicate catalog entries, which then fragment seed-lot history and
remainingcalculations across two rows that should have been one.Cases to handle deliberately:
Proposal: never auto-create silently. Extraction produces a proposal that the user confirms, with the match pre-selected and overridable — "we think this is your existing Garlic; create a new variety instead?".
find_plantalready does name resolution with multiple candidates (internal/agent/tools.go:53) and its behaviour is the right precedent: return candidates, let the caller choose.This also handles the confidence problem. A blurry packet, a foreign-language packet, or a photo of something that isn't a seed packet at all should degrade to "here's what I got, fix it" rather than writing nonsense into the catalog.
Other design notes
recordcall so they undo as one unit. Creating a plant and a lot from one photo is one user action and must undo as one.seed_lotshas no blob column today. Worth deciding explicitly rather than by omission.runtime.go:23-26.Tests
majordomo ships
provider/fake, a scriptable fake provider already used for the agent's loop tests — so extraction is testable hermetically with a cannedSeedPacketresponse. Worth covering: a confident match to an existing plant, a genuine new variety, a low-confidence result that must not auto-create, and a non-packet image.Per CLAUDE.md, the upload route needs an API-level test through the router — the journal PATCH/DELETE lesson applies to any new route.
Closing — this shipped end to end.
internal/service/seed_packet.go,internal/api/seed_packet.go):POST /seed-lots/scan(multipart photo → aPacketProposal, read-only) andPOST /seed-lots/from-packet(confirmed proposal → plant + lot). Extraction is one-shotmajordomo.Generate[SeedPacket]— no agent loop — and never auto-creates: it returns a proposal with the catalog match pre-selected and overridable, exactly thefind_plantprecedent this issue argued for. Plant + lot are created in a single change set so they undo as one unit.web/src/components/plants/ScanPacketModal.tsx): capture → editable proposal (match candidates as radios, or a brand-new variety) → confirm. Reachable from both the Plants catalog page and the editor's Plants mode, gated on a configured vision model. Shipped in PR #119 + the editor entry point (a72ddef).OLLAMA_CLOUD_API_KEYstays in the environment, never the DB.Verified the routes + service exist on
main. The only thing not exercised locally is the live camera→vision→proposal round-trip, which needs a vision-configured instance — a verification step, not missing work.