Seed-packet capture: vision model, extraction, catalog match, create (backend) (#81)
Photograph a seed packet → it fills in the plant and the purchase. This is the backend; the scan UI is a follow-up PR. Vision model config (mirrors the agent model from #79): - Migration 0011 adds instance_settings.vision_model; PANSY_VISION_MODEL is the env default. Precedence Settings → env → empty; the KEY stays in the env. - EffectiveVision resolves it; /capabilities advertises "vision" only when a model + key are configured, so the UI offers the scan button only when it works. Extraction is one-shot, NOT an agent loop (internal/vision): - majordomo.Generate[SeedPacket] derives a JSON schema from the struct tags and hands the image to the vision model; it can't call a tool, so it can't touch the garden — it only reads a picture and returns data. Numeric fields are pointers, so a field the packet doesn't print comes back nil, not a made-up 0. - Hermetic test: majordomo's fake provider returns canned packet JSON and Generate unmarshals it, image + derived schema included. No live model. The image is normalized to JPEG at the upload boundary (imagenorm from #80), which is where an iPhone HEIC becomes readable — majordomo's media path can't decode HEIC. imagenorm now links into the binary (~7 MB, the cost #80 deferred). The hard part is catalog matching, not OCR (internal/service/seed_packet.go): - A wrong auto-match splits a variety's seed-lot history across duplicate rows, so the service NEVER auto-creates. matchPlants surfaces RANKED candidates (exact name → variety-in-name → same species, conservative and name-based), the user confirms, and CreateFromPacket makes the plant (new or existing) + the lot. Exactly one of plantId/newPlant, refused otherwise. - Plants/lots aren't in the undo history (catalog/inventory), so no change set. - The extractor is injectable (service.WithPacketExtractor) so ExtractSeedPacket and the /scan endpoint test end to end against a fake, no live model. Endpoints: POST /seed-lots/scan (multipart image → proposal, reads only; extends the read deadline for a slow phone upload, caps the body, maps too-large/unreadable to clear statuses) and POST /seed-lots/from-packet (confirmed proposal → 201). Docs: README (PANSY_VISION_MODEL), DESIGN (routes + the decision and why the model can't touch the garden). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01H3zbym8Doka2d7D48maSgZ
This commit is contained in:
@@ -12,6 +12,7 @@ Work is tracked in Gitea issues; the tracking epic links every piece in dependen
|
||||
- **Users:** multi-user with ownership. Users own gardens; a garden can be shared with other users as viewer (read) or editor (edit content). Owner additionally shares/deletes. The first registered user is `is_admin` (set race-free inside the INSERT); admin gates instance-wide Settings (`requireAdmin`), the only thing that reads that flag.
|
||||
- **Auth:** OIDC-first (Authentik is the primary IdP), local argon2id passwords as an optional fallback.
|
||||
- **Instance settings (#79):** admin-editable, instance-wide config in a single-row `instance_settings` table — pansy's first DB-stored *instance* state (everything else hangs off a garden/object). Today it holds the agent model + on/off; **secrets never move here** — `OLLAMA_CLOUD_API_KEY` stays in the env so it doesn't land in backups or the undo history. Precedence: Settings value → env → default. The live agent Runner sits behind an `atomic.Pointer` in the API layer (`agentHolder`) with its routes always registered, so a settings change swaps it with no restart and no race against in-flight requests; `/capabilities` reads that pointer, so it reports what's live rather than what was configured at boot. The model/registry knowledge lives in one leaf package (`internal/agentmodel`) that both the runner and the settings validator import — agent imports service, so it can live in neither.
|
||||
- **Seed-packet capture (#81):** photograph a packet → a *vision* model (separate `vision_model` setting) reads it into structured fields via one-shot `majordomo.Generate[SeedPacket]` — NOT an agent loop, so the extraction can't touch the garden; it only reads a picture and returns data. The image is normalized to JPEG at the upload boundary (`internal/imagenorm`: decodes HEIC/webp/png/jpeg, since majordomo's stdlib media path can't do HEIC — the iPhone default). The hard part is **catalog matching, not OCR**: a wrong auto-match splits a variety's seed-lot history across duplicate rows, so the service NEVER auto-creates — it surfaces ranked candidates (`matchPlants`) and the user confirms, then `CreateFromPacket` makes the plant (new or existing) + the lot. Plants/lots aren't in the undo history (they're catalog/inventory), so there's no change set to wrap. The extractor is injectable on the service (`WithPacketExtractor`) so the whole path tests hermetically against majordomo's `fake` provider.
|
||||
- **Agentic future:** integration with majordomo/executus via typed Go tools (`llm.DefineTool[Args]`) wrapping the same service layer the REST API uses — not MCP/OpenAPI.
|
||||
|
||||
## Domain model
|
||||
@@ -69,12 +70,14 @@ POST /objects/:id/fill ← hex-pack a region with one plant; region by co
|
||||
POST /objects/:id/clear ← soft-remove every active plop, as ONE change set
|
||||
GET,POST /plants PATCH,DELETE /plants/:id (own plants only)
|
||||
GET,POST /seed-lots GET,PATCH,DELETE /seed-lots/:id (own lots only; private)
|
||||
POST /seed-lots/scan ← multipart image → a seed-packet proposal (reads only, no writes)
|
||||
POST /seed-lots/from-packet ← confirmed proposal → a plant (new or existing) + a lot
|
||||
GET,POST /gardens/:id/journal PATCH,DELETE /journal/:id (editor writes; author edits own)
|
||||
GET /gardens/:id/journal/counts ← entries per object, for the "has notes" indicator
|
||||
POST /agent/chat ← SSE: step events, then the finished turn (editor only)
|
||||
GET,DELETE /gardens/:id/agent/history (the actor's own thread)
|
||||
GET /capabilities ← what this instance can do RIGHT NOW (tracks the live agent, not just config)
|
||||
GET,PATCH /settings ← instance-wide config (admin only): agent model + on/off
|
||||
GET,PATCH /settings ← instance-wide config (admin only): agent model + on/off, vision model
|
||||
GET,POST /gardens/:id/shares PATCH,DELETE /gardens/:id/shares/:userId (invite by email)
|
||||
GET,POST,DELETE /gardens/:id/share-link ← the public read-only token for this garden
|
||||
GET /public/gardens/:token ← UNAUTHENTICATED read-only /full; the token is the capability
|
||||
|
||||
Reference in New Issue
Block a user