Files
pansy/internal/agent/tools_test.go
T
steveandClaude Opus 4.8 3f3a5b057c
Build image / build-and-push (push) Successful in 25s
Gadfly review (reusable) / review (pull_request) Canceled after 5m56s
Adversarial Review (Gadfly) / review (pull_request) Canceled after 5m56s
Agent runtime: majordomo in-process, Ollama Cloud config, chat endpoint (#56)
Everything below the run loop already existed. This is the thing that runs a
model.

The build tag is gone, deliberately. internal/agent's doc comment promised two
separations — cmd/pansy not importing the package, and the tool wiring behind
//go:build majordomo — and both have been rewritten rather than left as a stale
aspiration. A tag that keeps the agent out of the binary only earns its keep if
you would ever ship a build without the agent, and the agent is the point;
keeping it meant an untagged CI that never compiled the code that matters.
majordomo is a real dependency now, resolved from the Gitea instance as a
pseudo-version with no replace directive, so the Docker build (which has no
sibling checkout) resolves it the same way this machine does. It is stdlib-first
and pure Go, so CGO_ENABLED=0 and the single static binary survive.

A TURN IS ONE CHANGE SET. That is the whole reason acting without a confirmation
prompt is defensible: "empty the garlic bed and plant cucumbers" is one object
edit and a dozen planting inserts, and it has to undo as one action rather than
thirteen. The scope is opened even for a turn that turns out to be a question,
because a change set with no revisions is never written — so asking costs
nothing and history isn't littered with empty entries.

The model spec goes to majordomo.Parse verbatim. That grammar, including
comma-separated failover chains, is majordomo's; re-implementing any of it here
would only mean two places to update when it grows. The key needs a bridge
though: majordomo's ollama-cloud preset reads OLLAMA_API_KEY while pansy (like
gadfly) is configured with OLLAMA_CLOUD_API_KEY, so the provider is registered
explicitly on a private registry rather than depending on ambient environment.

Runs are bounded by a step cap, a timeout and majordomo's loop guards. This is
loop safety, not cost control — pansy is a personal tool and spend caps are
explicitly not a v2 concern. A capped run does NOT fail: it kept whatever it
managed to do, that work is recorded and undoable, and the reply says it stopped
early rather than going silent.

The chat endpoint streams. A turn that clears a bed and replants it makes a
dozen tool calls over tens of seconds, and without streaming that is a long
silence followed by everything at once — which reads as a hang, and defeats a
design that rests on watching the canvas change as it happens.

Conversations persist per (user, garden). Client-held history would be lost on a
refresh, which is exactly when someone reloads to check whether the agent's
change landed. Only the user/assistant TEXT is stored, not the model's full
transcript: continuity needs what was said and what came back, and replaying a
stored tool call would replay a decision made against a garden that has since
moved on. It also keeps majordomo's message shape out of the schema.

An instance with no key starts, serves the app, and doesn't advertise the agent
— the routes aren't registered at all, the same shape as OIDC 404ing when
unconfigured. A configured-but-unresolvable model logs and disables the
assistant rather than refusing to boot: a garden planner that won't start
because of a chat feature is worse than one without chat.

Tool refusals reach the model as tool results it can explain, not 500s. The ACL
story only works if it can narrate the refusal.

Closes #56

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01H3zbym8Doka2d7D48maSgZ
2026-07-21 02:15:22 -04:00

369 lines
13 KiB
Go
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
package agent
import (
"context"
"encoding/json"
"strings"
"testing"
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
"gitea.stevedudenhoeffer.com/steve/pansy/internal/config"
"gitea.stevedudenhoeffer.com/steve/pansy/internal/domain"
"gitea.stevedudenhoeffer.com/steve/pansy/internal/service"
"gitea.stevedudenhoeffer.com/steve/pansy/internal/store"
)
// TestToolboxScenario is the #19 demo harness: it drives the DESIGN scenario
// through the tool layer (JSON args → DefineTool → service) rather than a live
// model — fill the NE corner with garlic, the NW with basil, the south with
// beans — and checks describe_garden reports three groups in the right places.
// It also confirms the toolbox inherits pansy's ACL (a viewer is refused).
//
// Build/run with: go test -tags majordomo ./internal/agent/
func TestToolboxScenario(t *testing.T) {
ctx := context.Background()
svc, ownerID := newAgentTestService(t)
box := NewToolbox(svc, ownerID)
call := func(name string, args any) llm.ToolResult {
t.Helper()
raw, _ := json.Marshal(args)
return box.Execute(ctx, llm.ToolCall{ID: "1", Name: name, Arguments: raw})
}
// Garden + plants are set up directly (there are no create_garden/plant tools);
// the agent-facing bits — object + fills + describe — go through the toolbox.
g, err := svc.CreateGarden(ctx, ownerID, service.GardenInput{Name: "Plot", WidthCM: 2000, HeightCM: 2000})
if err != nil {
t.Fatalf("garden: %v", err)
}
garlic := mustPlant(t, svc, ownerID, "Garlic", 15, "🧄")
basil := mustPlant(t, svc, ownerID, "Basil", 25, "🌿")
beans := mustPlant(t, svc, ownerID, "Beans", 10, "🫘")
// create_object → a 400×400 bed.
res := call("create_object", map[string]any{
"gardenId": g.ID, "kind": "bed", "name": "Bed 1", "xCm": 1000, "yCm": 1000, "widthCm": 400, "heightCm": 400,
})
if res.IsError {
t.Fatalf("create_object: %s", res.Content)
}
var bed struct {
ID int64 `json:"id"`
}
if err := json.Unmarshal([]byte(res.Content), &bed); err != nil {
t.Fatalf("decode created object: %v", err)
}
// fill_region for each corner/half.
for _, f := range []struct {
region string
plantID int64
}{{"ne", garlic.ID}, {"nw", basil.ID}, {"south", beans.ID}} {
if r := call("fill_region", map[string]any{"objectId": bed.ID, "region": f.region, "plantId": f.plantID}); r.IsError {
t.Fatalf("fill_region %s: %s", f.region, r.Content)
}
}
// describe_garden → parse and check locations.
res = call("describe_garden", map[string]any{"gardenId": g.ID})
if res.IsError {
t.Fatalf("describe_garden: %s", res.Content)
}
var desc service.DescribeResult
if err := json.Unmarshal([]byte(res.Content), &desc); err != nil {
t.Fatalf("decode describe: %v", err)
}
if len(desc.Objects) != 1 {
t.Fatalf("objects = %d, want 1", len(desc.Objects))
}
seen := map[string]map[string]bool{}
for _, p := range desc.Objects[0].Plantings {
if seen[p.Plant] == nil {
seen[p.Plant] = map[string]bool{}
}
seen[p.Plant][p.Location] = true
}
if !seen["Garlic"]["NE corner"] {
t.Errorf("garlic at %v, want NE corner", seen["Garlic"])
}
if !seen["Basil"]["NW corner"] {
t.Errorf("basil at %v, want NW corner", seen["Basil"])
}
if len(seen["Beans"]) == 0 {
t.Error("beans produced no plops")
}
for loc := range seen["Beans"] {
if loc == "north" || loc == "NE corner" || loc == "NW corner" || loc == "center" {
t.Errorf("beans at %q, want only southern locations", loc)
}
}
// ACL: a viewer's fill_region is refused (the toolbox runs as that actor).
viewerUser, err := svc.Register(ctx, service.RegisterInput{Email: "[email protected]", DisplayName: "V", Password: "password123"})
if err != nil {
t.Fatalf("register viewer: %v", err)
}
if _, err := svc.AddShare(ctx, ownerID, g.ID, "[email protected]", domain.RoleViewer); err != nil {
t.Fatalf("share: %v", err)
}
viewerBox := NewToolbox(svc, viewerUser.ID)
vr := viewerBox.Execute(ctx, llm.ToolCall{ID: "2", Name: "fill_region", Arguments: mustJSON(t, map[string]any{
"objectId": bed.ID, "region": "all", "plantId": garlic.ID,
})})
if !vr.IsError {
t.Errorf("viewer fill_region succeeded, want a permission error")
}
}
func mustJSON(t *testing.T, v any) json.RawMessage {
t.Helper()
b, err := json.Marshal(v)
if err != nil {
t.Fatalf("marshal: %v", err)
}
return b
}
func mustPlant(t *testing.T, svc *service.Service, owner int64, name string, spacing float64, icon string) *domain.Plant {
t.Helper()
p, err := svc.CreatePlant(context.Background(), owner, service.PlantInput{
Name: name, Category: domain.CategoryVegetable, SpacingCM: spacing, Color: "#4a7c3f", Icon: icon,
})
if err != nil {
t.Fatalf("create plant %s: %v", name, err)
}
return p
}
// TestGarlicBedToCucumbers is the flagship interaction from #58, driven end to
// end through the tool layer: "change the garlic garden bed to instead be
// cucumbers this year".
//
// The sequence is the one a model would actually run — describe to find the bed,
// find_plant to turn the word "cucumber" into an id, clear, refill — and until
// find_plant existed step three had no way to get its plantId, which blocked the
// whole thing on one missing tool.
func TestGarlicBedToCucumbers(t *testing.T) {
ctx := context.Background()
svc, owner := newAgentTestService(t)
box := NewToolbox(svc, owner)
call := func(name string, args any) llm.ToolResult {
t.Helper()
raw, _ := json.Marshal(args)
return box.Execute(ctx, llm.ToolCall{ID: "1", Name: name, Arguments: raw})
}
g, err := svc.CreateGarden(ctx, owner, service.GardenInput{Name: "Plot", WidthCM: 2000, HeightCM: 2000})
if err != nil {
t.Fatalf("garden: %v", err)
}
garlic := mustPlant(t, svc, owner, "Garlic", 15, "🧄")
mustPlant(t, svc, owner, "Cucumber", 45, "🥒")
bed, err := svc.CreateObject(ctx, owner, g.ID, service.ObjectInput{
Kind: domain.KindBed, Name: "Garlic bed", XCM: 1000, YCM: 1000, WidthCM: 400, HeightCM: 400,
})
if err != nil {
t.Fatalf("bed: %v", err)
}
if _, err := svc.FillNamedRegion(ctx, owner, bed.ID, "all", garlic.ID, nil); err != nil {
t.Fatalf("seed the garlic: %v", err)
}
// 1. describe_garden — "the garlic bed" resolves to an object id, because the
// description carries the NAMES of what's planted in each object.
res := call("describe_garden", map[string]any{"gardenId": g.ID})
if res.IsError {
t.Fatalf("describe_garden: %s", res.Content)
}
if !strings.Contains(res.Content, "Garlic") {
t.Fatalf("describe_garden didn't name what's planted: %s", res.Content)
}
// 2. find_plant — the step that used to be impossible.
res = call("find_plant", map[string]any{"query": "cucumber"})
if res.IsError {
t.Fatalf("find_plant: %s", res.Content)
}
var matches []struct {
ID int64 `json:"id"`
Name string `json:"name"`
}
if err := json.Unmarshal([]byte(res.Content), &matches); err != nil {
t.Fatalf("decode find_plant: %v (%s)", err, res.Content)
}
if len(matches) == 0 || matches[0].Name != "Cucumber" {
t.Fatalf("find_plant(cucumber) = %+v", matches)
}
// 3. clear_object, 4. fill_region.
if r := call("clear_object", map[string]any{"objectId": bed.ID}); r.IsError {
t.Fatalf("clear_object: %s", r.Content)
}
if r := call("fill_region", map[string]any{
"objectId": bed.ID, "region": "all", "plantId": matches[0].ID,
}); r.IsError {
t.Fatalf("fill_region: %s", r.Content)
}
// The bed is cucumbers, and no garlic is still growing in it.
full, err := svc.GardenFull(ctx, owner, g.ID, nil)
if err != nil {
t.Fatalf("GardenFull: %v", err)
}
if len(full.Plantings) == 0 {
t.Fatal("the bed ended up empty")
}
for _, p := range full.Plantings {
if p.PlantID == garlic.ID {
t.Errorf("garlic is still active in the bed")
}
if p.PlantID != matches[0].ID {
t.Errorf("unexpected plant %d in the bed", p.PlantID)
}
}
}
// TestFindPlantReturnsCandidatesNotAGuess — "garlic" against a catalog holding
// both "Garlic" and "German Red Garlic" is genuinely ambiguous, and silently
// picking one is how the agent plants the wrong thing.
func TestFindPlantReturnsCandidatesNotAGuess(t *testing.T) {
ctx := context.Background()
svc, owner := newAgentTestService(t)
box := NewToolbox(svc, owner)
mustPlant(t, svc, owner, "German Red Garlic", 15, "🧄")
raw, _ := json.Marshal(map[string]any{"query": "garlic"})
res := box.Execute(ctx, llm.ToolCall{ID: "1", Name: "find_plant", Arguments: raw})
if res.IsError {
t.Fatalf("find_plant: %s", res.Content)
}
var matches []struct {
Name string `json:"name"`
}
if err := json.Unmarshal([]byte(res.Content), &matches); err != nil {
t.Fatalf("decode: %v", err)
}
names := map[string]bool{}
for _, m := range matches {
names[m.Name] = true
}
// The built-in catalog seeds a plain "Garlic"; the custom one is ours.
if !names["Garlic"] || !names["German Red Garlic"] {
t.Errorf("find_plant(garlic) = %+v, want both the built-in and the custom variety", matches)
}
// Exact match ranks first, so a caller taking [0] gets the least surprising one.
if len(matches) > 0 && matches[0].Name != "Garlic" {
t.Errorf("first match = %q, want the exact name", matches[0].Name)
}
}
// TestCreatePlantIsUserScoped — the catalog belongs to the user, not to any
// garden, so someone with no editable garden can still name a new variety. The
// issue asks for this to be asserted rather than assumed.
func TestCreatePlantIsUserScoped(t *testing.T) {
ctx := context.Background()
svc, owner := newAgentTestService(t)
other, err := svc.Register(ctx, service.RegisterInput{Email: "[email protected]", DisplayName: "B", Password: "password123"})
if err != nil {
t.Fatalf("register: %v", err)
}
box := NewToolbox(svc, other.ID)
raw, _ := json.Marshal(map[string]any{
"name": "Painted Mountain Corn", "category": "vegetable",
"spacingCm": 30.0, "color": "#c08a3f", "icon": "🌽",
})
res := box.Execute(ctx, llm.ToolCall{ID: "1", Name: "create_plant", Arguments: raw})
if res.IsError {
t.Fatalf("create_plant: %s", res.Content)
}
var created struct {
ID int64 `json:"id"`
OwnerID *int64 `json:"ownerId"`
Name string `json:"name"`
}
if err := json.Unmarshal([]byte(res.Content), &created); err != nil {
t.Fatalf("decode: %v", err)
}
if created.OwnerID == nil || *created.OwnerID != other.ID {
t.Errorf("ownerId = %v, want the acting user %d", created.OwnerID, other.ID)
}
// And it stays theirs: the other user's catalog doesn't gain it.
ownerPlants, err := svc.ListPlants(ctx, owner)
if err != nil {
t.Fatalf("ListPlants: %v", err)
}
for _, p := range ownerPlants {
if p.Name == "Painted Mountain Corn" {
t.Error("a plant created by one user showed up in another's catalog")
}
}
}
// TestJournalToolWritesADatedObservation — "note that the west bed has mildew"
// is squarely the kind of thing you say out loud while walking around.
func TestJournalToolWritesADatedObservation(t *testing.T) {
ctx := context.Background()
svc, owner := newAgentTestService(t)
box := NewToolbox(svc, owner)
g, err := svc.CreateGarden(ctx, owner, service.GardenInput{Name: "Plot", WidthCM: 2000, HeightCM: 2000})
if err != nil {
t.Fatalf("garden: %v", err)
}
bed, err := svc.CreateObject(ctx, owner, g.ID, service.ObjectInput{
Kind: domain.KindBed, Name: "West bed", XCM: 500, YCM: 500, WidthCM: 200, HeightCM: 200,
})
if err != nil {
t.Fatalf("bed: %v", err)
}
raw, _ := json.Marshal(map[string]any{
"gardenId": g.ID, "objectId": bed.ID,
"body": "Powdery mildew on the west bed", "observedAt": "2026-08-14",
})
res := box.Execute(ctx, llm.ToolCall{ID: "1", Name: "add_journal_entry", Arguments: raw})
if res.IsError {
t.Fatalf("add_journal_entry: %s", res.Content)
}
entries, _, err := svc.ListJournal(ctx, owner, g.ID, service.JournalQuery{ObjectID: &bed.ID})
if err != nil {
t.Fatalf("ListJournal: %v", err)
}
if len(entries) != 1 {
t.Fatalf("got %d entries, want 1", len(entries))
}
if entries[0].Body != "Powdery mildew on the west bed" || entries[0].ObservedAt != "2026-08-14" {
t.Errorf("unexpected entry: %+v", entries[0])
}
if entries[0].AuthorID != owner {
t.Errorf("author = %d, want the acting user %d", entries[0].AuthorID, owner)
}
}
// newAgentTestService spins up an in-memory pansy with one registered user.
func newAgentTestService(t *testing.T) (*service.Service, int64) {
t.Helper()
ctx := context.Background()
db, err := store.Open(":memory:")
if err != nil {
t.Fatalf("open: %v", err)
}
t.Cleanup(func() { db.Close() })
if err := db.Migrate(ctx); err != nil {
t.Fatalf("migrate: %v", err)
}
svc := service.New(db, &config.Config{Registration: config.RegistrationOpen, LocalAuth: true})
owner, err := svc.Register(ctx, service.RegisterInput{Email: "[email protected]", DisplayName: "A", Password: "password123"})
if err != nil {
t.Fatalf("register: %v", err)
}
return svc, owner.ID
}