Driven by measurement, not taste. Full findings: docs/audits/2026-07-24-gifsmith-quality.md in mort#1503. Agent-prompt companion: executus#26.
Method: 128 production gifsmith runs / 631 code_exec calls from the live DB, plus a local replica of the agent loop — GLM-5.2 over Ollama Cloud (the live thinking tier), same tool schemas, same skill_use payload, and code_exec running in the real mort-codeexec image under the production container contract. Four prompts × four variants, A/B'd.
What's wrong today
Staging, not code. The renders that "work" still read badly, and always the same four ways: the subject drawn 5–10% of frame height under an empty sky filling 60% of the frame; long runs of pixel-identical frames (35% of sampled pairs below the visible-change threshold); captions narrating action that was never drawn ("*CRASH* garbage everywhere!" with no garbage on screen); flat single-tone shapes with no shadows. The recipe never asked for anything else. Reproduced across seeds.
26% of code_exec calls crash — 161 of 631. Each is a regenerated 6–8k-token program, 60–100 s of model time, and one of only three allowed render passes. The largest identifiable cause is the sandbox's own scratch budget: 24 of the 161 are OSError: [Errno 28] No space left on device, because this skill tells the agent to render frames into /tmp and /tmp is a 16 MiB tmpfs — about 13 seconds of busy 640×480 PNGs. PIL buries it under an '_idat' object has no attribute 'fileno' chain, so the agent sees a PIL bug and retries the same thing.
What this changes
SKILL.md gains four checkable staging rules (fill the frame, every frame moves, show it don't caption it, give it depth) plus easing/anticipation/overshoot/squash-and-stretch, and the self-critique becomes a checklist where a "no" on depiction, subject size or caption legibility mandates a re-render rather than suggesting one.
scripts/gifsmith.py takes over everything the agent used to re-derive per run — scratch dir, frame loop, numbering, canvas size, contact sheet, encode — leaving it to write only @anim.scene(...) drawing functions. Consequences:
A crash stops being total: a scene that raises is skipped, the others still render, encode and ship, and the report names the broken one. Seen live — a run hit TypeError: draw_glass() got an unexpected keyword argument 'pivot' and still delivered.
Locked even canvas size, so ffmpeg can't fail on jittering frames (the bbox_inches='tight' trap).
Drawing vocabulary re-exported, so a missing from matplotlib.patches import … can't NameError.
Multi-minute pieces become possible at all. Anything past the GIF length threshold is an MP4, and an MP4 encodes in one streaming pass — so frames go straight into ffmpeg's stdin and never touch disk. Verified: 1350 frames / 90 s in a sandbox with 16 MiB /tmp and 64 MiB /workspace, 0 MiB of frames on disk. That's ~116 MiB of PNGs the old way.
encode.py is removed — the harness supersedes it.
Results
Four prompts per row. Output tokens are the model's own generation — the dominant cost and latency term.
variant
delivered
crashed calls
output tokens
median wall clock
near-duplicate frames
baseline (current)
4/4
5
107k
596 s
39%
staging rails only
4/4
4
97k
438 s
43%
rails + harness, first cut
4/4
5
80k
430 s
33%
rails + harness, hardened
4/4
1
52k
206 s
28%
The quality change isn't in the table: subjects went from unreadably small to filling the frame, and the action went from narrated to drawn. Consistent across both seeds of every prompt.
Two diagnostics closed the gap between the first cut and the hardened version, and are worth knowing: a SyntaxError in the emitted program happens before the harness loads and is the one failure it cannot absorb; and /workspace is wiped between calls, so gifsmith.py must be re-passed in files_in on every call including retries (one run forgot on its third attempt). Both are now explicit in the skill text.
Shipping
Gated as usual — merging here doesn't move production. .skillpack check then .skillpack apply gif moves the pin (currently fae89a1).
Driven by measurement, not taste. Full findings: `docs/audits/2026-07-24-gifsmith-quality.md` in [mort#1503](https://gitea.stevedudenhoeffer.com/steve/mort/pulls/1503). Agent-prompt companion: [executus#26](https://gitea.stevedudenhoeffer.com/steve/executus/pulls/26).
Method: 128 production `gifsmith` runs / 631 `code_exec` calls from the live DB, plus a local replica of the agent loop — GLM-5.2 over Ollama Cloud (the live `thinking` tier), same tool schemas, same `skill_use` payload, and `code_exec` running in the real `mort-codeexec` image under the production container contract. Four prompts × four variants, A/B'd.
## What's wrong today
**Staging, not code.** The renders that "work" still read badly, and always the same four ways: the subject drawn 5–10% of frame height under an empty sky filling 60% of the frame; long runs of pixel-identical frames (35% of sampled pairs below the visible-change threshold); captions *narrating* action that was never drawn (`"*CRASH* garbage everywhere!"` with no garbage on screen); flat single-tone shapes with no shadows. The recipe never asked for anything else. Reproduced across seeds.
**26% of `code_exec` calls crash** — 161 of 631. Each is a regenerated 6–8k-token program, 60–100 s of model time, and one of only three allowed render passes. The largest identifiable cause is the sandbox's own scratch budget: **24 of the 161 are `OSError: [Errno 28] No space left on device`**, because this skill tells the agent to render frames into `/tmp` and `/tmp` is a 16 MiB tmpfs — about 13 seconds of busy 640×480 PNGs. PIL buries it under an `'_idat' object has no attribute 'fileno'` chain, so the agent sees a PIL bug and retries the same thing.
## What this changes
**SKILL.md** gains four checkable staging rules (fill the frame, every frame moves, show it don't caption it, give it depth) plus easing/anticipation/overshoot/squash-and-stretch, and the self-critique becomes a checklist where a "no" on depiction, subject size or caption legibility *mandates* a re-render rather than suggesting one.
**`scripts/gifsmith.py`** takes over everything the agent used to re-derive per run — scratch dir, frame loop, numbering, canvas size, contact sheet, encode — leaving it to write only `@anim.scene(...)` drawing functions. Consequences:
- A crash stops being total: a scene that raises is skipped, the others still render, encode and ship, and the report names the broken one. Seen live — a run hit `TypeError: draw_glass() got an unexpected keyword argument 'pivot'` and still delivered.
- Locked even canvas size, so ffmpeg can't fail on jittering frames (the `bbox_inches='tight'` trap).
- Drawing vocabulary re-exported, so a missing `from matplotlib.patches import …` can't `NameError`.
- **Multi-minute pieces become possible at all.** Anything past the GIF length threshold is an MP4, and an MP4 encodes in one streaming pass — so frames go straight into ffmpeg's stdin and never touch disk. Verified: 1350 frames / 90 s in a sandbox with 16 MiB `/tmp` and 64 MiB `/workspace`, 0 MiB of frames on disk. That's ~116 MiB of PNGs the old way.
`encode.py` is removed — the harness supersedes it.
## Results
Four prompts per row. Output tokens are the model's own generation — the dominant cost and latency term.
| variant | delivered | crashed calls | output tokens | median wall clock | near-duplicate frames |
|---|---|---|---|---|---|
| baseline (current) | 4/4 | 5 | 107k | 596 s | 39% |
| staging rails only | 4/4 | 4 | 97k | 438 s | 43% |
| rails + harness, first cut | 4/4 | 5 | 80k | 430 s | 33% |
| **rails + harness, hardened** | **4/4** | **1** | **52k** | **206 s** | **28%** |
The quality change isn't in the table: subjects went from unreadably small to filling the frame, and the action went from narrated to drawn. Consistent across both seeds of every prompt.
Two diagnostics closed the gap between the first cut and the hardened version, and are worth knowing: a `SyntaxError` in the emitted program happens *before* the harness loads and is the one failure it cannot absorb; and `/workspace` is wiped between calls, so `gifsmith.py` must be re-passed in `files_in` on every call including retries (one run forgot on its third attempt). Both are now explicit in the skill text.
## Shipping
Gated as usual — merging here doesn't move production. `.skillpack check` then `.skillpack apply gif` moves the pin (currently `fae89a1`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01TfKDgrHTmB12T9JhL9bUYf
Two changes, both driven by measurement against 128 production gifsmith runs
and a local replica of the agent loop (docs/audits/2026-07-24-gifsmith-quality.md
in mort).
Staging rules. The renders that "worked" still read badly, always the same four
ways: the subject drawn 5-10% of the frame height under an empty sky, long runs
of pixel-identical frames, captions narrating action that was never drawn, and
flat untextured shapes. The recipe never asked for anything else. SKILL.md now
states four checkable rules — fill the frame, every frame moves, show it don't
caption it, give it depth — plus easing/anticipation/squash-and-stretch, and the
self-critique makes a "no" on depiction, subject size or caption legibility a
mandatory re-render instead of a suggestion.
Render harness. 26% of production code_exec calls end in a traceback, and every
crash class lives in plumbing the agent re-derives each run: scratch dir, frame
loop, numbering, canvas size, contact sheet, encode. scripts/gifsmith.py owns all
of it and the agent writes only @anim.scene drawing functions. A scene that
raises is now skipped while the others still render and ship, the canvas is
size-locked so ffmpeg cannot fail on jittering frames, and the drawing vocabulary
is re-exported so a missing import cannot NameError.
The harness also unblocks the multi-minute pieces this skill advertises but could
never produce: /tmp is a 16 MiB tmpfs (~13s of frames) and /workspace 64 MiB
(~51s), which is why "No space left on device" is 15% of all production crashes.
Anything past the GIF length threshold is an MP4, and an MP4 encodes in one
streaming pass, so those frames now go straight into ffmpeg's stdin and never
touch disk — verified at 1350 frames / 90 s with 0 MiB on disk.
encode.py is removed; the harness supersedes it.
This repo has no CI and no adversarial reviewer, so these come from reading the
harness back against the production sandbox contract rather than from a bot.
The size estimate was wrong by ~1000x in the wrong direction. `width * height *
9e-5` was never MiB — for a 640x480 frame it yields 27.6, so a trivial 2-second
GIF "projected" 800+ MiB and every render would have tripped the streaming
branch and come back as an MP4. The bench missed it because it bind-mounted
/workspace from the host disk, where statvfs reports hundreds of GB and the
threshold never fires; production mounts a 64 MiB tmpfs, where it always would
have. Replaced with a named PNG_BYTES_PER_PIXEL = 0.29, measured (86 KiB per
busy 640x480 frame). Verified against the real tmpfs: a 6s piece now estimates
7.6 MiB against a 38 MiB budget and stays a GIF, while a 90s piece streams.
Three smaller ones on the streaming path:
- ffmpeg's stderr was a pipe nobody drained until the encode finished, so a
chatty encoder could fill the 64 KiB buffer, block, stop reading stdin, and
deadlock a long render halfway. It writes to a file now.
- _close_pipe assumed a live pipe; if ffmpeg had already died, closing stdin
raised BrokenPipeError and buried the real exit code and log.
- A failed encode could leave a half-written /workspace/final.* behind, which
still comes back as a files_out file_id — a broken artifact that looks
deliverable is worse than none, so it's removed on the way out.
steve
merged commit 0c05ec8c60 into main2026-07-24 23:12:20 +00:00
steve
deleted branch feat/gif-staging-rails-and-harness2026-07-24 23:12:20 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Driven by measurement, not taste. Full findings:
docs/audits/2026-07-24-gifsmith-quality.mdin mort#1503. Agent-prompt companion: executus#26.Method: 128 production
gifsmithruns / 631code_execcalls from the live DB, plus a local replica of the agent loop — GLM-5.2 over Ollama Cloud (the livethinkingtier), same tool schemas, sameskill_usepayload, andcode_execrunning in the realmort-codeexecimage under the production container contract. Four prompts × four variants, A/B'd.What's wrong today
Staging, not code. The renders that "work" still read badly, and always the same four ways: the subject drawn 5–10% of frame height under an empty sky filling 60% of the frame; long runs of pixel-identical frames (35% of sampled pairs below the visible-change threshold); captions narrating action that was never drawn (
"*CRASH* garbage everywhere!"with no garbage on screen); flat single-tone shapes with no shadows. The recipe never asked for anything else. Reproduced across seeds.26% of
code_execcalls crash — 161 of 631. Each is a regenerated 6–8k-token program, 60–100 s of model time, and one of only three allowed render passes. The largest identifiable cause is the sandbox's own scratch budget: 24 of the 161 areOSError: [Errno 28] No space left on device, because this skill tells the agent to render frames into/tmpand/tmpis a 16 MiB tmpfs — about 13 seconds of busy 640×480 PNGs. PIL buries it under an'_idat' object has no attribute 'fileno'chain, so the agent sees a PIL bug and retries the same thing.What this changes
SKILL.md gains four checkable staging rules (fill the frame, every frame moves, show it don't caption it, give it depth) plus easing/anticipation/overshoot/squash-and-stretch, and the self-critique becomes a checklist where a "no" on depiction, subject size or caption legibility mandates a re-render rather than suggesting one.
scripts/gifsmith.pytakes over everything the agent used to re-derive per run — scratch dir, frame loop, numbering, canvas size, contact sheet, encode — leaving it to write only@anim.scene(...)drawing functions. Consequences:TypeError: draw_glass() got an unexpected keyword argument 'pivot'and still delivered.bbox_inches='tight'trap).from matplotlib.patches import …can'tNameError./tmpand 64 MiB/workspace, 0 MiB of frames on disk. That's ~116 MiB of PNGs the old way.encode.pyis removed — the harness supersedes it.Results
Four prompts per row. Output tokens are the model's own generation — the dominant cost and latency term.
The quality change isn't in the table: subjects went from unreadably small to filling the frame, and the action went from narrated to drawn. Consistent across both seeds of every prompt.
Two diagnostics closed the gap between the first cut and the hardened version, and are worth knowing: a
SyntaxErrorin the emitted program happens before the harness loads and is the one failure it cannot absorb; and/workspaceis wiped between calls, sogifsmith.pymust be re-passed infiles_inon every call including retries (one run forgot on its third attempt). Both are now explicit in the skill text.Shipping
Gated as usual — merging here doesn't move production.
.skillpack checkthen.skillpack apply gifmoves the pin (currentlyfae89a1).🤖 Generated with Claude Code
https://claude.ai/code/session_01TfKDgrHTmB12T9JhL9bUYf