899059a791
The kernel defined run.Ports.Checkpointer + the checkpoint battery but never drove them (the documented "P2 follow-up"). This wires durable recovery into the run loop so a run interrupted by shutdown can resume on the next boot instead of being lost — the executus-side half of mort's durable-agent-recovery parity (mort #1355). Kernel (run/): - Ports.Checkpointer is now a CheckpointerFactory (Begin per run → a per-run Checkpointer, or nil for a non-durable run). The single per-instance Checkpointer couldn't distinguish runs; a factory mints one per run, matching mort's agentexec.CheckpointerFactory. - RunInfo gains GuildID + ModelTier (so the factory can build resume meta); RunCheckpointState gains CompletedPhases + ActivePhase (+ PhaseOutput). - run/checkpoint.go: ResumeState + WithResumeState / WithExistingCheckpointer context carriers, classifyCheckpointOutcome (success→Complete, shutdown→leave for boot recovery, else→Fail using run.ErrShutdown), and finalizeCheckpoint. - run/executor.go: resolve the per-run checkpointer (existing-from-ctx on a recovery re-run, else factory.Begin); single-loop wraps the step observer to accumulate the transcript + Save each step (host throttles), and a recovered run seeds the saved transcript via WithHistory and continues with no new input; finalize on exit. - run/phases.go: phase-boundary checkpointing — record completed phases after each phase; a resumed run skips already-completed phases (the interrupted phase re-runs from its start — boundary-granular, documented; only the single-loop path resumes mid-loop). Battery (checkpoint/): NewFactory wires the battery into the factory port (per-run handle, meta derived from RunInfo); RunCheckpoint + handle.Save carry the phase fields. Tests (run/checkpoint_test.go): the finalize decision matrix; single-loop Save+Complete; terminal-error Fail; resume seeds history; phase-boundary Saves completed phases; resume skips completed phases. Full ./... green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
95 lines
3.4 KiB
Go
95 lines
3.4 KiB
Go
package run
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
|
|
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
|
|
)
|
|
|
|
// Durable-recovery plumbing for the executor. The Checkpointer port (set via
|
|
// Ports.Checkpointer, a CheckpointerFactory) persists a run's resumable progress
|
|
// during the loop; on boot a host re-dispatches an interrupted run through the
|
|
// executor with a ResumeState (the saved transcript / completed phases) so it
|
|
// CONTINUES rather than restarting, reusing the SAME durable record via an
|
|
// existing Checkpointer. Both are carried into Run via the context (mirrors
|
|
// mort's agentexec.WithResumeState / WithExistingCheckpointer).
|
|
|
|
// ResumeState carries a recovered run's prior progress into Run so the run
|
|
// continues instead of restarting. The host's recovery path sets it via
|
|
// WithResumeState; the executor reads it (single-loop seeds the saved transcript
|
|
// as history; multi-phase skips completed phases and seeds the active phase).
|
|
type ResumeState struct {
|
|
History []llm.Message // single-loop transcript OR active-phase transcript
|
|
CompletedPhases []PhaseOutput // multi-phase: outputs of finished phases, in order
|
|
ActivePhase string // multi-phase: the phase that was in flight
|
|
}
|
|
|
|
type resumeStateKey struct{}
|
|
|
|
// WithResumeState carries a recovered run's prior progress into Run.
|
|
func WithResumeState(ctx context.Context, rs *ResumeState) context.Context {
|
|
return context.WithValue(ctx, resumeStateKey{}, rs)
|
|
}
|
|
|
|
func resumeStateFromContext(ctx context.Context) *ResumeState {
|
|
rs, _ := ctx.Value(resumeStateKey{}).(*ResumeState)
|
|
return rs
|
|
}
|
|
|
|
type existingCheckpointerKey struct{}
|
|
|
|
// WithExistingCheckpointer carries a pre-existing Checkpointer into Run so a
|
|
// recovery re-run reuses the SAME durable record (the executor uses it instead of
|
|
// calling Ports.Checkpointer.Begin).
|
|
func WithExistingCheckpointer(ctx context.Context, cp Checkpointer) context.Context {
|
|
return context.WithValue(ctx, existingCheckpointerKey{}, cp)
|
|
}
|
|
|
|
func existingCheckpointerFromContext(ctx context.Context) Checkpointer {
|
|
cp, _ := ctx.Value(existingCheckpointerKey{}).(Checkpointer)
|
|
return cp
|
|
}
|
|
|
|
// checkpointOutcome is the finalize decision for a durable run.
|
|
type checkpointOutcome int
|
|
|
|
const (
|
|
checkpointComplete checkpointOutcome = iota
|
|
checkpointLeaveRunning
|
|
checkpointFail
|
|
)
|
|
|
|
// classifyCheckpointOutcome maps (run error, cancellation cause) to the durable
|
|
// finalize action: success clears the checkpoint (Complete); a shutdown-caused
|
|
// cancellation leaves the record so boot recovery picks it up (neither
|
|
// Complete nor Fail); anything else (model error, tool loop, the run's own
|
|
// deadline, a critic kill, a caller cancel) is terminal (Fail). Mirrors mort's
|
|
// agentexec.classifyCheckpointOutcome.
|
|
func classifyCheckpointOutcome(runErr, cause error) checkpointOutcome {
|
|
switch {
|
|
case runErr == nil:
|
|
return checkpointComplete
|
|
case errors.Is(cause, ErrShutdown):
|
|
return checkpointLeaveRunning
|
|
default:
|
|
return checkpointFail
|
|
}
|
|
}
|
|
|
|
// finalizeCheckpoint applies the outcome to the per-run checkpointer (nil-safe).
|
|
// Runs on a detached context so a cancelled run still records its terminal state.
|
|
func finalizeCheckpoint(ctx context.Context, cp Checkpointer, runErr error, cause error) {
|
|
if cp == nil {
|
|
return
|
|
}
|
|
switch classifyCheckpointOutcome(runErr, cause) {
|
|
case checkpointComplete:
|
|
_ = cp.Complete(detach(ctx))
|
|
case checkpointFail:
|
|
_ = cp.Fail(detach(ctx), runErr)
|
|
case checkpointLeaveRunning:
|
|
// Interrupted by shutdown: leave the record for boot recovery.
|
|
}
|
|
}
|