feat(run): durable checkpoint + resume (wire Ports.Checkpointer)
The kernel defined run.Ports.Checkpointer + the checkpoint battery but never drove them (the documented "P2 follow-up"). This wires durable recovery into the run loop so a run interrupted by shutdown can resume on the next boot instead of being lost — the executus-side half of mort's durable-agent-recovery parity (mort #1355). Kernel (run/): - Ports.Checkpointer is now a CheckpointerFactory (Begin per run → a per-run Checkpointer, or nil for a non-durable run). The single per-instance Checkpointer couldn't distinguish runs; a factory mints one per run, matching mort's agentexec.CheckpointerFactory. - RunInfo gains GuildID + ModelTier (so the factory can build resume meta); RunCheckpointState gains CompletedPhases + ActivePhase (+ PhaseOutput). - run/checkpoint.go: ResumeState + WithResumeState / WithExistingCheckpointer context carriers, classifyCheckpointOutcome (success→Complete, shutdown→leave for boot recovery, else→Fail using run.ErrShutdown), and finalizeCheckpoint. - run/executor.go: resolve the per-run checkpointer (existing-from-ctx on a recovery re-run, else factory.Begin); single-loop wraps the step observer to accumulate the transcript + Save each step (host throttles), and a recovered run seeds the saved transcript via WithHistory and continues with no new input; finalize on exit. - run/phases.go: phase-boundary checkpointing — record completed phases after each phase; a resumed run skips already-completed phases (the interrupted phase re-runs from its start — boundary-granular, documented; only the single-loop path resumes mid-loop). Battery (checkpoint/): NewFactory wires the battery into the factory port (per-run handle, meta derived from RunInfo); RunCheckpoint + handle.Save carry the phase fields. Tests (run/checkpoint_test.go): the finalize decision matrix; single-loop Save+Complete; terminal-error Fail; resume seeds history; phase-boundary Saves completed phases; resume skips completed phases. Full ./... green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,94 @@
|
||||
package run
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
|
||||
"gitea.stevedudenhoeffer.com/steve/majordomo/llm"
|
||||
)
|
||||
|
||||
// Durable-recovery plumbing for the executor. The Checkpointer port (set via
|
||||
// Ports.Checkpointer, a CheckpointerFactory) persists a run's resumable progress
|
||||
// during the loop; on boot a host re-dispatches an interrupted run through the
|
||||
// executor with a ResumeState (the saved transcript / completed phases) so it
|
||||
// CONTINUES rather than restarting, reusing the SAME durable record via an
|
||||
// existing Checkpointer. Both are carried into Run via the context (mirrors
|
||||
// mort's agentexec.WithResumeState / WithExistingCheckpointer).
|
||||
|
||||
// ResumeState carries a recovered run's prior progress into Run so the run
|
||||
// continues instead of restarting. The host's recovery path sets it via
|
||||
// WithResumeState; the executor reads it (single-loop seeds the saved transcript
|
||||
// as history; multi-phase skips completed phases and seeds the active phase).
|
||||
type ResumeState struct {
|
||||
History []llm.Message // single-loop transcript OR active-phase transcript
|
||||
CompletedPhases []PhaseOutput // multi-phase: outputs of finished phases, in order
|
||||
ActivePhase string // multi-phase: the phase that was in flight
|
||||
}
|
||||
|
||||
type resumeStateKey struct{}
|
||||
|
||||
// WithResumeState carries a recovered run's prior progress into Run.
|
||||
func WithResumeState(ctx context.Context, rs *ResumeState) context.Context {
|
||||
return context.WithValue(ctx, resumeStateKey{}, rs)
|
||||
}
|
||||
|
||||
func resumeStateFromContext(ctx context.Context) *ResumeState {
|
||||
rs, _ := ctx.Value(resumeStateKey{}).(*ResumeState)
|
||||
return rs
|
||||
}
|
||||
|
||||
type existingCheckpointerKey struct{}
|
||||
|
||||
// WithExistingCheckpointer carries a pre-existing Checkpointer into Run so a
|
||||
// recovery re-run reuses the SAME durable record (the executor uses it instead of
|
||||
// calling Ports.Checkpointer.Begin).
|
||||
func WithExistingCheckpointer(ctx context.Context, cp Checkpointer) context.Context {
|
||||
return context.WithValue(ctx, existingCheckpointerKey{}, cp)
|
||||
}
|
||||
|
||||
func existingCheckpointerFromContext(ctx context.Context) Checkpointer {
|
||||
cp, _ := ctx.Value(existingCheckpointerKey{}).(Checkpointer)
|
||||
return cp
|
||||
}
|
||||
|
||||
// checkpointOutcome is the finalize decision for a durable run.
|
||||
type checkpointOutcome int
|
||||
|
||||
const (
|
||||
checkpointComplete checkpointOutcome = iota
|
||||
checkpointLeaveRunning
|
||||
checkpointFail
|
||||
)
|
||||
|
||||
// classifyCheckpointOutcome maps (run error, cancellation cause) to the durable
|
||||
// finalize action: success clears the checkpoint (Complete); a shutdown-caused
|
||||
// cancellation leaves the record so boot recovery picks it up (neither
|
||||
// Complete nor Fail); anything else (model error, tool loop, the run's own
|
||||
// deadline, a critic kill, a caller cancel) is terminal (Fail). Mirrors mort's
|
||||
// agentexec.classifyCheckpointOutcome.
|
||||
func classifyCheckpointOutcome(runErr, cause error) checkpointOutcome {
|
||||
switch {
|
||||
case runErr == nil:
|
||||
return checkpointComplete
|
||||
case errors.Is(cause, ErrShutdown):
|
||||
return checkpointLeaveRunning
|
||||
default:
|
||||
return checkpointFail
|
||||
}
|
||||
}
|
||||
|
||||
// finalizeCheckpoint applies the outcome to the per-run checkpointer (nil-safe).
|
||||
// Runs on a detached context so a cancelled run still records its terminal state.
|
||||
func finalizeCheckpoint(ctx context.Context, cp Checkpointer, runErr error, cause error) {
|
||||
if cp == nil {
|
||||
return
|
||||
}
|
||||
switch classifyCheckpointOutcome(runErr, cause) {
|
||||
case checkpointComplete:
|
||||
_ = cp.Complete(detach(ctx))
|
||||
case checkpointFail:
|
||||
_ = cp.Fail(detach(ctx), runErr)
|
||||
case checkpointLeaveRunning:
|
||||
// Interrupted by shutdown: leave the record for boot recovery.
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user