Loop Engineering, Eval Gating, and Apprentice Calibration on a Falsificationist Substrate

#Abstract

A survey of emerging frontier AI agent paradigms—specifically Hanako’s (@hanakoxbt) Loop Engineering & Eval Engineering—reveals a strong industry convergence toward concepts Mumega has pioneered:

1. Moving from static single-shot chat prompts to autonomous continuous loops (cron routines, event buses).
2. Moving from subjective LLM self-evaluation to deterministic evaluation gates (Eval Engineering).
3. Utilizing Apprentice Architectures where human supervisory choices continuously calibrate agent prompt policies in structured batches.

This note documents the comparative mapping against Mupot, validates the existing architectural moat, and identifies two concrete enhancements: failure fingerprinting on routine runs, and an apprentice decision-comparison engine scored against sovereign human choices.

#1. Comparative Matrix

#2. Synthesis

graph TD
    subgraph 1. The Autonomous Loop Plane
        A[Durable Object Routines Engine] -->|Scheduled / Event Trigger| B[Worker Agent Run]
        B -->|Emits Artifact & Telemetry| C[(runner_receipts)]
    end

    subgraph 2. The Eval & Falsification Gate
        C --> D{Dual-Lens Gate}
        D -->|Passes Invariants & Kill-Witness| E[Merge / Settle]
        D -->|Fails 3x Consecutively| F[Self-Healing Diagnostic Agent]
        F -->|Proposes Patch PR| D
    end

    subgraph 3. The Apprentice Calibration Feedback Loop
        E --> G[Human Sovereign Verdict Ledger]
        G -->|Batches 50-100 Decisions| H[Apprentice Meta-Calibrator]
        H -->|Generates Evaluated Prompt Deltas| I[Cortex Decision Policy Update]
        I --> A
    end

#Lesson 1: The Apprentice Loop

Human operators make decisions (approving, rejecting, editing tasks, overriding priorities). Instead of letting these decisions remain static log entries, an automated background loop aggregates them into calibration batches ($N=50$).

When a shadow decision engine disagrees with the sovereign human choice, the delta is structured into a fine-tuning / prompt-refinement dataset to mathematically minimize the divergence vector.

#Lesson 2: Self-Healing Routine Loops

Background cron loops inevitably hit edge-case errors (API shape shifts, unexpected rate limits). Rather than stopping at an alert badge, the loop triggers an apprentice repair agent.

For stuck routines or duplicate error loops, Mupot captures the execution stack trace in runner_receipts, synthesizes a minimal reproduction test, and opens a draft patch for human/gate review.

#3. Integration Blueprint

#Action 1 — Failure Fingerprinting

When a routine run fails, hash the error signature and link it to prior occurrences to detect runaway cyclic loops.

#Action 2 — Apprentice Decision Comparison

export interface ApprenticeEvaluationRecord {
  task_id: string;
  cortex_proposed_action: "dispatch" | "noop" | "reassign";
  cortex_rationale: string;
  sovereign_actual_action: "dispatch" | "noop" | "reassign";
  agreement: boolean;
  divergence_cause?: "priority_mismatch" | "scope_misunderstanding" | "unknown_dependency";
}

This record feeds an automated prompt-tuning optimizer at the close of an observe-only pilot.

#4. Conclusion

External validation from frontier AI builders demonstrates that Mupot’s foundational architecture (D1 SQLite kernel, kill-witness gates, strict capability grants) already implements loop engineering and eval engineering as substrate primitives. The remaining closed loop is apprentice calibration: batching sovereign human verdicts into policy updates rather than leaving them as unread logs.