Distributed Runner Telemetry: Kernel-Clamped Provenance and Immutability Across Multi-Cloud Seats

#Abstract

As autonomous AI development swarms scale, execution workloads are distributed across heterogeneous cloud topologies: dedicated bare-metal infrastructure (Hetzner), virtual machines (Google Cloud Platform), serverless edge compute (Cloudflare Workers), and local developer harnesses. Collecting verifiable execution telemetry (logs, runner status, compute metrics, terminal exit codes) across these disjoint seats introduces two critical security challenges: (1) Cross-Seat Provenance Spoofing, where compromised or misconfigured runners forge execution claims under another agent’s identity, and (2) Post-Execution Mutability, where terminal receipts are altered after task completion.

In this paper, we present the Kernel-Clamped Runner Provenance Architecture implemented in the Mupot substrate (Flight-004 runner_receipts, Flight-005 PR #1070). We formulate the mathematical model of Authoritative Entity Clamping, define the mechanics of terminal status locks and anti-replay nonce tracking, and establish a dual-fence defense-in-depth against stored injection attacks.

We present empirical production telemetry from multi-seat runner clusters across Hetzner, GCP VM seats (loom-vm), and edge gateways, demonstrating how cryptographic receipt clamping achieves tamper-proof observability across zero-trust multi-cloud agent swarms.

#1. Introduction: The Observability Dilemma in Multi-Cloud Agent Fleets

In centralized software architectures, telemetry and execution logs are captured within a single trusted perimeter (e.g., Kubernetes cluster or VPC).

In sovereign multi-agent swarms, however, compute seats are inherently decentralized and heterogeneous:

• Coordinator Seats: Run in high-memory edge runtimes or specialized CLI harnesses (e.g. agent:loom on Hetzner).
• Heavy Compute Seats: Run on GPU/cloud VM clusters (e.g. loom-vm on GCP us-central1-a).
• Companion Gate Seats: Run in isolated sandboxes executing adversarial mutations (e.g. agent:river).

graph TD
    subgraph Heterogeneous Multi-Cloud Runner Seats
        A1[Hetzner Dedicated Server: seat_1] -->|Emits Telemetry Record| GW[Secure Ingestion Gateway]
        A2[GCP VM Node: seat_2] -->|Emits Telemetry Record| GW
        A3[Local CLI Node: seat_3] -->|Emits Telemetry Record| GW
    end

    subgraph The Anti-Spoofing & Immutability Pipeline
        GW --> B[Protocol Whitelist Sanitizer: http/https/file only]
        B --> C[Ed25519 Replay-Resistant Nonce Validator]
        C --> D[D1 Kernel Entity Clamping Engine]
        D -->|Derives seat_agent_id & squad_id directly from Token Context| E[(D1 runner_receipts Table)]
        
        E --> F[Terminal Status Mutation Lock: completed/failed is Immutable]
        E --> G[Mission Control Radar Visualization /radar?tab=tentacles]
    end

#1.1 The Vulnerability of Trusting Caller Foreign Keys

When an agent seat emits an execution receipt via an HTTP or MCP tool call (runner_record), naive architectures accept foreign key parameters from the request payload:

{
  "seat_agent_id": "7affb004-...", // Unsafe: Caller-supplied ID!
  "squad_id": "squad-alpha",
  "status": "completed",
  "log_url": "file:///var/log/run.log"
}

If a subagent on a shared VM is compromised, misconfigured, or running an outdated prompt, it can submit forged telemetry claiming that an entirely different agent completed a mission, corrupting the historical audit ledger.

To solve this, Mupot establishes Server-Side Authoritative Entity Clamping.

#2. Mathematical Formalization of Kernel-Clamped Provenance

Let $\mathcal{K}$ be the set of cryptographic bearer tokens, and $\mathcal{A}$ be the set of registered agent identities.

Let $\sigma: \mathcal{K} \to \mathcal{A}$ be the authoritative token resolution function executed within the database kernel: $\sigma(k) = a_{\text{bound}}$

#2.1 The Entity Clamping Invariant (Theorem 1)

Let $P = \langle a_{\text{payload}}, s_{\text{payload}}, \text{status}, \text{log\_url} \rangle$ be a telemetry payload submitted by bearer token $k$.

Definition 1 (Authoritative Entity Clamping): A receipt recording function $\mathcal{R}_{\text{clamped}}$ ignores caller-supplied identity claims and derives ownership strictly from kernel token context: $\mathcal{R}_{\text{clamped}}(k, P) = \langle \sigma(k), \text{Squad}(\sigma(k)), P.\text{status}, \text{Sanitize}(P.\text{log\_url}) \rangle$

Theorem 1 (Anti-Spoofing Immunity): For any two distinct agent identities $a_1, a_2 \in \mathcal{A}$ where $a_1 \ne a_2$: $\forall k \in \mathcal{K} \text{ such that } \sigma(k) = a_1, \quad \mathcal{R}_{\text{clamped}}(k, P).\text{seat\_agent\_id} = a_1$ Regardless of whether $P.\text{seat\_agent\_id} = a_2$.

Proof: $\mathcal{R}_{\text{clamped}}$ strictly ignores $P.\text{seat\_agent\_id}$ and assigns $\text{seat\_agent\_id} \leftarrow \sigma(k)$. Since $\sigma(k) = a_1$, the persisted record is bound to $a_1$. Forgery of $a_2$ is mathematically impossible without possessing $k_2$ where $\sigma(k_2) = a_2$. $\blacksquare$

#3. Substrate Implementation Mechanics

#3.1 D1 Schema for Runner Telemetry (Migration 0105)

In Mupot’s D1 SQLite kernel, the runner_receipts table anchors telemetry:

-- migrations/0105_runner_receipts.sql
CREATE TABLE IF NOT EXISTS runner_receipts (
  id TEXT PRIMARY KEY,
  squad_id TEXT NOT NULL,
  seat_agent_id TEXT NOT NULL,
  task_id TEXT,
  session_id TEXT NOT NULL,
  status TEXT NOT NULL CHECK(status IN ('starting', 'running', 'completed', 'failed', 'cancelled')),
  exit_code INTEGER,
  log_url TEXT,
  signature TEXT,
  nonce TEXT UNIQUE,
  created_at INTEGER NOT NULL,
  updated_at INTEGER NOT NULL,
  FOREIGN KEY (squad_id) REFERENCES squads(id),
  FOREIGN KEY (seat_agent_id) REFERENCES agents(id)
);

CREATE INDEX IF NOT EXISTS idx_runner_receipts_seat ON runner_receipts(seat_agent_id, created_at DESC);

#3.2 Terminal Status Immutability Lock

Once a runner receipt transitions to a terminal state (completed, failed, cancelled), it enters an immutable lock:

// src/runner/service.ts
export async function recordRunnerReceipt(
  db: D1Database,
  caller: AuthenticatedTokenContext,
  input: RunnerReceiptInput
): Promise<RunnerReceiptRow> {
  const existing = await getReceiptById(db, input.id);
  
  if (existing) {
    // 1. Cross-Seat Mutation Lock
    if (existing.seat_agent_id !== caller.bound_agent_id) {
      throw new SecurityError("Cross-seat receipt mutation forbidden");
    }
    
    // 2. Terminal Status Immutability Lock
    if (["completed", "failed", "cancelled"].includes(existing.status)) {
      throw new StateError(`Receipt ${input.id} is in terminal status ${existing.status} and cannot be mutated`);
    }
  }

  // 3. Authoritative Entity Clamping & URL Protocol Sanitization
  return insertOrUpdateReceipt(db, {
    ...input,
    seat_agent_id: caller.bound_agent_id, // Authoritative kernel binding
    squad_id: caller.bound_squad_id,
    log_url: sanitizeLogUrl(input.log_url) // Enforces ^https?:// | ^file:///
  });
}

#3.3 Dual-Fence Stored XSS Prevention

To protect human and agent operators inspecting logs on the /radar?tab=tentacles dashboard:

1. Write-Time Defense: The service layer rejects any log_url that does not match ^https?:// or ^file:///.
2. Render-Time Defense: The frontend view layer runs all log URLs through safeLogUrl(), disarming dangerous schemes (e.g. javascript:, data:) before rendering clickable DOM anchors.

#4. Empirical Case Studies from Live Council Operations

#4.1 Case Study: Flight-004 TENTACLES Telemetry Landing (PR #1056)

• The Deployment: Introduced 0105_runner_receipts.sql and the live telemetry stream hook agents/river/scripts/runner-collector.py.
• The Visualization: Wired the /radar?tab=tentacles dashboard panel, providing real-time fan-out telemetry, live seat counts, and log cards across Hetzner, GCP, and local nodes.
• Evidence: 16/16 tests PASS; 13/13 CI checks green; dual sign-off stamped by Athena & River (Commit cc48751d).

#4.2 Case Study: Flight-005 Runner Provenance Hardening (PR #1070)

• The Finding: During multi-cloud expansion to loom-vm on GCP, an audit revealed that unbound MCP tokens could theoretically supply arbitrary seat_agent_id values.
• The Resolution: PR #1070 introduced authoritative D1 entity clamping and replay-resistant Ed25519 nonce tracking.
• Kill-Witness Test: In tests/runner-receipts.test.ts, deliberately forging foreign keys failed with test RED. 13/13 tests PASS (Commit 01b9ed30).

#5. Architectural Axioms for Distributed Agent Provenance

1. Axiom of Kernel Clamping: An ingest API must derive identity from the bearer token’s authoritative cryptographic session, never from caller-supplied payload fields.
2. Axiom of Terminal Immutability: Once a distributed process claims terminal status (exit code recorded), the ledger record must freeze against all subsequent writes.
3. Axiom of Dual-Fence Sanitization: External URI references must be validated at the write-layer boundary and sanitized at the render-layer boundary.
4. Axiom of Replay Resistance: Cryptographic signatures accompanying runner receipts must anchor a unique, non-repeating nonce in a persistent ledger.

#6. Conclusion & Series Synthesis

With the completion of Paper 200.406, the foundational Mumega 200.40x Applied Systems Series forms a unified, mathematically closed architecture for autonomous multi-agent software engineering:

                              Mumega 200.40x Substrate Architecture
                                                │
         ┌──────────────────┬───────────────────┼───────────────────┬──────────────────┐
         ▼                  ▼                   ▼                   ▼                  ▼
     200.401             200.402             200.403             200.404            200.405 & 200.406
  Popperian Loops      Kill-Witness       Clock Invariance     Two-Sided Authz     Decoupled Dispatch &
  (done_when Tasks)    Adversarial Gate   (nowMs Injection)    (No False Promises) Kernel-Clamped Provenance

By substituting descriptive wishes with falsifiable predicates, passive CI with kill-witness mutations, unmocked clocks with parameter injection, and unverified telemetry with kernel-clamped receipts, Mupot proves that autonomous AI swarms can operate with total epistemic integrity.

The complete substrate code, schemas, and test harnesses are openly available under the Mumega Open Science Initiative:

• Repository: https://github.com/Mumega-com/mupot
• Telemetry Surface: /radar?tab=tentacles
• Paper Series DOI: 10.5281/zenodo.mumega.200.406

#References

1. Mumega Synthetic Council. (2026). Paper 200.401: Falsificationist Substrates for Multi-Agent Systems: Beyond Tautological Task Execution. Mumega Paper Series.
2. Mumega Synthetic Council. (2026). Paper 200.402: Kill-Witness Verification: Adversarial Falsification of Code in Autonomous AI Swarms. Mumega Paper Series.
3. Mumega Synthetic Council. (2026). Paper 200.403: Clock Invariance in Decentralized Agent Presence: Eliminating Ephemeral Liveness Spoofs. Mumega Paper Series.
4. Mumega Synthetic Council. (2026). Paper 200.404: The Two-Sided Invariant: Formal Mirroring of Render-Time UI Controls and Distributed Write-Path Authorization. Mumega Paper Series.
5. Mumega Synthetic Council. (2026). Paper 200.405: Safe Planning in Autonomous Swarms: Architectural Decoupling of Backlog Intake from Event-Driven Dispatch. Mumega Paper Series.
6. Mumega Synthetic Council. (2026). ADR-007: Domain-Substrate Decoupling and Autonomous Gate Governance. Mumega Architecture Repository.