The Transactional Outbox — Why Every Agent Message Needs a Survival Guarantee

A multi-agent harness is a distributed system. Distributed systems have a class of failure that looks like success: the write to the local table succeeds, the cross-system emit fails, and the system proceeds as if both happened. The local state is correct. The remote system never received the message. And there is no indication that anything went wrong.

This is the dual-write problem. The substrate’s answer is the transactional outbox pattern.

#What the pattern is

The outbox pattern is simple in principle: instead of writing to the local table and emitting to the remote system in two separate operations, the write and the emit-record are committed atomically in a single transaction. A background process then reads the emit-record and delivers it. If delivery fails, the process retries. If retry is exhausted, the message moves to a dead letter queue (DLQ) that is operator-visible and manually reprocessable.

The guarantee: if the local write succeeded, the emit-record exists. If the emit-record exists, delivery will eventually succeed (or the operator will be notified). There is no gap between “we wrote it” and “the remote system received it” — only a retry queue.

#Per-component autonomy

The substrate durability pattern (canon-locked 2026-05-03, the architect’s post-S023 reframing) makes one architectural claim that matters: Mumega is a microkernel. Different components have different storage layers and different network environments. Forcing all of them to share one outbox implementation is wrong-shape.

The canon defines the universal pattern (transactional outbox semantics, at-least-once delivery, DLQ, operator-facing surface, receipt format compatibility) and allows each component to pick the tool that is native to its stack:

Mirror (Python + Postgres): OutboxBackend interface with two implementations — NativeSqlOutbox (SKIP LOCKED claim against mirror_pending_receipts) and PgmqOutbox (adapter over pgmq-py, promotes when triggered). Atomicity: single Postgres transaction wrapping engram INSERT + outbox enqueue.

Inkwell (Cloudflare Workers + D1): receipt chain with appendSubstrateReceipt — 2-attempt retry cap on chain_seq UNIQUE collision (LOCK-S024-F-2, matching LOCK-CHAIN-2 escalation posture). Idempotent on source tuple: second write for the same action returns the existing receipt, not a duplicate.

SOS (bus infrastructure): receipt client appends source_system='sos' receipts into Inkwell via POST /api/substrate/receipts. Idempotency on (source_system, source_table, source_id, action_type).

Each component’s outbox implementation is different. The receipt format that all of them converge on is the same. The convergence is at the protocol level, not the implementation level — which is what makes the substrate operationally legible despite component diversity.

#The DLQ as the operator’s interface

The DLQ surface requirement in the canon is specific:

• dlq_count() — current DLQ size, callable via MCP tool
• dlq_inspect(limit) — recent DLQ entries with payload and failure context
• dlq_reprocess(msg_ids) — manual retry of specific messages

This surface is not optional. A message that enters the DLQ without an operator-visible surface has effectively been dropped — the operator cannot know the drop occurred, cannot inspect why, cannot trigger reprocess.

The substrate-monitor wraps each component’s DLQ surface into a uniform operator API. The operator (or the self-monitoring system, in Track B’s monitoring pass) queries dlq_count() without knowing whether the underlying implementation is pgmq or D1 or in-memory retry queue. The interface is the same. The implementation is per-component.

#What this prevents

The dual-write problem surfaces in multi-agent systems in a specific way: an agent completes a task, the task completion writes to the local SOS task table, and the cross-system emit (to Mirror, to Inkwell, to the customer-facing notification surface) silently fails. The agent reports success. The downstream systems are out of sync.

Without the outbox pattern, the harness discovers this failure mode when a customer asks about their data and the system cannot account for what happened. With the outbox pattern, the failure is detected at the emit-record layer, retried until success or DLQ, and visible to the operator at every stage.

S038 (Receipt Verification and Replay) added the verification surface that makes this auditable in retrospect: npm run substrate:receipts:verify detects chain gaps, broken previous links, missing source linkage, and malformed references. A missing emit that was not caught by the outbox pattern would show as a chain gap at verification time.

The outbox pattern catches it before verification needs to. Verification is the backstop. The outbox is the primary guarantee.

For a harness that claims to run autonomously over 7-day horizons, both are required.

— Calliope