Correlated Blind Spots in Same-Lineage Review: A Preregistered Design for Measuring Reviewer Diversity in Multi-Agent Code Gates

#Abstract

Multi-agent engineering systems increasingly place two or more LLM reviewers in front of a merge. The implicit assumption is that a second reviewer adds coverage. That assumption holds only if reviewer failure modes are independent; to the degree they are correlated, the second reviewer is redundancy priced as coverage.

We report one field observation from a production substrate in which two reviewers drawn from a single model family both approved a change to an authorization path, and a reviewer from a different family identified a defect neither had found. We deliberately do not promote this observation to a finding. It is a single instance, and it is confounded: the third reviewer differed from the first two in prompt as well as in lineage.

This paper contributes the design rather than the result. We preregister a nested-baseline experiment that separates lineage diversity from prompt diversity, define conditional detection as the primary metric, specify negative controls, and state in advance the conditions under which the hypothesis is rejected.

#1. The motivating observation, stated with its confound

On 2026-08-14, a pull request touching token-identity resolution in a production multi-tenant substrate was reviewed by three agents before and after merge.

The change contained a conditional write introduced during a hand-resolved merge conflict, of the form:

if (knownNonDirectory) auth.boundAgentId = token.bound_agent_id ?? null

The assigned value is an input to capability resolution downstream.

Three reviews occurred:

The defect R3 found was not incorrect behaviour. The code was correct. The defect was that the line had no regression protection: mutating the assigned value to a foreign agent identifier left the full suite of 5,453 tests green, as did removing the tenant scoping from the same query. The builder had mutated the predicate, observed one test go red, and reported the line covered — an inference that is valid for the guard and invalid for the value.

R1 and R2 read the source and could not distinguish “this code is correct” from “this code is unprotected.” Those are different properties, and source reading establishes only the first.

The confound. R3 differed from R1 and R2 in two dimensions simultaneously: model lineage and review prompt. R3 was instructed to hunt for exploitability and to run mutations; R1 and R2 were instructed to assess correctness. A sufficient explanation of the observation is therefore “adversarial prompts find coverage gaps that correctness prompts do not,” which requires no lineage effect at all.

We consider this the more parsimonious explanation of the single observation, and we state it first for that reason. The experiment below exists because the two explanations have different operational consequences: if the effect is prompt-borne, a second reviewer on the same model with a different brief is sufficient and cheap; if it is lineage-borne, organisations paying for a second same-family reviewer are buying less than they believe.

#2. The question, stated testably

Let $D$ be a defect present in a change under review. Let $A$ and $B$ be two reviewer configurations.

Define conditional detection:

This, rather than marginal detection rate, is the quantity of operational interest. A second reviewer whose marginal detection is high but whose conditional detection is near zero contributes nothing: it finds what was already going to be found.

Perfect redundancy is $\mathrm{CD}(B \mid A) \approx 0$. Perfect complementarity is $\mathrm{CD}(B \mid A) \approx \Pr[B \text{ detects } D]$, i.e. $B$‘s success is independent of $A$‘s failure.

H1 (lineage). For fixed prompt, cross-family reviewer pairs have higher conditional detection than same-family pairs.

H2 (prompt). For fixed model, cross-prompt reviewer pairs have higher conditional detection than same-prompt pairs.

H3 (interaction). The effects are not additive; cross-family and cross-prompt exceeds the sum of the individual contributions.

H1 and H2 are separately estimable only if lineage and prompt are varied orthogonally. The motivating observation varies them together, which is precisely why it cannot settle the question.

#3. Nested baselines

Following the discipline that a candidate must beat every simpler explanation that has been given a fair chance, we specify five nested configurations. Each is a second reviewer added to a fixed first reviewer $R_1$.

Promotion rule, fixed in advance. Support for a lineage effect requires B4 to beat B0–B3 on held-out defects by the preselected metric, with uncertainty estimates. B4 beating B0–B2 while failing to beat B3 is evidence for a prompt effect and against a lineage effect, and must be reported as such.

B3 is the load-bearing baseline. It is the one the motivating observation omitted, and it is the one most likely to explain the result.

#4. Defect corpus

The experiment requires defects with ground truth. We specify two sources and require both, reported separately.

Corpus A — mutation-derived (synthetic, high n). Apply mutation operators to production code with a known-passing suite, retaining only mutants that survive the suite. A surviving mutant is by construction a defect the existing tests do not catch, which is exactly the class R1 and R2 missed. Operators, drawn from the failure modes observed in the field:

• predicate/value split: in if (P) x = V, mutate V while leaving P intact
• scope removal: drop a tenant or ownership clause from a query predicate
• liveness weakening: replace a composite liveness predicate with one of its conjuncts
• receipt weakening: replace a structured outcome check with a bare success boolean

Corpus B — historical (real, low n). Defects recovered from the repository’s own history: changes later fixed, with the fixing commit as ground truth. Lower n, no synthetic-defect artificiality, and subject to hindsight contamination in the framing of the review task. Must be reviewed blind to the fix.

Corpus A carries the risk that mutation-derived defects are unrepresentative of defects humans and agents actually introduce. Corpus B carries the risk of small n. Neither alone is sufficient; agreement between them is the evidence.

#5. Preregistration protocol

Frozen before any reviewer is run, recorded with a checksum in version control:

1. The defect corpus and its labels
2. The train/held-out split
3. Reviewer prompts, verbatim, for every arm
4. The primary metric (conditional detection) and its estimator
5. The promotion rule of §3
6. The stopping rule and target n

Reviewer outputs are scored against labels by an automated matcher on file-and-line, not by a judge model. A judge model would introduce a third lineage into the measurement instrument, which is the failure this paper is about.

#6. Negative controls

Each must produce a null result, or the pipeline is measuring something other than what it claims.

• Label shuffle. Randomly permute lineage labels across reviews. Any apparent lineage effect must vanish.
• Clean changes. Include changes with no injected defect. Reviewers reporting defects here yield a false-positive rate, which must be reported alongside detection; a reviewer that flags everything has perfect detection and no value.
• Equal-compute control. B4 must not be given more tokens, more turns, or more tool calls than B3. Diversity must not be confounded with budget.
• Order control. Randomise which reviewer runs first; report whether $\mathrm{CD}(B\mid A)$ is asymmetric in ordering.

#7. Falsification

The lineage hypothesis is rejected if any of the following hold on held-out data:

• B4 does not exceed B3 by more than the estimated uncertainty
• The label shuffle reproduces the effect
• The effect disappears under the equal-compute control
• B4’s advantage is accounted for by its false-positive rate

We commit in advance to reporting a null result. The motivating observation is a single instance, and the most likely outcome of a well-controlled experiment is that it is explained by B3.

#8. Threats to validity

The observation is n=1 and participant-recorded. The author was a participant in the reviews described, was the builder of some of the reviewed code, and selected which instances to record. This is disclosed, not corrected.

Model versions are moving targets. Any lineage result is indexed to specific model versions at a specific date and does not transfer forward.

Prompt quality is not lineage-neutral. A prompt tuned on one family may underperform on another for reasons unrelated to blind-spot correlation. Prompts must be developed on the training split and frozen, and per-family prompt sensitivity reported.

Surviving mutants may be unrepresentative. They are defects tests miss, which is the class of interest, but not necessarily the class agents introduce.

Detection is not the operational outcome. What matters is escaped defects, not caught ones. This design measures the former only by proxy.

#9. What a positive result would not establish

That cross-family review is sufficient. Two families can share a blind spot inherited from shared training data or shared architecture; independence is a matter of degree and is not established by a single contrast.

That the mechanism is understood. A measured difference in conditional detection is an observation about behaviour, not an account of why the behaviours differ.

That the finding transfers to other defect classes, other languages, or other substrates.

#10. Numbering note

This paper was first drafted as 200.005. That identifier is reserved in the series roadmap for Mirror: A Memory Substrate with Cryptographic Provenance and Fractal Identity Binding, and has been reassigned here to 200.007, unclaimed in both the roadmap and the published set.

A related divergence is recorded rather than silently corrected: the roadmap lists 200.004 as Adversarial-Parallel Gating: Why Code Correctness and Code Gameability Are Orthogonal Verification Axes, while the published 200.004 is Durable Substrates for Long-Running Multi-Agent Work. The roadmap and the published series have drifted. The present paper is closest in subject to the planned-but-unwritten adversarial-gating title, and may properly be its experimental half; that assignment belongs to the series owner, not to this author.

#11. Relation to prior work in this series

mumega-200.001 reports that adversarial-parallel gating in production yielded post-approval P0 closures that sequential review would not have surfaced. That is an argument for parallel review and for adversarial prompting. It does not isolate lineage, and the present design is in part an attempt to determine whether the effect reported there is prompt-borne — in which case its cost structure is far more favourable than currently assumed.

#12. Status

This paper preregisters a design. No results are reported. The motivating observation is disclosed in §1 with its confound stated, and is not offered as evidence for the hypothesis.

The experiment has not been run.