When can two AIs help a weaker judge find the truth?

Debate is best understood as an oversight mechanism with a load-bearing judge assumption. The formal result shows that adversarial decomposition can make a weak verifier computationally powerful. It does not show that human judges reward truth, that self-play finds the right equilibrium, or that natural-language debate stays safe near equilibrium.

Geoffrey Irving and 2 others · arXiv preprint arXiv:1805.00899 · May 2, 2018 prototype 5 min read Explained by Superalignment Research

Read the original

Concept explainer: AI safety via debate in the Superalignment wiki

Topics: scalable oversight, debate, game theory, mechanism design, theory

The 60-second answer

Debate made scalable oversight concrete as an incentive-design problem. It also supplies a disciplined question for later work: which local claims can a bounded judge actually verify after adversarial decomposition?

Irving, Christiano, and Amodei propose training two agents by self-play to argue opposite sides before a weaker judge. Each debate follows one adversarially chosen path through a much larger argument tree. In a formal analogy, a polynomial-time judge plus polynomially many alternating claims can decide PSPACE problems under optimal play. The practical case depends on a harder empirical claim: a lie must be easier for an equally capable opponent to expose than for the judge to accept. A sparse-pixel MNIST game supplies an early proof of concept, while most of the paper maps the conditions under which human debate could fail.

Debate asks two matched agents to expose one decisive path through a much larger argument tree for a weaker judge.
The PSPACE theorem applies to an ideal algorithmic judge, while practical success depends on human truth recognition and stable self-play.
The MNIST result is a useful mechanism test with truthful pixel reveals, not a demonstration of natural-language debate.

Written for: Technical generalists who understand basic machine learning but have not studied scalable oversight. Useful prerequisites: Basic reinforcement learning and self-play, The idea of a computational verifier.

The question: Can two strong agents expose a decisive line of argument that a weaker judge can evaluate, even when the judge cannot solve the original task?
What the authors did: The paper defines a two-player zero-sum debate protocol, gives a complexity-theory analogy in which polynomial-length debate reaches PSPACE, analyzes assumptions and failure modes, and reports one toy MNIST experiment with a fixed sparse classifier as judge. Natural-language debate with human judges remains a proposal in this work.
The source: AI Safety via Debate

Can the judge settle the point the debaters expose?

Checkable mode selected. The judge can resolve the local claim exposed by the adversaries.

The paper's judge switch

Terminal claim	What the opponent contributes	What follows
Judge can check it	Selects the strongest flaw and exposes admissible evidence.	Debate can compress a large argument tree into one decision path.
Judge cannot check it	Identifies a disagreement but cannot supply a usable verdict.	Winning may track persuasion or a shared blind spot instead of truth.

The control changes the paper's load-bearing judge assumption. When a local claim is checkable, adversarial play can narrow a large argument tree to one decisive branch. When it remains opaque, the same game can reward persuasion instead of truth. Branch count, path length, box size, and color do not encode computational complexity, empirical success rates, or probability of truthful play.

Walk through the argument

Each step below points to the source location that carries it. The source map follows the walkthrough for source-level checking.

Start with the oversight bottleneck

Suppose a model can solve a task that you cannot solve or even grade. Asking for one answer gives you no reliable error signal. Debate adds a second model with the same capability and an incentive to find the most damaging flaw in its opponent's answer.

The judge still does important work, but on a narrower object. Instead of solving the original problem, the judge chooses which side made the more useful and truthful case along the line of dispute the agents exposed.

Source: PDF pages 1 to 2, Section 1, PDF pages 3 to 4, Section 2 and Figure 1

Why one path can represent a tree

A good chess move depends on a huge tree of replies, but two strong players can demonstrate one adversarial path while a weaker observer checks the rules and final score. The paper applies the same idea to arguments: each side selects the reply that most threatens the other side's case.

This compression works only if changing the subject counts as conceding the disputed branch. Otherwise a debater could keep opening new branches and force the judge to absorb the whole tree after all.

Source: PDF pages 3 to 4, Section 2 and Figure 1

What the PSPACE result says

Replace the human with a polynomial-time program that checks the final statement. One existential claim gives the familiar shape of an NP witness. Alternating existential and universal moves climbs the polynomial hierarchy. A polynomial number of moves reaches PSPACE.

That theorem establishes expressive power under optimal play and error-free checking. It does not establish learnability, human judgment, or safety near equilibrium. The authors repeatedly label the complexity result an analogy for the practical proposal.

Source: PDF pages 5 to 6, Section 2.2, Table 1, and Theorem 1

What the MNIST game tests

Both agents see an MNIST digit. A fixed classifier judge sees only four or six nonzero pixels revealed during play. One agent must defend the true label and the other an incorrect label. Revealed pixels are guaranteed to be genuine, so disagreement is about which evidence to show.

With six random pixels the judge is 59.4 percent accurate. With debate and a precommitted lie, the honest side wins 88.9 percent of test images. Four pixels move from 48.2 to 85.2 percent. Precommitment matters because it stops the liar from changing its claim across branches.

Source: PDF pages 6 to 7, Section 2.3, PDF pages 7 to 10, Section 3.1, Figures 2 to 4, and Table 2

Flip the judge assumption

In the toy game, the terminal evidence is a real pixel and the judge has a defined classifier score. Natural language has no equivalent guarantee. A false local claim can contain an unfamiliar concept, exploit a prior belief, or remain too subtle for the judge even after the opponent points to it.

This is the paper's load-bearing switch. If the judge can settle the exposed local issue, opposition can amplify oversight. If the judge cannot, the game may optimize for persuasive victory. More capable debaters can then become better at finding the judge's blind spots.

Source: PDF pages 11 to 12, Section 3.2 and Figure 5, PDF pages 12 to 18, Sections 4 and 5

Keep the proposal boundary visible

The paper offers several repairs, including matched agents, simultaneous moves, opponent pools, judge ensembles, smooth scores, and activation sharing. Each changes a particular failure mode, but none is evaluated as a complete natural-language system here.

The conclusion is unusually direct: human judgment, richer games, natural language, and interaction with other safety methods all need further work. Debate does not address distribution shift, safe exploration, or adversarial robustness by itself.

Source: PDF pages 12 to 18, Sections 4 and 5, PDF pages 18 to 22, Sections 6 to 8

Source map

These are the source locations that carry the argument. Use them to check this explanation against the original rather than trusting the summary alone.

Where the argument lives

Locus	Why it matters	Source
PDF pages 1 to 2, Section 1	Introduces scalable oversight, the hierarchy from demonstration to judging to recursive criticism, and the proposal status of natural-language debate.	Open source →
PDF pages 3 to 4, Section 2 and Figure 1	Defines the two-agent zero-sum game and states the central empirical claim that lying is harder than refuting a lie.	Open source →
PDF pages 5 to 6, Section 2.2, Table 1, and Theorem 1	Derives the polynomial-hierarchy progression and the PSPACE result for an idealized polynomial-time judge.	Open source →
PDF pages 6 to 7, Section 2.3	Explains large inputs and outputs, truthful revelation, learned judge models, environment interaction, and long-term state.	Open source →
PDF pages 7 to 10, Section 3.1, Figures 2 to 4, and Table 2	Reports the sparse-pixel MNIST setup, the precommitment rule, tree-search procedure, accuracy results, and experimental limitations.	Open source →
PDF pages 11 to 12, Section 3.2 and Figure 5	Describes informal cat-versus-dog play and leaves controlled human experiments, including moral judgment, to future work.	Open source →
PDF pages 12 to 18, Sections 4 and 5	Analyzes ignorance, self-play, belief bias, judge comprehension, near-equilibrium behavior, training stability, security, and stochastic simulation.	Open source →
PDF pages 18 to 22, Sections 6 to 8	Covers symmetry fixes, activation sharing, payoff variants, the relation to amplification, and the explicit limits of the proposal.	Open source →

The Assumption Switch

One result. One assumption exposed. Turn it and see what changes.

Assumption under test

A weaker judge can reliably settle the local point that optimal debaters expose.

Held in the source: If the judge can check the decisive local claim and an equally capable opponent can expose a lie, adversarial play can compress a large argument tree into one useful path.
Turn it: If the local point remains opaque, exploits a shared misconception, or persuades without being checkable, the opponent may identify a dispute that the judge still cannot resolve.
What changes: The same zero-sum incentive can then select for judge-winning rhetoric rather than truth. More debate steps do not repair a verifier that cannot recognize the terminal claim.

The common misreading

DEBATE equals PSPACE is a theorem about an idealized game with an algorithmic judge, not evidence that ordinary people can judge superhuman natural-language arguments. The MNIST result shows an advantage for honesty in one constrained game. It does not establish that honesty is the equilibrium of realistic debate.

Outside the ML frame

Adversarial adjudication

When does opposition improve a decision, and when does it merely optimize for the decision maker's blind spots?

Courts, audits, and red teams also rely on an opponent to surface hidden defects for a bounded reviewer. Their design problem is not competition alone. It includes balanced resources, admissible evidence, burden of proof, reviewer competence, and appeal. This lens makes the paper's judge instructions and symmetry assumptions look like institutional design choices. This is our interpretation, not a result tested by the paper.

Where the result stops

The natural-language system the paper ultimately targets is not tested. The only completed ML experiment uses MNIST, a fixed sparse classifier, truthful pixel reveals, prespecified honest and lying roles, and near-optimal tree search rather than learned debaters. The PSPACE result assumes error-free polynomial-time judging and unbounded agent computation. The authors also identify belief bias, opaque concepts, unstable training, first-mover effects, stochastic simulation, security, and persuasion as open problems.

The numbers, with their measurands

Each value below states its measurand, evidence type, source location and evidence base when the source reports one. These fields distinguish self reported results from independent measurements.

59.4% to 88.9%. six-pixel sparse judge accuracy before debate and honest win rate with precommitment. Reported as measured, Section 3.1, Table 2 and results paragraph, PDF pages 8 to 10. Evidence base: MNIST test set. Check it →
48.2% to 85.2%. four-pixel sparse judge accuracy before debate and honest win rate with precommitment. Reported as measured, Section 3.1, Table 2 and results paragraph, PDF pages 8 to 10. Evidence base: MNIST test set. Check it →

What remains open

Which task families let human judges reliably verify the final local claim without understanding the whole argument?
How can an evaluator distinguish honest admission of ignorance from a strategically convenient refusal to commit?
What training methods keep agents able to expose lies after honest play becomes common?
How should debate be tested when both debaters share the same model, data, and blind spots?

What this bears on

Superalignment maintains a public register of the claims it makes and the evidence that would change its mind. The entries below connect this source to the exact public claims it bears on.

C5. The binding constraint on overseeing stronger workers is conditions, not capability. This record bears on it, suggestively. The proposal shifts supervision from solving a hard task to judging a contested local claim, then shows that judge competence, agent symmetry, and training stability remain binding conditions. See the claim and what would change our mind →

Source audit and review status

What was checked, and against what

Field	Result	Checked	By	Against
`title`	exact	2026-08-17	codex-primary-source-review	source
`authors`	exact	2026-08-17	codex-primary-source-review	source
`date`	exact	2026-08-17	codex-primary-source-review	source
`venue`	exact	2026-08-17	codex-primary-source-review	source
`full_text`	exact	2026-08-17	codex-primary-source-review	source

Review status prototype.
Program collection seminal v1; theory; wave 3, release slot unassigned.
Source access public full text, PDF. Open the reading copy →
Provider-authored safety claim yes.
Explained by Superalignment Research.
Reviewed by No named human reviewer yet.
Explainer dates created 2026-08-17; updated 2026-08-17.
AI assistance AI assisted with canonical-source retrieval, full-text extraction, page rendering, section-by-section reading, locus checking, first-pass prose, figure design, and implementation. The page remains a prototype until a named human review is recorded.
Rights and access The complete arXiv v2 manuscript is publicly accessible from the canonical record. This prototype read and pinned that 24-page version rather than relying on the abstract or later summaries.
Corrections Read the correction policy or report an error.

Provenance

First seen 2026-08-16, via seed_library.py, arxiv shard of the Stampy snapshot.
Work id work:ai-safety-via-debate, which groups manifestations of the same intellectual work.
Record id arxiv:1805.00899, the natural key for this catalog manifestation.
2026-08-16 seeded from the arxiv shard
2026-08-17 full arXiv v2 read and implementation-ready Explained prototype prepared

Full audit data, including this record under id arxiv:1805.00899: /library/records.jsonl. Compact browser index: /library/corpus.json. Catalog method and counts: /library/index.json.