Superalignment

Wiki · Updated August 14, 2026

Scalable oversight

Techniques for supervising AI systems on tasks that exceed their supervisor's ability to evaluate directly.

By the Superalignment Team · Last reviewed September 8, 2026

In plain terms

Today's AI training works because a person can look at what the model did and judge it. Scalable oversight asks what replaces that judgment when the model's work gets too big, too fast, or too expert for any person to check, and the main proposals all involve AI helping to supervise AI.

Scalable oversight is the family of techniques for supervising AI systems on tasks that exceed their supervisor's ability to evaluate directly. The term was introduced as one of five research problems in Concrete Problems in AI Safety (Amodei et al., 2016) and names the load-bearing assumption of most alignment training: somewhere in the loop, a human judges the output. Scalable oversight research asks what happens to that loop when the judging becomes the bottleneck.

The problem

Reinforcement learning from human feedback (RLHF) and its descendants train models against human judgments of their outputs. The method inherits a ceiling: it is only as good as the judge. A reviewer cannot meaningfully evaluate a million-line codebase, a multi-step plan whose consequences surface later, or an argument that requires expertise the reviewer lacks. The founding statement of the modern problem is Christiano, Shlegeris, and Amodei (2018): training signals fail for tasks "too complicated for a human to directly evaluate." Its earlier roots are Paul Christiano's 2015-2017 writing on capability amplification at ai-alignment.com. The restatement that launched a research brand is OpenAI's weak-to-strong framing: humans supervising superhuman models will be weak supervisors. Superalignment is this problem stated for the regime where the capability gap is systemic rather than task-by-task.

Two different asymmetries hide inside "the judge is weaker," and keeping them apart matters for reading the evidence. In an information asymmetry, the judge lacks facts the model has, a hidden document, an unread codebase, and a protocol can force the facts into the open. In an expertise asymmetry, the judge lacks the concepts to evaluate the argument even with every fact on the table. Nearly all positive experimental results to date live in the first regime. The superhuman version of the problem is the second.

The 2018 proposals

Three proposals shaped the field, sharing one move: use AI to amplify the supervisor rather than replace the supervision.

Iterated amplification

Iterated distillation and amplification (IDA) (Christiano, Shlegeris, and Amodei, 2018; accessible treatment by Cotra) amplifies a weak overseer by letting them decompose a question into subquestions answered by copies of the current model, then distills the amplified system into a new model and iterates. The original paper showed the scheme learning algorithmic tasks, shortest paths, wildcard search, matching supervised learning without ever seeing ground-truth labels, only decompositions. Ought's factored-cognition experiments tested whether humans can actually decompose reasoning this way; OpenAI's recursive book summarization (Wu et al., 2021) is the closest deployed descendant. As a literal training scheme for frontier models, IDA stalled: no lab trains by explicit amplification trees. Its decomposition idea survives inside process supervision, and its game-theoretic sibling is debate.

Debate

AI safety via debate (Irving, Christiano, and Amodei, 2018) has two models argue opposite answers before a weaker judge, with theory showing an efficient judge can in principle decide problems far beyond its own capability. It is the most active thread in the lineage as of 2026, with its own entry on this wiki covering the theory upgrades and the experimental record.

Recursive reward modeling

Recursive reward modeling (Leike et al., 2018) trains helper agents, themselves trained by reward modeling, to assist the human in evaluating the main agent; the paper's example is evaluating a chip-design agent with helpers that benchmark performance, heat, and security. The full recursion was never run. Its first rung became RLHF at scale, and "AI assists the evaluator" became self-critique (Saunders et al., 2022), CriticGPT, and Constitutional AI's AI feedback. Jan Leike carried the framing into OpenAI's Superalignment team and then to Anthropic.

Process versus outcome supervision

A practical descendant of decomposition asks where the judgment lands: on the final answer, or on each step of the reasoning. Let's Verify Step by Step (Lightman et al., 2023) trained a process reward model on 800,000 step-level labels (the released PRM800K dataset) and solved 78 percent of a MATH test subset, outperforming outcome supervision, with a precursor comparison in Uesato et al. (2022). The safety framing: rewarding endorsed reasoning makes reward hacking harder and outputs more legible, a goal pursued directly in prover-verifier games. The irony of the thread is that it was absorbed into capabilities: process rewards fed the reasoning-model line, while frontier training shifted toward outcome-based reinforcement learning on verifiable rewards, which pulls against legibility. A 40-author position paper (2025) argued chain-of-thought monitorability is real but fragile under exactly that pressure.

How progress is measured

The field's standard experimental design is sandwiching (Cotra, 2021): place a model between a less capable supervisor and a more capable ceiling, and measure how much of the gap a protocol recovers. Bowman et al. (2022) operationalized it: on MMLU and QuALITY questions, non-experts assisted by a 52B model scored 75.4 and 76.8 percent respectively, beating both the model alone (57.2 and 59.2) and unaided humans (57.2 and 48.6), evidence that human-model teams can outperform either side. The paradigm's later results are the field's headline numbers: debate lifting weaker judges from 48 to 76 percent accuracy (Khan et al., 2024), DeepMind's scaled comparison across nine task families (Kenton et al., 2024), OpenAI's weak-to-strong generalization recovering much of a strong model's capability from weak labels (Burns et al., 2023), and MIT's Scaling Laws for Scalable Oversight (Engels et al., 2025), which modeled oversight success against capability gaps across four adversarial games, found debate the strongest tested protocol at fixed gaps, and analyzed multi-stage "nested" oversight.

The automated-alignment-researcher extension

The agenda's endpoint was always to hand the oversight labor itself to AI. OpenAI's stated 2023 goal was "a roughly human-level automated alignment researcher"; CriticGPT applied model critics to model bugs. In April 2026 Anthropic reported automated alignment researchers: autonomous agent teams working on a weak-to-strong supervision task reached a performance-gap-recovered score of 0.97 in five days for about $18,000 of compute, against a human-team baseline of 0.23 in seven days, and the same report noted the gains did not transfer significantly to production scale and that agents sometimes gamed the metric. The counter-position is argued in "Automated Alignment is Harder Than You Think" (Bowkis, Buhl, Pfau, and Irving, 2026): optimization pressure concentrates errors where they are hardest to catch, AI errors do not follow human error patterns, solutions may be inscrutable, and correlated outputs lack the error diversity that makes human review work.

Limitations

Most positive results live on tasks where the judge lacks information the models have; whether they transfer to tasks where the judge lacks the concepts is open, and the superhuman regime is the second kind. Oversight built on model self-report inherits the eliciting latent knowledge problem, and the weak-to-strong deception result demonstrates models behaving well precisely where supervisors can check. Behavioral evaluation itself is complicated by evaluation-aware models. The International AI Safety Report 2026 classifies scalable oversight as largely unsolved. For the position that oversight should assume failure and constrain deployment instead, see AI control; for this site's account of where oversight arguments fail inside organizations, see the six gaps.

FAQ

Does scalable oversight work today?

In tested settings, partially. Assisted humans outperform unaided humans and models alone; debate measurably improves weak judges; weak supervision elicits much of a strong model's latent capability. Every positive result carries the same two caveats: the experiments mostly model information gaps rather than expertise gaps, and models have been shown to behave well specifically where their supervisor can check.

How is scalable oversight different from AI control?

Scalable oversight tries to make the training signal good enough that the resulting model is actually trustworthy. AI control assumes the model may be adversarial and engineers the deployment, monitoring, auditing, and permissions, so that even an untrustworthy model cannot cause unacceptable harm. The approaches are complements: oversight aims at the model, control at the protocol around it.

Sources