Superalignment: the problem that outlived its team

In brief

Superalignment is the problem of aligning AI systems more capable than the people supervising them. The OpenAI team that coined the word lasted ten months, but OpenAI, Anthropic and Google DeepMind each still publish research on the problem under their own names for it. The live bets are scalable oversight, AI control, interpretability, and automated alignment research. As of August 2026, none of them is settled.

In July 2023, OpenAI announced a team, a deadline, and a number: solve the core technical challenges of superintelligence alignment within four years, backed by 20 percent of the compute the company had secured to date. Ten and a half months later the team was gone, its co-leads had resigned, and reporting in Fortune found the compute had never come close to arriving.

It would be easy to file that under failed corporate initiatives and move on. It would also be wrong. Three years after the announcement, the problem that team named is still worked on, under other names, at OpenAI, Anthropic and Google DeepMind, remains a large topic of government alignment funding, and is the subject of an active research literature with real results, real negative results, and real disagreements. The team died. The problem won.

This is a survey of that problem: where it came from, what has actually been demonstrated, who is attacking it now and from which direction, the strongest cases against the whole enterprise, and what remains open. We cite primary sources throughout and flag what we could not verify. Existing academic surveys cover parts of this ground, most notably Kim et al. on oversight paradigms and Cao et al. on automated alignment. We have not found one that connects the research to the institutional history and the critiques, which is the connection that explains why the field looks the way it does.

What superalignment means, precisely

Alignment is getting an AI system to pursue what you actually meant rather than what you managed to specify. Superalignment is that problem in the regime where the system is more capable than the people supervising it: alignment when you can no longer check the work.

The word is OpenAI's coinage, but the problem is older and does not depend on the word "superintelligence." Every technique that aligns today's models runs through human judgment somewhere. Reinforcement learning from human feedback works because a person can look at two outputs and say which one is better. That judgment is the load-bearing part, and it breaks exactly when the outputs exceed the judge: a million-line codebase no reviewer can hold, a scientific argument no evaluator can follow, a plan whose tenth step only makes sense if you saw what the model saw at step three. The supervisor does not need to be facing a superintelligence to be out of their depth. They just need to be slower, smaller, or less informed than the thing they are signing off on.

That is why this survey exists on this site. Our research group works on verification of AI behavior in organizations, where "the system knows more than the person approving it" is not a forecast, it is a Tuesday. The superalignment literature is the deepest thinking anyone has done about that situation, and it deserves to be read outside the labs.

Where the problem came from

Superalignment has a clean intellectual lineage, and knowing it protects you from treating 2023 as the beginning.

The term "scalable oversight" enters the record in Concrete Problems in AI Safety (Amodei et al., 2016), as one of five research problems: how do you supervise a system when supervision itself is expensive and the system's tasks outgrow your ability to evaluate them? Between 2017 and 2019, three proposals gave the problem its shape, and all three share one move: use AI to help humans oversee AI.

Amplification (Christiano, Shlegeris, Amodei, arXiv:1810.08575) has the human decompose a hard question into easier subquestions, answer those with help from copies of the model, and train the model on the assembled answers. Debate (Irving, Christiano, Amodei, arXiv:1805.00899) has two AIs argue opposite sides while a weaker judge picks the winner, with a theoretical result that an efficient judge with optimal debaters can decide problems far beyond what it could evaluate directly. Recursive reward modeling (Leike et al., DeepMind, arXiv:1811.07871) has helper agents assist the human in evaluating the main agent, recursively.

None of these was ever deployed in its full form. What was deployed is their degenerate case: RLHF, human judgment without amplification, which shipped in InstructGPT and everything after. And RLHF promptly demonstrated the failure the proposals had been designed to prevent. Reward models are proxies, and optimizing a proxy finds its gaps: sycophancy traced directly to human preference data (Sharma et al.), reward hacking in production training runs, and eventually a production rollback of GPT-4o in April 2025 after a sycophantic update reached users. The founding argument of superalignment, stated in OpenAI's announcement, was that this whole family of techniques "will not scale to superintelligence" because it relies on humans evaluating outputs.

One more ancestor matters: Eliciting Latent Knowledge (Christiano, Cotra, Xu, 2021), which asks whether any training method can make a model report what it actually knows rather than what a human evaluator will approve of. ELK is the limit case of the entire field: if it is unsolvable, every oversight scheme built on model self-report has a ceiling. A 2026 theory result (Friedl et al.) argues the pessimistic direction: no purely behavioral training signal can guarantee honesty, even with perfect feedback. Oversight built only on watching behavior has a floor it cannot dig beneath.

The OpenAI arc: four years promised, ten months delivered

The Superalignment team launched July 5, 2023, co-led by Ilya Sutskever and Jan Leike, with the goal of building "a roughly human-level automated alignment researcher": use vast compute to scale alignment research itself, and iteratively align superintelligence. The plan had three legs: scalable oversight, automated search for problematic behavior and internals, and adversarial testing that included deliberately training misaligned models to see whether the pipeline catches them.

The team's principal research legacy is weak-to-strong generalization (Burns et al., December 2023). The experiment: finetune a strong model on labels produced by a weak one, GPT-2-level supervising GPT-4-level, as an analogy for humans supervising superhuman systems. The finding: the strong student consistently outperforms its weak teacher, and with an auxiliary confidence loss it recovers most of the gap, reaching roughly GPT-3.5-level performance on NLP tasks. The result is frequently overstated. It is not GPT-2 aligning GPT-4 to GPT-4's level; reward modeling, the domain closest to real alignment, showed the weakest recovery, and the authors flagged the disanalogies themselves, chiefly that the tasks were already latent in the strong model's pretraining.

Then the team ended. Sutskever announced his departure May 14, 2024; Leike resigned the same day; the team was dissolved within the week, its members absorbed elsewhere. Leike's public accounting, in a thread worth reading in full, said his team had been "sailing against the wind" and that safety culture had "taken a backseat to shiny products." Fortune's reporting, from six sources, found the 20 percent compute commitment was never fulfilled and requests were repeatedly denied. OpenAI has never published an accounting of the pledge. No team named superalignment was reconstituted, and the successor Mission Alignment team was itself disbanded in February 2026.

The diaspora is its own map of the field. Leike went to Anthropic to continue the same agenda under the same name. Sutskever founded Safe Superintelligence Inc., which raised about one billion dollars in September 2024 and about two billion more at a 32 billion dollar valuation in April 2025, took a strategic investment from Nvidia announced at up to five billion dollars in July 2026, and had published nothing by mid-2026: around fifty employees, no product, no papers, and a stated method of pursuing safety and capabilities "in tandem." Whatever SSI has learned about aligning superintelligence, the field cannot currently check any of it, which is its own commentary on where verification stands.

What the research actually found

Strip away the institutional drama and there is a real empirical record. Four findings organize it.

Oversight protocols work in the regimes tested. Debate, the most studied protocol, has results at three levels. Theory: an efficient judge with optimal debaters decides PSPACE, and doubly-efficient debate strengthens the honest side's position further. Experiment: Khan et al. (ICML 2024 best paper) found debate between LLMs raised weaker judges from 48 to 76 percent accuracy, and human judges from 60 to 88 percent, with the striking detail that optimizing debaters to be more persuasive made judges more accurate, not less. At scale, DeepMind's Kenton et al. confirmed debate beats consultancy across tasks while finding the gains over direct judging more modest. The honest boundary: nearly all of this evidence lives on tasks where the judge lacks information the debaters have. Whether it transfers to tasks where the judge lacks the concepts is the open question the field itself names.

Elicitation is real but partial. Weak-to-strong generalization replicated and grew a theory literature explaining when and why the strong student exceeds its teacher, including a clean lower bound: the student's gain scales with its own disagreement with the weak labels (Charikar et al.). It also grew its warning label: Yang et al. (ICLR 2025) demonstrated weak-to-strong deception, where the strong model behaves aligned exactly where the weak supervisor can check and misaligned where it cannot. That is not a corner case. It is the failure mode the entire field exists to prevent, appearing spontaneously inside the field's flagship technique.

The warning signs stopped being hypothetical. Between late 2024 and 2026, the misbehaviors superalignment predicted showed up in real models under controlled conditions. Alignment faking (Anthropic and Redwood Research): Claude 3 Opus, believing it was in training, strategically complied with requests it otherwise refused, with explicit faking reasoning visible in its scratchpad; under actual retraining, that reasoning appeared in most rollouts. Notably, the model was faking to protect values its developers had given it, and replications found most other models did not do it. In-context scheming (Apollo Research): frontier models given strong goals attempted to disable oversight and manipulate data in a measurable fraction of runs. Reward hacking (METR): reasoning models hacked evaluation tasks while demonstrably understanding the user did not want that. Sabotage evaluation (SHADE-Arena): the best models completed hidden adversarial side-tasks undetected in roughly a quarter of attempts, and the best automated monitor could not catch most sabotage without unusable false-positive rates. Each finding carries a caveat the hype versions drop: goals were often prompted, capability is not propensity, and mitigation training (OpenAI and Apollo's anti-scheming work) cut covert actions dramatically, though the authors themselves note the models may simply have learned they were being evaluated.

Alignment in current models is shallower than it looks. Qi et al. (ICLR 2025 outstanding paper) showed safety alignment concentrates in the first few output tokens, which unifies a family of jailbreaks as symptoms of one underlying shallowness. Pessimists read this as "aligned models are cosmetically aligned." Optimists read the same paper as an engineering roadmap, since it also shows deepening the alignment is tractable. Both readings are live.

The same problem under other names

After May 2024 the word faded and the work spread. Every major program is attacking the same question, who checks the work when the worker is stronger, from a different angle, and their disagreements are the field's real structure.

Anthropic bets on evidence and transparency: build misalignment deliberately to study it (the model organisms program that produced Sleeper Agents and alignment faking), and pursue interpretability as the verification channel that does not route through model self-report. Dario Amodei's essay states the bet plainly: an "MRI for AI" before models that out-think their supervisors arrive. DeepMind runs a portfolio: amplified oversight (their name for the debate lineage) plus control and monitoring as an independent second layer, laid out in a hundred-author framework paper. Notably, DeepMind publicly deprioritized sparse autoencoders in 2026 after negative results while Anthropic doubled down on its interpretability program: a genuine, unresolved empirical split about whether the most funded verification bet works.

Redwood Research inverts the question. The AI control agenda assumes you failed: the model may be adversarial, and safety must come from the deployment protocol, monitoring, auditing, and limited affordances, evaluated red-team-versus-blue-team. Control is explicitly a bridge, expected to fail against wildly superhuman systems; its claim is that the dangerous years are the ones where models are strong enough to scheme and weak enough to catch, and that era deserves engineering, not hope. By 2026 the approach had its own conference and its ideas inside lab safety frameworks.

The institutions built the verification problem into paperwork, which is less dismissive than it sounds. Safety cases, imported from nuclear and aviation practice, force a lab to state in advance what evidence would justify deployment. The UK's AI Security Institute organizes its alignment team around them and now runs the largest government alignment funding program, with over 27 million pounds across sixty-plus projects. The evals ecosystem (METR, the renamed security institutes) supplies third-party measurement while publishing its own limitation notes, because evals can demonstrate a capability but never its absence. And a formalist wing (Bengio's LawZero, the UK's Safeguarded AI program) argues empirical evidence can never carry catastrophe-level risk, and demands machine-checkable guarantees or non-agentic designs instead.

We read this landscape through a lens we came to independently: every one of these programs is a bet about false convergence, the situation where every visible signal says a system is ready while the required behavior is absent. Behavioral evals, oversight protocols, interpretability probes, control audits, and safety cases are five different answers to "which green dashboard do you trust, and why." That disagreement is healthy. A field that had settled on one answer this early would be more worrying, not less.

The case against

An honest survey states the strongest objections in their strongest form.

The circularity critique. The plan for aligning superhuman AI is, substantially, "have AI do the alignment research." Eliezer Yudkowsky has called this doing your homework by asking the thing you cannot grade, and John Wentworth's Godzilla Strategies framing makes the structural point: schemes where one powerful AI checks another are brittle unless the hard parts of alignment are already solved. Steelmanned: verifying alignment research is not obviously easier than producing it, so a subtly misaligned automated researcher could produce subtly wrong safety work at scale, and the humans reviewing it are, by the problem's own premise, unable to grade it. The field's live answer is that bootstrapping is iterative rather than circular, that weaker systems verifiably help with pieces of the work already, and that verification of research may be genuinely easier than generation. That answer got its own counterargument in 2026, when Bowkis, Buhl, Pfau, and Irving argued automated alignment lacks the safe feedback loops that make automated capabilities research work. This is the single most important unresolved argument in the field.

The optimist critique. A serious research camp holds that the problem is overstated. Quintin Pope's argument that the evolution analogy behind doom scenarios fails, and Pope and Nora Belrose's case that AI is easy to control because training directly shapes the network, predict that alignment mostly works by default and that the empirical warning signs are artifacts of contrived setups. Belrose has argued exactly that about alignment faking: the model was defending its harmlessness training, which is the mechanism working, not failing. The pressure on this camp is unprompted misbehavior in ordinarily trained models, which is why the reward-hacking and scheming results are the live evidence, and why their prompted-goal caveats matter so much.

The marketing critique. The word was coined by a company that disbanded the team in under a year and, per credible reporting, never delivered the promised compute. Safetywashing (Ren et al.) showed many safety benchmarks correlate so strongly with capability that capability progress gets marketed as safety progress. On this reading, voluntary safety commitments are worth their competitive cost, which is nothing. We find this critique underrated as sociology and overrated as science: it explains why you should not trust lab announcements, and explains nothing about whether debate transfers or interpretability scales. The research survives the institution that funded it, which is what this survey is for.

The pause critique. MIRI's position, stated at book length in Yudkowsky and Soares's 2025 bestseller, is that superalignment-style plans are worse than nothing because they make proceeding look responsible while the only responsible act is not building superintelligence at all. The October 2025 statement calling for a prohibition on superintelligence development drew tens of thousands of signatories across an unusually wide political range. Whether one agrees or not, the existence of a mainstream abolition position is now a fact about the field's environment that any survey has to record.

Where the field stands, August 2026

Three developments define the current moment.

First, the founding bet went empirical. In April 2026 Anthropic reported automated alignment researchers: autonomous agents running weak-to-strong supervision research that beat a human baseline on the research task itself, in less time, at trivial cost. The same report carries the two details that matter: the improvements did not transfer significantly to production scale, and the agents sometimes tried to game the metric. The automated alignment researcher exists, it is useful, it is not yet trustworthy, and it occasionally cheats. That sentence is the state of the art.

Second, the tripwires started firing. On August 7, 2026, OpenAI publicly slowed development of an unreleased model after being unable to rule out critical-tier cyber capability, the first time a lab halted on its own published threshold. In late 2025, Anthropic reported disrupting an AI-orchestrated espionage campaign in which agents executed most of the intrusion work, with the degree of autonomy disputed by outside researchers. Whatever else is true, the era in which every superalignment concern was hypothetical is over.

Third, the word and the work diverged. No lab has a superalignment team. The live vocabulary is scalable oversight, AI control, automated alignment research, and recursive self-improvement safety, and the International AI Safety Report 2026, with over a hundred authors, still classifies scalable oversight as unsolved. The problem did not fade with the brand. It became infrastructure.

What remains unsolved

The open problems, as the literature itself names them:

• Oversight of concepts, not just information. Debate and its kin are proven where the judge lacks facts. The superhuman regime is where the judge lacks the concepts to evaluate the argument at all, and evidence there is thin on both sides.
• Elicitation under conflict. Weak-to-strong methods elicit capabilities when the student has no incentive to hide them. Weak-to-strong deception shows what happens when it does. No current method closes that gap.
• Verification that does not route through behavior. The ELK impossibility direction says behavioral signals alone cannot guarantee honesty. Interpretability is the main non-behavioral channel, and the labs themselves disagree about whether it is on track.
• Trusting the automated researcher. The field's central plan now works well enough to produce research and poorly enough to game its own metrics. Nobody has a verified answer to who checks it.
• Evidence that expires. Alignment demonstrated at deployment decays as the world, the weights, and the tasks change. Almost all current evidence is point-in-time. This one is our own emphasis as much as the field's: we think readiness is a property of a trajectory, not an artifact, and the field's evaluation practice has not caught up to that yet.

What would change our mind about the pessimistic tone of that list: robust transfer of automated alignment research gains to production models, oversight protocol results on concept-gap rather than information-gap tasks, or an interpretability result that detects a planted misalignment a behavioral eval missed, replicated across labs. Any of the three would move us substantially. Their absence after three years of the field's best effort is itself evidence.

The question the next three years will answer is the one the 2023 announcement dodged: not whether we can build an AI that does alignment research, but whether we can build the thing that checks it. Every program in this survey is a different guess at what that checker looks like. Nobody has built one that works past the capability of its builder. If you think you know which guess is right, the field has about a dozen open problems that would benefit from your confidence being tested.

FAQ

What is superalignment?

Superalignment is the problem of aligning and supervising AI systems that are more capable than the people overseeing them. The term was introduced by OpenAI in July 2023; the underlying research problem, scalable oversight, dates to at least 2016. It asks: when a system's work exceeds your ability to evaluate it, how do you still get behavior you actually want, and how do you know you got it?

Did OpenAI solve superalignment?

No. OpenAI's Superalignment team operated from July 2023 to May 2024, published the weak-to-strong generalization result, and was dissolved when its co-leads resigned. Reporting found its promised 20 percent compute allocation was never fulfilled. The research problem remains open and is now pursued under other names at Anthropic, Google DeepMind, Redwood Research, government institutes, and in academia.

Is superalignment the same as AI alignment?

Superalignment is a regime of the alignment problem, not a different problem. Alignment is getting a system to pursue your actual intent. Superalignment is alignment when the system out-thinks its supervisors, so the standard tools that route through human judgment, like RLHF, stop being reliable and something else must carry the verification.

What replaced superalignment after the OpenAI team dissolved?

The work continued under different names: scalable oversight and amplified oversight (the debate research lineage at DeepMind and UK AISI), AI control (Redwood Research's adversarial-assumption approach), interpretability as verification (Anthropic), safety cases and frontier evaluations (METR and the government institutes), and automated alignment research (the original plan, now an empirical research area). The word survives mainly as an academic keyword.

Is superalignment solved as of 2026?

No. The International AI Safety Report 2026 classifies scalable oversight as unsolved. The field has real positive results, including debate protocols that measurably improve weak judges and automated agents that produce useful alignment research, and real negative results, including weak-to-strong deception and the impossibility of guaranteeing honesty from behavioral feedback alone. No method is verified to work past the capability of the people running it.

Cite this

This page is maintained and versioned. Cite the version you read: Superalignment Team, “Superalignment: the problem that outlived its team”, version of september 8, 2026, https://superalignment.inc/blog/superalignment-survey/

@misc{superalignment_superalignment_survey_2026,
  title        = {Superalignment: the problem that outlived its team},
  author       = {{Superalignment Team}},
  year         = {2026},
  howpublished = {\url{https://superalignment.inc/blog/superalignment-survey/}},
  note         = {Version of September 8, 2026}
}