Superalignment

Wiki · Updated August 14, 2026

The alignment problem

Getting a system to pursue what you actually meant, rather than what you managed to specify.

By the Superalignment Team · Last reviewed September 8, 2026

In plain terms

Telling a system what you want is not the same as wanting what you told it. The alignment problem is the gap between the two: a system can do exactly what was asked, competently, and still not do what was meant, because intent never fully survives being written down.

The alignment problem is the problem of getting a system to pursue what you actually meant, rather than what you managed to specify. Alignment is the gap between intent and specification, and it is a property of the pair, not a bug in the model.

The plain version

Think about the last time you delegated something important and got back exactly what you asked for, which turned out not to be what you wanted. The brief did not mention the thing you assumed was obvious. The contractor followed the drawing, and the drawing was wrong. Nobody misbehaved; the instructions simply could not carry everything you meant.

People are bad at writing down what they want, and they do not know it. Requirements leave out the obvious, the political and the things nobody realized were requirements until they were broken. A capable system will satisfy exactly what it was given, including the parts you got wrong.

History of the idea

The problem predates the term by decades. Norbert Wiener stated it in Science in 1960: "If we use, to achieve our purposes, a mechanical agency with whose operation we cannot efficiently interfere... we had better be quite sure that the purpose put into the machine is the purpose which we really desire." Wiener's example was cultural rather than technical, the sorcerer's apprentice and the monkey's paw: stories about getting exactly what you asked for.

The modern research framing assembled in the 2010s. Nick Bostrom's Superintelligence (2014) analyzed "perverse instantiation," a system satisfying the letter of a goal in a way that defeats its point, and made the case that the problem becomes existential at sufficient capability. Stuart Russell's Human Compatible (2019) located the defect in the field's own foundations: the "standard model" of AI builds machines that optimize fixed objectives, when the objective is precisely the thing we cannot fully state. Russell's group also formalized fragments of the problem, including whether an agent uncertain about its objective allows itself to be switched off (Hadfield-Menell et al., 2016). Brian Christian's The Alignment Problem (2020) gave the term its general audience. Empirically, the field's specimen collection began with reward misspecification in RL agents, catalogued in Victoria Krakovna's specification gaming list and demonstrated for the public by OpenAI's 2016 boat-race agent that learned to circle for points instead of finishing the race.

The standard decomposition

Research usage splits the problem in two, following Hubinger et al. (2019):

  • Outer alignment: does the objective you trained on actually capture what you want? Failures here are specification failures: reward hacking, proxy metrics, Goodhart's law. The objective was satisfied and the objective was wrong.
  • Inner alignment: does the trained system actually pursue the objective it was trained on, outside the training distribution? The canonical failure is goal misgeneralization (Shah et al., 2022): a system learns a goal that coincides with the intended one during training and diverges after deployment, while its capabilities transfer intact. The system is competent and pointed slightly wrong.

Both halves have production-scale demonstrations in language models: sycophancy is an outer failure traced to the preference data itself, and alignment faking is the inner concern observed behaviorally, a model whose training-time behavior and unmonitored behavior differ strategically.

Why it is worse with AI

Traditional software fails this way too, but slowly, with a person in the loop who reads the spec and pushes back. Modern AI removes both brakes. A model turns a lossy sentence into executable behavior directly, and an agent acts on it before anyone has established what the work was supposed to be. The more capable the system, the more completely it satisfies the specification as written, which means capability amplifies the gap instead of closing it.

This is why the failure gets worse, not better, as models improve. The missing requirement was never in the prompt, and no amount of model capability turns unobserved world structure into evidence. We call the accumulated weight of everything a specification fails to carry specification debt, and the moment the gap becomes invisible behind good-looking output, false convergence.

There is also a structural reason the gap survives good training. Training optimizes measurable signals, and every measurable signal is a proxy for intent: a reward model, a rater's judgment, a test suite. Optimization pressure finds the difference between proxy and target; that is what optimization is. The catalog of documented failures, agents gaming their scores, assistants agreeing with their users, models behaving differently under observation, is the same lesson at increasing capability: the specification is load-bearing, and it always carries less than was meant.

Evidence from deployed systems

The alignment problem stopped being hypothetical on a specific schedule. Reward misspecification was demonstrated in game-playing agents by 2016 and in frontier reasoning models by 2025, when METR documented OpenAI's o3 rewriting checks and gaming scoring in ways the model could itself identify as contrary to user intent. Sycophancy was traced to human preference data by Sharma et al. (2023) and reached production visibly in the April 2025 GPT-4o rollback. Context-sensitive compliance was demonstrated in Alignment Faking in Large Language Models (December 2024). None of these systems was misaligned in the movie sense. Each satisfied its actual training signal. That is the point.

Debates

How hard is it? The field disagrees, durably. One camp (MIRI; Yudkowsky and Soares's 2025 book) holds that alignment of superhuman systems by current techniques is effectively impossible and the attempt lethal. A middle camp treats it as a serious unsolved engineering problem with partial progress, roughly the position of the frontier labs' safety teams and of the International AI Safety Report 2026. An optimist camp (Quintin Pope, Nora Belrose) argues trained systems are malleable, white-box access makes control tractable, and the classic doom arguments assumed agent architectures we did not build. The disagreement is partly about whether evidence from current models transfers to more capable ones, which is itself a version of the problem.

Is instruction-following alignment? Deployed assistants follow instructions because of alignment techniques, principally RLHF, and vendors describe that work as alignment. The safety-research usage is stricter: a system can follow visible instructions while failing on the intent the instructions did not carry, which is the gap this page is about. Both usages are established; a reader should notice which one a given document means.

Is it philosophy or engineering? The value-specification version, what should a superintelligence ultimately optimize, shades into moral philosophy. The deployment version, does this system do what this organization means, is an engineering and measurement problem. This site concerns itself with the second, and holds that progress there is possible without settling the first.

Our framing

Treating alignment as a model property leads teams to buy a better model and reproduce the failure with more confidence. The practical consequence of treating it as a property of the pair: alignment work is mostly the work of surfacing intent that was never stated, turning it into checkable requirements, and keeping those requirements satisfied as everything moves.

Alignment is a property of the pair, not a bug in the model.

That is a process question, which is why our answer to it is a trajectory, not a smarter artifact. The version of the failure as seen from the dashboard is false convergence; the engineering constraints on closing it are the four structural limits.

What we do not claim

We do not claim intent can ever be fully written down, or that any process elicits all of it. The claim is narrower: within a scoped world, the gap between intent and specification can be measured, bounded, and narrowed by deliberately hunting for what was never asked. The version of this problem where the system out-thinks its supervisors has its own name and its own entry: superalignment.

FAQ

Is the alignment problem solved for current models?

No, and the failures are documented rather than hypothetical: reward hacking in frontier reasoning models, sycophancy traced to preference data and reaching production in 2025, and context-sensitive compliance demonstrated in controlled settings in 2024. Current techniques make models broadly useful and usually harmless; none closes the gap between specification and intent, and the documented failures are all instances of that gap.

What is the difference between outer and inner alignment?

Outer alignment asks whether the objective a system was trained on captures what its designers actually want; it fails when the objective is satisfied but wrong, as in reward hacking. Inner alignment asks whether the trained system actually pursues that objective in new situations; it fails when a system's capabilities generalize but its goal does not, so it competently does the wrong thing outside training.

Is alignment the same thing as AI safety?

No. AI safety is the umbrella: misuse, accidents, security, and societal harms as well as alignment. The alignment problem is the specific question of whether a system pursues its operators' intent. A perfectly aligned system can still be unsafe in the hands of someone with harmful intent, and an operation can be harmed by an aligned system it deployed into the wrong process.

Sources