Superalignment

Wiki · Updated September 8, 2026

Constitutional AI

A training method in which AI feedback guided by an explicit written constitution replaces human preference labels, the first production-scale use of AI labor for oversight.

By the Superalignment Team · Last reviewed September 8, 2026

In plain terms

Instead of thousands of people rating model outputs, Constitutional AI writes the standards down as an explicit list of principles and has an AI apply them, critiquing and revising outputs against the written constitution. It made the oversight standards inspectable and made AI itself do the oversight labor, which is the move the whole superalignment agenda depends on.

Constitutional AI (CAI) is a training method, introduced by Anthropic (Bai et al., December 2022), in which AI-generated feedback guided by an explicit written set of principles, the constitution, replaces human preference labels in alignment training. It is the founding production-scale instance of reinforcement learning from AI feedback (RLAIF) and, within the superalignment literature, the first deployed demonstration that AI labor can substitute for human oversight labor.

The problem it was built for

The human-feedback pipeline it modified, RLHF, had two costs the paper names. The practical one: harmlessness training consumed large volumes of human labeling, including labelers reading harmful content. The epistemic one: the standards being trained into the model lived nowhere except the aggregate of thousands of unrecorded individual judgments, so nobody could read, criticize, or amend "what the raters reward" as an object. The paper's title framing, "harmlessness from AI feedback," announced the substitution: the only human input to the harmlessness training would be the written principles themselves.

Mechanism

CAI has two phases, both consuming the constitution.

  1. Supervised phase (critique and revision). The model is prompted with adversarial inputs designed to elicit harmful outputs. For each draft response, the model is asked to critique its own output against a sampled constitutional principle and then revise it. The model is finetuned on the revised responses. The step is worth pausing on: the model's own capability is used to improve the training data it will learn from, supervised by a document rather than a person.
  2. Reinforcement phase (RLAIF). An AI evaluator compares pairs of responses and judges which better satisfies a sampled principle. The resulting preference dataset trains a preference model, and the policy is optimized against it: the RLHF pipeline with the human rater replaced by a principled AI rater.

The original paper reported the trained models were judged both more harmless and less evasive than RLHF-trained baselines: constitutional training pushed toward engaging with sensitive requests and explaining refusals, rather than the flat refusals human harmlessness labels had rewarded. The constitution itself is short, explicit, and public, drawing on sources including the UN Universal Declaration of Human Rights, other labs' published terms, and principles written in-house.

Why it matters for oversight

Two properties made CAI more than an efficiency win.

Inspectability. RLHF's standards live implicitly in thousands of unrecorded human judgments, while CAI's live in a document that can be read, criticized, and amended. Moving oversight standards from tacit to explicit is the same institutional move a safety case makes for deployment arguments: the standard becomes an artifact that can be disagreed with, which is a precondition for it being improved.

Oversight leverage. The AI applying the constitution does evaluative work that would otherwise consume human attention. This is the first rung of the ladder that recursive reward modeling proposed, that OpenAI's automated-alignment-researcher plan aimed at, and that Anthropic's later automated alignment researchers extended to research tasks: every rung substitutes AI labor for human oversight labor while trying to keep the standards human-owned.

Successors

The line continued in three directions. Collective Constitutional AI (2023) sourced a constitution from a representative public process, testing whether the standards document can be democratically produced rather than lab-authored. Constitutional Classifiers (2025) moved the constitution from training time to inference time, using constitution-guided classifiers to filter jailbreak attempts against deployed models. And the constitution itself became a governance surface: published, versioned, and debated in public, which is what "inspectable standards" turns out to mean in practice.

Limitations

CAI relocates rather than removes the oversight problem. The AI rater enforcing the constitution is itself a trained model, subject to reward hacking and interpretation drift, and the constitution's application inherits the model's reading of it, a concern that sharpens as the systems doing the enforcing approach the capability of the systems being trained. A written principle is still a specification, so the whole apparatus sits inside the alignment problem: the constitution carries less than its authors meant, and the enforcing model optimizes what the document says as read, not as intended.

Principle-based training also does not by itself address context-sensitive behavior of the kind demonstrated in alignment faking, a result found in a CAI-trained model, and in that experiment the values the model was strategically protecting were the constitutionally trained ones, which cuts both ways: evidence the training instilled durable values, and evidence that durable values complicate retraining. The method is best understood as moving oversight standards into the open and oversight labor onto AI, while leaving open the question every AI-supervises-AI scheme shares: who checks the checker.

FAQ

Does Constitutional AI remove humans from alignment training?

No. It removes humans from the labeling loop for harmlessness, replacing per-example human judgments with AI judgments guided by human-written principles. Humans still author the constitution, still supply helpfulness preference data in the original recipe, and still evaluate the resulting models. The substitution is of human labor at the scale point, thousands of ratings, not of human authority over the standard.

What is actually in the constitution?

A few dozen short written principles, published by Anthropic, that the AI rater samples from when critiquing and comparing outputs. The published sources include the UN Universal Declaration of Human Rights, platform terms of service, principles from other labs' research, and principles written in-house, phrased as instructions like choosing the response least likely to be harmful or dishonest. The document is short enough to read in one sitting, which is part of the design: an inspectable standard has to be a readable one.

Primary source in the Library

The Library's explained record of Bai et al. (2022), Constitutional AI and RLAIF, explained, covers the two training phases and what the paper measured.

Sources