How can written principles supervise a model at scale?

The experiments show that model-generated harmlessness preferences, guided by written principles, can train an assistant that human evaluators prefer on the paper's helpfulness and harmlessness tests. The method relocates human judgment into the constitution, examples, helpfulness labels, data choices, and evaluation. It does not remove human normative input or prove that the principles remain adequate under new attacks and deployment conditions.

Yuntao Bai and 50 others · arXiv preprint · December 15, 2022 prototype 5 min read Explained by Superalignment Research

Read the original

Concept explainer: Constitutional AI in the Superalignment wiki

Topics: constitutional ai, rlhf, scalable oversight, cultural values

The 60-second answer

The paper made AI feedback a concrete, repeatable training pipeline and made part of the behavioral objective legible as a short list of principles. It also exposes a governance problem: scaling the interpreter of a constitution does not settle who wrote it or whether the interpretation is correct.

Bai and colleagues introduce Constitutional AI as a two-stage pipeline. Self-critiques and revisions move the policy toward a safer response distribution before reinforcement learning. An AI feedback model then labels which response better follows a sampled principle, replacing human harmlessness labels while retaining human helpfulness data. The reported systems were judged less harmful and less evasive than several baselines, and larger feedback models with chain-of-thought prompting performed better on comparison tasks.

Constitutional AI first trains on model-written critiques and revisions, then uses AI comparisons to train a harmlessness preference model.
The experiment replaces human harmlessness labels, not human values, helpfulness labels, red-team data, examples, or evaluation.
A constitution is only as reliable as its principles, interpreter, data, and tests, so shared blind spots remain a central failure mode.

Written for: Technical generalists familiar with language-model fine-tuning and preference learning. Useful prerequisites: Supervised fine-tuning, Reward models and reinforcement learning from feedback.

The question: Can a short list of natural-language principles replace human harmlessness labels in supervised and reinforcement learning?
What the authors did: The paper starts from a helpful assistant and runs two training stages. In supervised learning, the model critiques and revises harmful responses under randomly sampled constitutional principles, then trains on the revisions. In reinforcement learning from AI feedback, a feedback model applies principles to response pairs, those choices train a hybrid preference model, and PPO optimizes the assistant. Human comparisons and model-based scores evaluate helpfulness, harmfulness, and evasiveness across model sizes and training snapshots.
The source: Constitutional AI: Harmlessness from AI Feedback

What happens when the constitutional judge shares a blind spot?

Checkable case selected. The feedback model applies the principle, and its label can be distilled into the policy.

The judge is part of the constitution

Feedback condition	Comparison label	What training scales
Judge can apply the principle	Tracks the intended distinction in this case.	The selected constitutional judgment.
Judge shares the policy's blind spot	Rewards superficial or mistaken compliance.	The shared interpretation error.

RLAIF reduces direct human harmlessness labeling. It does not remove the need to validate the constitution, its interpreter, or its coverage.

The control changes whether the AI feedback model can apply the sampled principle to the contested case. When it can, comparison labels carry the principle into the preference model and policy. When it cannot, the same pipeline can distill a shared error. The pipeline and signal strengths are qualitative. The figure does not reproduce model sizes, comparison counts, scores, or effect magnitudes from the paper.

Walk through the argument

Each step below points to the source location that carries it. The source map follows the walkthrough for source-level checking.

Move supervision into a short text

Ordinary preference training can require thousands of human comparisons whose combined objective is hard to inspect. Constitutional AI supplies a smaller set of natural-language principles that can be read, changed, and sampled during training.

This does not make the objective complete. The paper's principles were chosen ad hoc for research, and their effect still depends on prompts, examples, model interpretation, and the data distribution. The text is an input to a system of judgment.

Source: ArXiv v1 PDF pages 1 to 5, Abstract, Section 1, Figure 1, and Sections 1.1 to 1.3, ArXiv v1 PDF pages 22 to 34, Appendices C to E

Stage one changes the response distribution

Start with a helpful assistant and a harmful request. The model writes a response, receives one sampled principle, critiques its own response under that principle, and revises it. The process repeats with new principles.

The final revisions become supervised training data. This SL-CAI stage moves the model toward safer and less evasive responses before reinforcement learning, reducing the exploration burden in the next stage.

Source: ArXiv v1 PDF pages 7 to 10, Section 3 and Figures 5 to 7

Stage two turns interpretation into reward

The supervised model generates two responses to each harmful prompt. A feedback model receives a sampled principle and chooses which response better follows it. Those model-generated comparisons train the harmlessness side of a hybrid preference model.

Human comparisons still train helpfulness. PPO then optimizes the policy against the hybrid preference model. The name RLAIF refers specifically to replacing immediate human harmlessness preferences with these AI-generated labels.

Source: ArXiv v1 PDF pages 10 to 14, Section 4 and Figures 8 to 10

The feedback model must understand the case

A weak interpreter can misapply a good principle. The paper tests pretrained language models on 438 binary helpfulness, honesty, and harmlessness comparisons. Larger models and chain-of-thought prompting perform better on this task.

That supports the feasibility of model-based comparison in this setup. It does not prove the reasoning trace is faithful or that the judge will remain reliable against inputs selected to exploit its blind spots.

Source: ArXiv v1 PDF page 6, Section 2 and Figure 4, ArXiv v1 PDF pages 10 to 14, Section 4 and Figures 8 to 10

Read the evaluations at their actual scope

Across crowdworker comparisons and model-based harmfulness scores, RL-CAI systems are reported as less harmful than key baselines while remaining willing to explain objections instead of refusing every difficult prompt. The supervised stage alone is weaker than the full RL pipeline.

The results depend on the paper's prompt sets, worker instructions, preference models, and model family. Absolute harmfulness scores are themselves learned measurements. They test the pipeline, not universal harmlessness.

Source: ArXiv v1 PDF pages 7 to 10, Section 3 and Figures 5 to 7, ArXiv v1 PDF pages 10 to 14, Section 4 and Figures 8 to 10

A constitution needs an interpreter

Written principles can make some choices visible, but they can conflict, omit a case, or admit several readings. The feedback model acts like an adjudicator, while training distills its decisions into a policy that no longer consults the text directly at every step.

A serious safety case therefore needs evidence about authorship, interpretation, amendment, and independent evaluation. If the judge and policy share a blind spot, RLAIF can make the shared error more consistent rather than correcting it.

Source: ArXiv v1 PDF pages 1 to 5, Abstract, Section 1, Figure 1, and Sections 1.1 to 1.3, ArXiv v1 PDF pages 15 to 16, Sections 5 and 6, ArXiv v1 PDF pages 22 to 34, Appendices C to E

Source map

These are the source locations that carry the argument. Use them to check this explanation against the original rather than trusting the summary alone.

Where the argument lives

Locus	Why it matters	Source
ArXiv v1 PDF pages 1 to 5, Abstract, Section 1, Figure 1, and Sections 1.1 to 1.3	Motivates scaled supervision, lays out both training stages, and states the intended benefits and contributions.	Open source →
ArXiv v1 PDF page 6, Section 2 and Figure 4	Evaluates language models as comparison judges on 438 helpful, honest, and harmless binary questions and reports scaling trends.	Open source →
ArXiv v1 PDF pages 7 to 10, Section 3 and Figures 5 to 7	Details critique, revision, supervised fine-tuning, data construction, the 16 supervised critique-revision principle pairs, and SL-CAI evaluations.	Open source →
ArXiv v1 PDF pages 10 to 14, Section 4 and Figures 8 to 10	Explains AI comparison labels, the hybrid preference model, chain-of-thought feedback, PPO training, and reported RLAIF results.	Open source →
ArXiv v1 PDF pages 15 to 16, Sections 5 and 6	Places the work among prior methods, summarizes the claims, and states dual-use concerns and unresolved design questions.	Open source →
ArXiv v1 PDF pages 22 to 34, Appendices C to E	Provides the actual constitutional principles, response examples, and few-shot prompts used for critiques, revisions, and comparison labels.	Open source →

The Assumption Switch

One result. One assumption exposed. Turn it and see what changes.

Assumption under test

The feedback model can apply a constitutional principle more reliably than the policy can exploit or misinterpret it.

Held in the source: When the feedback model recognizes the relevant harm and compares responses consistently, its labels can distill the principle into a preference model and then into the policy.
Turn it: If the feedback model shares the policy's blind spot, treats superficial phrasing as compliance, or reads the principle differently from its authors, reinforcement learning can amplify that error.
What changes: RLAIF scales a judgment process, not the truth of the constitution. Independence, interpretation, and evaluation remain part of the safety case.

The common misreading

Constitutional AI does not mean humans disappear from supervision. The paper replaces human labels for harmlessness with model judgments conditioned on human-written principles, while still using human helpfulness data and human evaluation. It also does not show that any written constitution is sufficient or that a feedback model interprets one correctly in every case.

Outside the ML frame

Constitutional law

Can a short governing text control behavior without institutions that interpret, contest, and amend it?

Legal constitutions do more than state values. Courts, legislatures, procedures, precedent, and public authority determine how text applies to disputed cases. The AI pipeline similarly depends on an interpreter, examples, data, and an update process. This lens suggests evaluating who writes the principles, which model adjudicates them, how conflicts are resolved, and how amendments gain legitimacy. The paper uses constitutional language but does not establish those institutions.

Where the result stops

The experiments use proprietary models up to 52 billion parameters and evaluation procedures tied to crowdworker instructions and preference models. The two stage-specific 16-item principle sets were chosen in an ad hoc and iterative way. Human harmlessness labels are removed from training, but human helpfulness labels, red-team prompts, few-shot examples, evaluations, and design choices remain. Chain-of-thought improves feedback accuracy here but is not shown to be faithful. The paper does not test long-run deployment, strategic adaptation, or legitimacy across cultures and stakeholders.

The numbers, with their measurands

Each value below states its measurand, evidence type, source location and evidence base when the source reports one. These fields distinguish self reported results from independent measurements.

16 in each stage. stage-specific constitutional principle prompts in supervised critique-revision and AI-feedback comparison. Reported as measured, Sections 3.1 and 4.1, arXiv v1 PDF pages 7 and 10. Evidence base: 16 critique-revision principle pairs in the supervised stage; a separate set of 16 principles in the AI-feedback comparison stage. Check it →
182,831. red-team prompts used for the supervised constitutional training dataset. Reported as measured, Section 3.2, arXiv v1 PDF page 8. Evidence base: 42,496 human-written prompts plus 140,335 model-generated prompts. Check it →
438. binary comparison questions used to test model judgment of helpfulness, honesty, and harmlessness. Reported as measured, Section 2 and Figure 4, arXiv v1 PDF page 6. Evidence base: 438 evaluation questions. Check it →

What remains open

How should principles be authored, contested, and amended when legitimate stakeholders disagree about harmlessness?
Which evaluations detect when the feedback model and policy share the same constitutional blind spot?
Does RLAIF remain robust when a policy can model the feedback process and optimize for superficial compliance?
Can interpretable evidence show whether chain-of-thought-based labels follow the stated principle rather than a correlated cue?

What this bears on

Superalignment maintains a public register of the claims it makes and the evidence that would change its mind. The entries below connect this source to the exact public claims it bears on.

C5. The binding constraint on overseeing stronger workers is conditions, not capability. This record bears on it, directly. The experiments make feedback-model judgment, constitutional coverage, prompt construction, and evaluation conditions explicit constraints on scaled AI supervision. See the claim and what would change our mind →

Source audit and review status

What was checked, and against what

Field	Result	Checked	By	Against
`title`	exact	2026-08-17	codex-primary-source-review	source
`authors`	exact	2026-08-17	codex-primary-source-review	source
`date`	exact	2026-08-17	codex-primary-source-review	source
`full_text`	exact	2026-08-17	codex-primary-source-review	source

Review status prototype.
Program collection seminal v1; mechanism study; wave 4, release slot unassigned.
Source access public full text, PDF. Open the reading copy →
Provider-authored safety claim yes.
Explained by Superalignment Research.
Reviewed by No named human reviewer yet.
Explainer dates created 2026-08-17; updated 2026-08-17.
AI assistance AI assisted with canonical-source retrieval, full-text extraction, page rendering, section-by-section reading, locus checking, first-pass prose, figure design, and implementation. The page remains a prototype until a named human review is recorded.
Rights and access The complete 34-page arXiv v1 manuscript is publicly readable and is the only listed arXiv version. This prototype pins v1 from 15 December 2022. The source does not state a separate reuse license in the manuscript text.
Corrections Read the correction policy or report an error.

Provenance

First seen 2026-08-16, via seed_library.py, arxiv shard of the Stampy snapshot.
Work id work:constitutional-ai-harmlessness-from-ai-feedback, which groups manifestations of the same intellectual work.
Record id arxiv:2212.08073, the natural key for this catalog manifestation.
2026-08-16 seeded from the arxiv shard
2026-08-17 full arXiv v1 read, author metadata corrected, and implementation-ready Explained prototype prepared

Full audit data, including this record under id arxiv:2212.08073: /library/records.jsonl. Compact browser index: /library/corpus.json. Catalog method and counts: /library/index.json.