Why did 1.3B InstructGPT beat 175B GPT-3?

InstructGPT established the modern language-model RLHF pipeline: demonstrations teach a starting policy, ranked outputs teach a reward model, and reinforcement learning optimizes that proxy. Its strongest result is that this post-training signal beat a far larger base model on the distribution it was built to serve. The same result also exposes the specification question, because better alignment to labeler judgments is only as broad as the people, prompts, rubric, and measurements behind those judgments.

Long Ouyang and 19 others · Advances in Neural Information Processing Systems 35 (NeurIPS 2022) · 2022 prototype 6 min read Explained by Superalignment Research

Read the original

Concept explainer: RLHF in the Superalignment wiki

Topics: rlhf, scalable oversight, evals, benchmarks, measurement, training data, robustness

The 60-second answer

This paper supplied the practical recipe behind a generation of instruction-tuned assistants and made the governance of human feedback a core technical question.

Ouyang and colleagues show that post-training can matter more than parameter count for instruction following. On held-out API Playground prompts, labelers preferred 175B InstructGPT to 175B GPT-3 85 percent of the time, and preferred the 1.3B PPO-ptx model to 175B GPT-3 despite the size gap. The models also improved on TruthfulQA and reduced some toxic outputs. The target was not a universal human value function. It was a reward model fitted to judgments from about 40 contractors, shaped by researcher instructions and a narrow customer prompt distribution.

InstructGPT turned demonstrations and ranked answers into a reward signal that could train a language model at scale.
On held-out API Playground prompts, post-training mattered more than a greater than 100-fold parameter increase.
PPO-ptx mixed pretraining updates into reinforcement learning to reduce capability regressions, but also preserved more base-model behavior.
The target came from a specific chain of researchers, contractors, prompts, and proxy evaluations, not from humanity as a whole.

Written for: Technical generalists who understand language models and basic supervised learning. Useful prerequisites: A language model predicts the next token, Supervised fine-tuning changes a model using example answers, A proxy score can differ from the outcome it is meant to represent.

The question: Can human feedback make a language model follow user intent better than a model with over 100 times as many parameters?
What the authors did: The authors start from GPT-3 models with 1.3B, 6B, or 175B parameters. Contractors first write demonstrations for supervised fine-tuning. They then rank several model answers to the same prompt, producing comparisons for a 6B reward model. Proximal policy optimization trains the supervised model against that learned reward, with a KL penalty and, for PPO-ptx, an added pretraining-data objective. The main evaluation uses held-out prompts from the same OpenAI API Playground source as the training data, plus public tests of truthfulness, toxicity, bias, and standard NLP capability.
The source: Training language models to follow instructions with human feedback

Where does each human judgment enter InstructGPT?

Supervised fine-tuning selected. Contractors write demonstrations for 12,725 training prompts.

Human signal and training data by stage

Stage	Human signal	Training target	Training prompts
SFT	Written demonstrations	Predict demonstration tokens	12,725: 11,295 labeler and 1,430 customer
Reward model	Ranked model outputs	Predict which answer a contractor prefers	33,207: 6,623 labeler and 26,584 customer
PPO	Reward-model score learned from rankings	Increase predicted reward with a KL penalty to SFT	31,144 customer
PPO-ptx	Reward-model score plus pretraining text	PPO objective plus pretraining next-token loss	Same 31,144 PPO prompts; Supplemental Table 3 does not count pretraining samples

Counts are from Supplemental Table 3. Validation prompts and pretraining samples are not included in these training-prompt totals.

Select a stage to see the source of its human signal, the training target, and the reported number of training prompts. The fallback table preserves every stage without interaction.

Walk through the argument

Each step below points to the source location that carries it. The source map follows the walkthrough for source-level checking.

Separate model size from following intent

GPT-3 could continue text fluently without reliably doing what a user asked. More parameters improved language modeling, but did not specify which answer a user would consider helpful, honest, or harmless. The paper treats instruction following as a post-training problem.

Its headline comparison makes that distinction concrete. Human evaluators preferred the 1.3B PPO-ptx model to 175B GPT-3 on the sampled prompt distribution, even though both used the GPT-3 architecture and the base model had over 100 times as many parameters.

Source: PDF pages 1 to 4, Abstract, Section 1, and Figures 1 to 2, PDF pages 10 to 12, Section 4.1 and Figures 3 to 5

Follow the signal through three stages

First, contractors write desired answers and supervised fine-tuning teaches the model to imitate them. Second, contractors rank several answers to the same prompt. A reward model learns to predict those rankings. Third, PPO changes the policy to produce answers that the reward model scores highly.

The final stage also penalizes movement away from the supervised model. PPO-ptx adds gradients from the original pretraining objective. That mixture reduced regressions on public NLP benchmarks, showing that retention was an explicit part of the recipe rather than an automatic result of preference training.

Source: PDF pages 6 to 9, Sections 3.1 to 3.5 and Equations 1 to 2, Combined arXiv manuscript PDF pages 40 to 43, Appendix C

Trace who supplied each kind of data

The SFT training split used 11,295 labeler-written prompts and 1,430 customer prompts. Reward-model training used 6,623 labeler prompts and 26,584 customer prompts. PPO used 31,144 customer prompts. Customer prompts came from the API Playground research preview and were split by user to reduce leakage.

These are not independent votes from a general public. Researchers chose about 40 contractors and wrote their instructions. Customers selected their own prompts. The pipeline therefore combines three sources of judgment with different roles and no built-in rule for resolving their conflicts.

Source: NeurIPS main-paper PDF pages 6 to 8, Sections 3.2 to 3.4, NeurIPS supplemental PDF page 18, Appendix A.3 and Table 3, Combined arXiv manuscript PDF pages 17 to 19, Sections 5.1 to 5.3

Read the preference result on its own distribution

On held-out API prompts, 175B InstructGPT outputs were preferred to 175B GPT-3 outputs 85 plus or minus 3 percent of the time. Against a prompted GPT-3 baseline, the win rate was 71 plus or minus 4 percent. Labelers also preferred 1.3B PPO-ptx to 175B GPT-3.

The held-out prompts came from the same broad source as training prompts, and most comparisons had one judgment. The result is strong evidence that the pipeline improved contractor-rated instruction following on that distribution. It is not a distribution-free measure of user intent.

Source: NeurIPS main-paper PDF pages 9 to 10, Section 3.6, Combined arXiv manuscript PDF pages 34 to 40, Appendix B, PDF pages 10 to 12, Section 4.1 and Figures 3 to 5

Keep each safety proxy attached to its scope

InstructGPT was about twice as likely as GPT-3 to produce answers judged truthful and informative on TruthfulQA. In a closed-domain task, 175B PPO hallucinated 21 percent of the time versus 41 percent for GPT-3. Toxicity fell when the prompt explicitly asked for a respectful answer.

Those tests did not add up to a general safety score. The paper found no clear improvement on two bias datasets, and models still followed harmful requests, fabricated facts, and failed instructions. Each benchmark probes one observable slice of a larger claim.

Source: PDF pages 12 to 15, Section 4.2 and Figures 6 to 7, Combined arXiv manuscript PDF pages 17 to 19, Sections 5.1 to 5.3

Ask who the model is aligned to

Section 5 makes the paper unusually explicit about the target. Researchers influence the rubric and data process. Contractors supply demonstrations and rankings. Customers supply prompts and some direct preferences. People affected by outputs may have no place in the loop.

That boundary is not a side issue. A reward model can predict the chosen raters perfectly while excluding another group's interests or effects that no rater sees. InstructGPT showed how to optimize a feedback channel. It did not settle how the channel should be constituted.

Source: Combined arXiv manuscript PDF pages 17 to 19, Sections 5.1 to 5.3

Source map

These are the source locations that carry the argument. Use them to check this explanation against the original rather than trusting the summary alone.

Where the argument lives

Locus	Why it matters	Source
PDF pages 1 to 4, Abstract, Section 1, and Figures 1 to 2	States the instruction-following problem, main preference results, remaining failures, and three-stage training overview.	Open source →
PDF pages 6 to 9, Sections 3.1 to 3.5 and Equations 1 to 2	Defines supervised fine-tuning, ranked-output reward modeling, PPO, the KL penalty, and the pretraining mix used in PPO-ptx.	Open source →
NeurIPS main-paper PDF pages 6 to 8, Sections 3.2 to 3.4	Documents the labeler and API Playground prompt sources, user-level splits, and filtering.	Open source →
NeurIPS supplemental PDF page 18, Appendix A.3 and Table 3	Reports the exact SFT, reward-model, and PPO training and validation prompt counts by source.	Open source →
NeurIPS main-paper PDF pages 9 to 10, Section 3.6	Defines API-distribution preference judgments, public benchmark proxies, and metadata labels.	Open source →
Combined arXiv manuscript PDF pages 34 to 40, Appendix B	Documents evaluator selection and instructions, most comparisons receiving one judgment, and measured agreement limits.	Open source →
PDF pages 10 to 12, Section 4.1 and Figures 3 to 5	Reports preference wins over GPT-3, model-size comparisons, held-out labeler results, and FLAN and T0 comparisons.	Open source →
PDF pages 12 to 15, Section 4.2 and Figures 6 to 7	Reports TruthfulQA, closed-domain hallucination, toxicity, bias, and public NLP benchmark results with their proxy boundaries.	Open source →
Combined arXiv manuscript PDF pages 17 to 19, Sections 5.1 to 5.3	Explains who defined the target, why labelers are not the whole affected population, and which model failures remain.	Open source →
Combined arXiv manuscript PDF pages 40 to 43, Appendix C	Documents model sizes, reward-model choice, optimization settings, pretraining mix, and approximate compute costs.	Open source →

The Assumption Switch

One result. One assumption exposed. Turn it and see what changes.

Assumption under test

Contractor preferences on the sampled prompt distribution are a useful proxy for the intent and welfare of the people the model should serve.

Held in the source: Researchers select and instruct about 40 contractors, then evaluate held-out prompts from the same API Playground source used to build the training datasets.
Turn it: Change the reference group, the affected population, the language mix, or the kinds of consequences that raters can observe.
What changes: The learned reward can still be optimized successfully while ranking outputs against a different target. The pipeline does not decide whose preferences should govern or which effects belong in the score.

The common misreading

The paper did not show that RLHF aligned GPT-3 with humanity or made it safe. It showed that one feedback pipeline improved outputs under contractor judgments and several proxy tests. The authors explicitly describe this as alignment to a specific group of labelers and researchers, and report harmful requests, falsehoods, bias, and instruction failures that remained.

Outside the ML frame

Political representation

Who is authorized to turn a diverse public into one training signal?

The paper's Section 5 asks who the model is aligned to and names researchers, labelers, and customers as distinct sources of influence. Political representation adds a sharper test: selection, mandate, affected parties, and appeal determine whether an aggregate judgment is legitimate, not only whether raters agree. This is our institutional interpretation of the paper's stated reference-group problem.

Where the result stops

The main prompts came from the API Playground research preview, not all API traffic or downstream effects. More than 96 percent were English. About 40 contractors were selected by researchers, and their judgments do not represent all users or people affected by model outputs. Most API-distribution comparisons had one label for cost reasons. Helpfulness received priority in training labels, while the final evaluation rubric prioritized truthfulness and harmlessness. The models still fabricated facts, followed harmful requests, showed bias, and failed simple instructions. Public benchmarks cover only narrow proxies, and PPO-ptx mixes pretraining behavior back into the policy.

The numbers, with their measurands

Each value below states its measurand, evidence type, source location and evidence base when the source reports one. These fields distinguish self reported results from independent measurements.

85 plus or minus 3%. preference for 175B InstructGPT over 175B GPT-3 on held-out API Playground prompts. Reported as self reported, Section 4.1, PDF pages 10 to 11. Evidence base: held-out API prompt evaluation with contractor comparisons. Check it →
1.3B versus 175B parameters. model-size comparison in which the smaller PPO-ptx model was preferred to GPT-3. Reported as self reported, Abstract, Section 1, and Figure 1, PDF pages 1 to 2. Evidence base: held-out API prompt evaluation. Check it →
21% versus 41%. closed-domain hallucination rate for 175B PPO and GPT-3 respectively. Reported as self reported, Section 4.2, PDF pages 12 to 13. Evidence base: closed-domain question-answering evaluation. Check it →
about 40. contractors used for demonstrations, comparisons, and evaluations. Reported as self reported, Sections 3.4 and 5.2, combined arXiv manuscript PDF pages 8 and 17. Evidence base: the paper's contractor pool. Check it →

What remains open

How should a feedback system represent durable disagreement instead of compressing it into one scalar reward?
Which affected people need standing when they are not the user who wrote the prompt?
How can reward models be tested after policy optimization moves behavior beyond the comparison data?
Which safety properties survive changes in language, culture, task, and deployment context?
Can truthfulness and harmlessness be measured without relying on proxies that are easy to satisfy superficially?
What governance process should control updates to the labeler rubric and reference group?

How it sits against other work

extends Deep Reinforcement Learning from Human Preferences

Source audit and review status

What was checked, and against what

Field	Result	Checked	By	Against
`title`	exact	2026-08-17	codex-primary-source-review	source
`authors`	minor variant	2026-08-17	codex-primary-source-review	source
`date`	exact	2026-08-17	codex-primary-source-review	source
`venue`	exact	2026-08-17	codex-primary-source-review	source
`full_text`	exact	2026-08-17	codex-primary-source-review	source

Review status prototype.
Program collection seminal v1; mechanism study; wave 4, release slot unassigned.
Source access public full text, PDF. Open the reading copy →
Provider-authored safety claim yes.
Explained by Superalignment Research.
Reviewed by No named human reviewer yet.
Explainer dates created 2026-08-17; updated 2026-08-17.
AI assistance AI assisted with primary-source retrieval, full-paper and appendix extraction, source-locus checking, first-pass prose, figure design, and implementation. The page remains a prototype until a named human review is recorded.
Rights and access The 15-page NeurIPS main paper, separate 47-page supplement, combined 68-page arXiv manuscript, and linked code repository are publicly accessible. Public access is not a claim about reuse rights beyond their stated terms.
Corrections Read the correction policy or report an error.

Provenance

First seen 2026-08-16, via seed_library.py, arxiv shard of the Stampy snapshot.
Work id work:training-language-models-to-follow-instructions-with-human-feedback, which groups manifestations of the same intellectual work.
Record id arxiv:2203.02155, the natural key for this catalog manifestation.
2026-08-16 seeded from the arXiv shard
2026-08-17 full NeurIPS main paper, supplement, and combined arXiv manuscript read; implementation-ready Explained prototype prepared

Full audit data, including this record under id arxiv:2203.02155: /library/records.jsonl. Compact browser index: /library/corpus.json. Catalog method and counts: /library/index.json.