How can a reward model learn a goal from human comparisons?

The paper made learned reward models practical enough for contemporary deep reinforcement learning and established the core feedback loop later associated with RLHF. It did not show that pairwise preferences recover human values. It showed that a small, task-specific comparison channel could sometimes replace a much denser programmatic reward in simulated control and games.

Paul F. Christiano and 5 others · Advances in Neural Information Processing Systems 30 (NIPS 2017) · 2017 prototype 5 min read Explained by Superalignment Research

Read the original

Concept explainer: RLHF in the Superalignment wiki

Topics: rlhf, scalable oversight, reward hacking, agents, measurement

The 60-second answer

The paper is a direct ancestor of modern RLHF. It separated human judgment from the reinforcement-learning loop through a reward model, making oversight cheaper while creating a new object that optimization can exploit.

Christiano and colleagues show that sparse pairwise comparisons can train a reward predictor that supplies a dense signal to deep reinforcement learning. With 700 real-human comparisons, a single run nearly matched true-reward training on the eight simulated robotics tasks. With 5,500 comparisons, Atari results were mixed but showed substantial learning on most games. The same loop learned a repeating backflip, one-legged running, and driving alongside traffic without a programmatic task reward. Ablations show that feedback timing, clip context, ensembles, regularization, and query selection can materially change performance.

A learned reward predictor turns sparse pairwise feedback into the dense signal needed by deep reinforcement learning.
The results cover task-specific Atari and simulated control, not the recovery of general human values.
Feedback must follow the changing policy because a fixed reward model can become a target for unintended optimization.

Written for: Technical generalists who understand neural networks and basic reinforcement learning. Useful prerequisites: Basic reinforcement learning, Supervised classification and cross-entropy, Why a proxy reward can differ from a desired outcome.

The question: Can a deep reinforcement learning agent learn complex goals from a small number of human comparisons instead of receiving the environment's reward function?
What the authors did: The authors ran an asynchronous loop with a policy, a learned reward predictor, and human comparison queries. Contractors compared one-to-two-second trajectory clips. An ensemble fit those pairwise choices with a Bradley-Terry-style model, while A2C or TRPO optimized the predicted reward. The paper tested eight MuJoCo tasks, seven Atari games, three novel behaviors, and six ablations, with benchmark rewards hidden from training but retained for evaluation where available.
The source: Deep Reinforcement Learning from Human Preferences

Does feedback move with the policy?

Online queries selected. New comparison labels can follow changes in policy behavior.

What the paper changes in the ablation

Condition	Human comparisons	What keeps changing	Paper's observation
Main online setup	Gathered throughout training	Policy and reward predictor	New behavior can enter later queries.
No online queries	Gathered only at the beginning	Policy continues to optimize the learned reward	The paper reports undesirable behavior, including very long Pong volleys that avoid losing without trying to score.

The paper's loop gathers comparisons while reinforcement learning changes the policy. The switch freezes queries at the start, matching the no-online-queries ablation. The paper reports undesirable reward-model behavior in that condition, but does not provide a single effect size for the mechanism. Arrow length, box size, color, and update spacing do not encode query frequency, training time, performance, or effect magnitude.

Walk through the argument

Each step below points to the source location that carries it. The source map follows the walkthrough for source-level checking.

Begin with two short clips

Imagine training a stunt performer by repeatedly showing a coach two one-second clips and asking which attempt is better. The coach never writes a complete score function. Their comparisons supply examples from which another system tries to infer what better means.

That is the paper's interface. Contractors see pairs of one-to-two-second trajectory segments and select the better clip, a tie, or cannot tell. The agent's task is to produce trajectories the human would prefer while asking as few questions as possible.

Source: Pages 1 to 2, Section 1, Pages 2 to 5, Section 2.2 and Equation 1, Supplement pages 13 to 16, Appendices A and B

Follow the three-process loop

The policy generates trajectories. A query process selects pairs of recent segments for a human. A reward predictor fits every comparison collected so far, then returns predicted rewards to the policy optimizer. All three processes run asynchronously.

The architecture matters because human labels are sparse but simulated interaction is cheap. The reward model is the compression layer: it generalizes a few judgments across many steps, while A2C on Atari or TRPO on MuJoCo optimizes the resulting dense signal.

Source: Pages 2 to 5, Section 2.2 and Equation 1

See what the choice model assumes

Equation 1 treats each clip's score as the sum of predicted per-step rewards and uses a Bradley-Terry choice rule to turn two sums into a preference probability. Cross-entropy then trains the reward predictor against human choices.

The implementation adds an ensemble, validation-based regularization, dropout in some domains, and a ten percent random-response term. Queries favor segment pairs on which ensemble members disagree, although the paper says this crude uncertainty heuristic can hurt on some tasks.

Source: Pages 2 to 5, Section 2.2 and Equation 1

Separate the evidence from the headline

On eight MuJoCo tasks, 700 real-human comparisons nearly matched reinforcement learning with the true reward, but each real-human curve is one run. On seven Atari games, 5,500 comparisons produced substantial learning on most games, with a mix of matches, shortfalls, and one clear failure on Qbert.

For benchmark tasks, the true reward was hidden from training but retained for evaluation. That makes the experiment auditable, but it is not the setting in which learned reward is most needed. For a genuinely novel goal, there is no programmatic metric to reveal whether the learned proxy is correct.

Source: Pages 5 to 6, Section 3.1.1 and Figure 1, Pages 6 to 7, Section 3.1.2 and Figure 2

Look at what no written reward specified

Using the same loop, the authors trained a Hopper to repeat backflips, a Half-Cheetah to move on one leg, and an Enduro driver to stay alongside other cars. The backflip used 900 queries in less than an hour. These examples show flexibility, not quantitative alignment to an external ground truth.

The feedback came from the authors, and the behaviors were evaluated qualitatively. The distinction is important: the standard tasks test sample efficiency against a known score, while the novel tasks test whether a rater can steer visible behavior toward a description.

Source: Page 7, Section 3.2

Keep feedback on the moving target

A policy optimized against a learned reward changes which states and behaviors the reward model must judge. In the main setup, new comparisons arrive throughout training. The no-online-queries ablation freezes feedback near the initial policy and lets optimization move away from the labeled distribution.

The paper reports bizarre offline behavior, including Pong agents that avoid losing without trying to score. This is the key assumption switch: a reward model is not merely a passive measurement. Once optimized, it becomes part of the environment and must be tested on the behavior it induces.

Source: Pages 7 to 9, Section 3.3 and Figures 3 to 4, Page 9, Section 4

Source map

These are the source locations that carry the argument. Use them to check this explanation against the original rather than trusting the summary alone.

Where the argument lives

Locus	Why it matters	Source
Pages 1 to 2, Section 1	Motivates learning a goal from feedback when a reward is hard to write and dense human evaluation is too expensive.	Open source →
Pages 2 to 5, Section 2.2 and Equation 1	Defines the asynchronous policy, query, and reward-model loop, the short-clip comparison protocol, the Bradley-Terry preference model, and ensemble query selection.	Open source →
Pages 5 to 6, Section 3.1.1 and Figure 1	Reports eight simulated robotics tasks, the 700-comparison condition, run counts, and comparison with true-reward training.	Open source →
Pages 6 to 7, Section 3.1.2 and Figure 2	Reports seven Atari tasks, the 5,500-comparison human condition, and the games where learned reward matched, lagged, or failed.	Open source →
Page 7, Section 3.2	Describes the repeating backflip, one-legged Half-Cheetah, and alongside-traffic behaviors learned without a programmatic task reward.	Open source →
Pages 7 to 9, Section 3.3 and Figures 3 to 4	Turns off online queries and other components, reports reward-model-induced behavior, and compares single frames with longer clips.	Open source →
Page 9, Section 4	States the claimed reduction in human interaction cost and the paper's narrower conclusion about scaling preference feedback to deep reinforcement learning.	Open source →
Supplement pages 13 to 16, Appendices A and B	Documents hidden supervision controls, model and label schedules, and the exact instructions that limit contractors to events visible inside each clip.	Open source →

The Assumption Switch

One result. One assumption exposed. Turn it and see what changes.

Assumption under test

A reward predictor trained on early behavior remains valid after the policy changes what behavior is available.

Held in the source: The main method gathers comparison queries throughout reinforcement learning, so the reward model receives labels from newer parts of the policy's trajectory distribution.
Turn it: The no-online-queries ablation gathers comparisons only at the beginning and then optimizes a fixed reward predictor as the policy moves elsewhere.
What changes: The paper reports undesirable behavior under offline reward training, including Pong policies that avoid losing without trying to score and produce extremely long volleys. Feedback timing is part of the method, not an implementation detail.

The common misreading

This is not evidence that generic human approval aligns a capable system or that the paper trained a conversational model. Contractors received specific task instructions and compared short clips from Atari and MuJoCo. Standard-task success was evaluated with a hidden programmatic reward that would be unavailable for an open-ended human goal, while the novel behaviors were judged qualitatively.

Outside the ML frame

Performance measurement

What can a supervisor evaluate when the measurement window is only a short clip?

The contractors were told to judge only events they actually witnessed. That instruction makes labels more consistent, but it also defines what cannot enter the learned reward: delayed damage, hidden state, and consequences outside the clip. In organizational measurement, a local score often becomes the operational goal precisely because broader outcomes are costly to observe. This is our interpretation, not a tested comparison in the paper.

Where the result stops

The environments are Atari and simulated robotics, not open-ended real deployment. Real-human curves are single runs, and authors supplied feedback for some tasks and all novel behaviors. Raters judged short clips and were instructed to consider only events they witnessed, which hides delayed or off-screen consequences. The preference model assumes additive per-step latent reward over each clip. The learned reward can be exploited as the policy changes, and the paper reports bizarre behavior under offline feedback. No agent models or strategically manipulates its evaluator.

The numbers, with their measurands

Each value below states its measurand, evidence type, source location and evidence base when the source reports one. These fields distinguish self reported results from independent measurements.

700. real-human pairwise comparison queries used for each simulated robotics task. Reported as self reported, Section 3.1.1 and Figure 1, pages 5 to 6. Evidence base: eight MuJoCo tasks; one real-human run per task. Check it →
5,500. real-human pairwise comparison queries used for each Atari task. Reported as self reported, Section 3.1.2 and Figure 2, pages 6 to 7. Evidence base: seven Atari games; one real-human run per game. Check it →
900 queries in less than one hour. feedback used to train the repeating Hopper backflip. Reported as self reported, Section 3.2, page 7. Evidence base: one author-feedback behavior run. Check it →
roughly 3 orders of magnitude. claimed reduction in human interaction complexity from using a separately learned reward model. Reported as self reported, Section 4, page 9. Evidence base: paper-level conclusion across the reported domains. Check it →

What remains open

How can a reward learner represent delayed, hidden, or disputed consequences that a short clip omits?
What query policy remains reliable when the optimized policy actively searches for reward-model errors?
Can independent replications recover the task-level results with modern open models and complete training artifacts?
How should preference disagreement be represented instead of compressed into one scalar reward?
What evaluation is possible when no hidden programmatic reward exists even for the test set?

What this bears on

Superalignment maintains a public register of the claims it makes and the evidence that would change its mind. The entries below connect this source to the exact public claims it bears on.

C5. The binding constraint on overseeing stronger workers is conditions, not capability. This record bears on it, indirectly. The experiments hold reinforcement-learning capability roughly fixed while changing feedback timing, representation, and reward-model training. These conditions materially affect performance, but the paper does not study supervision of a stronger-than-human worker. See the claim and what would change our mind →

How it sits against other work

responds to Concrete Problems in AI Safety

Source audit and review status

What was checked, and against what

Field	Result	Checked	By	Against
`title`	exact	2026-08-17	codex-primary-source-review	source
`authors`	minor variant	2026-08-17	codex-primary-source-review	source
`date`	exact	2026-08-17	codex-primary-source-review	source
`venue`	exact	2026-08-17	codex-primary-source-review	source
`full_text`	exact	2026-08-17	codex-primary-source-review	source
`supplement`	exact	2026-08-17	codex-primary-source-review	source

Review status prototype.
Program collection seminal v1; mechanism study; wave 2, release slot unassigned.
Source access public full text, PDF. Open the reading copy →
Provider-authored safety claim yes.
Explained by Superalignment Research.
Reviewed by No named human reviewer yet.
Explainer dates created 2026-08-17; updated 2026-08-17.
AI assistance AI assisted with primary-source retrieval, full-text and supplement extraction, section-by-section reading, locus checking, first-pass prose, figure design, and implementation. The page remains a prototype until a named human review is recorded.
Rights and access The full conference paper and its supplemental archive are publicly accessible from the NeurIPS proceedings page. Public access is not a claim about reuse rights.
Corrections Read the correction policy or report an error.

Provenance

First seen 2026-08-16, via seed_library.py, arxiv shard of the Stampy snapshot.
Work id work:deep-reinforcement-learning-from-human-preferences, which groups manifestations of the same intellectual work.
Record id arxiv:1706.03741, the natural key for this catalog manifestation.
2026-08-16 seeded from the arXiv shard
2026-08-17 full conference paper and supplement read; implementation-ready Explained prototype prepared

Full audit data, including this record under id arxiv:1706.03741: /library/records.jsonl. Compact browser index: /library/corpus.json. Catalog method and counts: /library/index.json.