Can a stronger model learn past a weak supervisor's mistakes?

This paper establishes an experimental apparatus, not a solution. Positive performance gap recovered is common in its model-to-model proxy, but the amount recovered changes with the task, supervisor size, training objective and the learnability of the supervisor's errors. The last variable is the most consequential: when the weak answer becomes trivial to copy, average recovery collapses.

Collin Burns and 11 others · Proceedings of the 41st International Conference on Machine Learning, PMLR 235:4971-5012 · 2024 prototype 5 min read Explained by Superalignment Research

Read the original

Concept explainer: Weak-to-strong generalization in the Superalignment wiki

Topics: weak to strong, scalable oversight, rlhf, capability elicitation, measurement, organizational psychology

The 60-second answer

Scalable oversight depends on strong systems learning the intended task instead of merely imitating a weak evaluator's characteristic mistakes.

Burns and colleagues report positive performance gap recovered in virtually all studied model-size pairs, but the size and direction of the effect depend on the task, supervisor gap and training method. On the 22 NLP tasks, an auxiliary confidence loss raised median performance gap recovered from about 25 percent to nearly 80 percent for the smallest-supervisor and largest-student pairing. Chess showed worse scaling as student size rose, and naive reward modeling usually recovered roughly 10 percent. In the paper's sharpest assumption test, appending the weak answer to the prompt made the supervisor's error easy to copy and collapsed average recovery to 0.2 percent with naive training and 4.6 percent with the confidence loss.

The experiment uses a weaker model as a stand-in supervisor, not weak humans supervising a superhuman agent.
Positive recovery appears in most studied pairings, but its size changes sharply across tasks, model gaps and training objectives.
Performance gap recovered is a normalized task score, not a percentage of intelligence or alignment recovered.
When the weak answer is made trivial to copy, average recovery nearly disappears even with the paper's auxiliary loss.

Written for: Technical generalists who can follow an ML experiment without specialist training. Useful prerequisites: Supervised finetuning, A held-out test set, The difference between labels and ground truth.

The question: When a stronger pretrained model is trained on a weaker model's labels, does it learn the intended task or the weaker model's mistakes?
What the authors did: The authors finetuned a small model on ground truth, used its predictions as weak labels, and finetuned a larger proprietary GPT-4-family base model on those labels. A same-size model trained on ground truth supplied the comparison reference. They measured performance gap recovered across 22 binary NLP datasets, chess puzzles and a proprietary reward-modeling task.
The source: Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision

Can the student copy the supervisor's mistake?

Original proxy selected. The paper rounds average PGR to about 20 and 80 percent.

Values reported in published Figure 26

Setup	Naive	Confidence loss
Original NLP proxy, rounded in caption	about 20%	about 80%
Weak answer appended to prompt	0.2% (0.002 ± 0.003)	4.6% (0.046 ± 0.108)

The paper's Appendix G.3 turns one assumption directly. On the original NLP proxy it rounds average performance gap recovered to roughly 20 percent with naive training and 80 percent with the confidence loss. When the weak answer is appended to every prompt and becomes trivial to copy, the reported values fall to 0.2 percent and 4.6 percent. The switch changes error imitability, not supervisor accuracy. Reported values from published Figure 26. The original-task values are rounded in the figure caption. The easy-copy values are reported point estimates; the caption prints plus-minus terms but does not define whether they are standard deviations, standard errors or another quantity.

Walk through the argument

Each step below points to the source location that carries it. The source map follows the walkthrough for source-level checking.

First, build a proxy where the answer is known

The real superalignment problem withholds ground truth: a weak evaluator cannot reliably judge work beyond its competence. The paper creates a measurable proxy. A small model is finetuned on ground truth, its predictions become weak labels, and a larger model is finetuned on those labels. A same-size model trained on ground truth supplies the upper comparison.

That design isolates a useful question, but it also removes humans, strategic behavior and genuinely unknowable answers. The apparatus measures whether a stronger pretrained model can recover task performance from noisy model labels.

Source: Section 2, experimental setup, PMLR pages 4972 to 4974, Section 6 and Section 6.1, discussion and remaining disanalogies, PMLR pages 4978 to 4979

Read performance gap recovered as a ratio

Performance gap recovered compares three scores: the weak supervisor, the strong student trained on weak labels, and the strong model trained on ground truth. Zero means the student stays at weak-supervisor performance. One hundred percent means it closes the entire task-specific gap to the strong ground-truth reference.

The ratio is convenient because raw accuracy differs across tasks. It does not measure a fraction of the student's total capability, and repeated access to ground truth makes it easier to study than the deployment problem it represents.

Source: Section 2, experimental setup, PMLR pages 4972 to 4974

The headline result is variation, not one scaling law

Across the 22 binary NLP tasks, positive recovery is common. For the largest student and smallest supervisor, the auxiliary confidence loss raises median recovery from about 25 percent to nearly 80 percent. That is the paper's strongest aggregate result.

Chess gets worse as the student grows in some settings, while naive reward modeling usually recovers only about 10 percent and almost never more than 20 percent. A method that helps one proxy is not yet a general oversight method.

Source: Sections 3.1 and 3.3.1, Figure 4, PMLR pages 4974 to 4976, Figures 2 and 3 and Section 3.2, PMLR pages 4973 to 4975

Then make the supervisor's error easy to imitate

Appendix G.3 adds the weak answer to every input prompt. Supervisor accuracy has not changed, but its particular mistake is now cheap for the student to identify and copy. This turns error imitability while leaving the headline weak-to-strong setup recognizable.

Average recovery falls to 0.2 percent with naive finetuning and 4.6 percent with the confidence loss. The intervention suggests that the structure of weak errors matters at least as much as the supervisor's score.

Source: Appendix G.3, Figure 26, PMLR page 5010

Carry the proxy forward without promoting it to a solution

The experiment establishes a benchmark and a failure mode. It does not show that a weak human can control a strategic superhuman system. The models were pretrained on human material, most NLP tasks are balanced binary classification, and the reward-modeling study stops before downstream reinforcement learning.

The useful next question is operational: which evaluator errors can a capable system model and cheaply imitate? Without an answer, a positive average recovery score can hide the exact case that scalable oversight is meant to survive.

Source: Section 6 and Section 6.1, discussion and remaining disanalogies, PMLR pages 4978 to 4979, Appendix G.3, Figure 26, PMLR page 5010

Source map

These are the source locations that carry the argument. Use them to check this explanation against the original rather than trusting the summary alone.

Where the argument lives

Locus	Why it matters	Source
Section 2, experimental setup, PMLR pages 4972 to 4974	Defines the weak supervisor, strong student, ground-truth comparison and performance gap recovered metric.	Open source →
Sections 3.1 and 3.3.1, Figure 4, PMLR pages 4974 to 4976	Reports the aggregate binary NLP results and the effect of the auxiliary confidence loss across model-size pairs.	Open source →
Figures 2 and 3 and Section 3.2, PMLR pages 4973 to 4975	Shows that chess and reward modeling do not follow one uniform scaling pattern.	Open source →
Appendix G.3, Figure 26, PMLR page 5010	Turns error imitability directly by appending the weak answer to each prompt and reports the resulting collapse.	Open source →
Section 6 and Section 6.1, discussion and remaining disanalogies, PMLR pages 4978 to 4979	States why the model-to-model proxy, task mix and lack of strategic behavior limit the alignment claim.	Open source →

The Assumption Switch

One result. One assumption exposed. Turn it and see what changes.

Assumption under test

The strong student cannot cheaply identify and copy the weak supervisor's characteristic errors.

Held in the source: In the original NLP proxy, the weak answer is used as a training label but is not placed inside every input prompt.
Turn it: The authors append the weak answer to every prompt, making imitation of the supervisor's mistakes trivial.
What changes: Average performance gap recovered falls from roughly 20 and 80 percent to 0.2 and 4.6 percent for naive training and the confidence loss. The result depends on error imitability, which may be worse when a capable system can model its human evaluator.

The common misreading

The experiment does not show that the public GPT-2 model supervised GPT-4, or that weak humans can control a superhuman agent. The labels describe performance levels inside a proprietary GPT-4-family model series. Performance gap recovered is a task-specific normalized score, not the percentage of a model's intelligence that was aligned.

Outside the ML frame

Organizational behavior

What happens when a system is rewarded for A while its designers hope for B?

The training objective rewards agreement with the weak label while the designers hope the student recovers the true concept behind it. The confidence loss partly changes that incentive. This resembles Steven Kerr's organizational warning about rewarding one behavior while hoping for another. It also exposes what the experiment omits: an agent that can model, persuade or game the evaluator. This is our interpretation, not a result Burns and colleagues tested.

Where the result stops

The proxy removes direct human supervision and gives researchers ground-truth labels that would be unavailable for genuinely superhuman work. The strong models were pretrained on human-generated material, which may make target capabilities easier to elicit. Most NLP tasks are balanced binary classifiers, no model has a strategic incentive to persuade or deceive its evaluator, and the reward-modeling study does not include downstream reinforcement learning. No method works consistently across settings. The frontier models and reward data are proprietary, while the archived public code says it is not well tested and does not reproduce the paper's exact settings.

The numbers, with their measurands

Each value below states its measurand, evidence type, source location and evidence base when the source reports one. These fields distinguish self reported results from independent measurements.

22. balanced binary NLP datasets used for the main aggregate results. Reported as self reported, Section 3.1, PMLR page 4974. Evidence base: 22 datasets. Check it →
about 25% to nearly 80%. median performance gap recovered after adding the auxiliary confidence loss for the smallest-supervisor and largest-student NLP pairing. Reported as self reported, Section 3.3.1 and Figure 4, PMLR page 4975. Evidence base: median across 22 NLP datasets. Check it →
0.002 ± 0.003 and 0.046 ± 0.108. average performance gap recovered with naive training and the auxiliary confidence loss when the weak answer is appended to the prompt; the caption does not define the plus-minus statistic. Reported as self reported, Appendix G.3, Figure 26, PMLR page 5010. Evidence base: nine NLP task identifiers shown. Check it →
usually about 10%, almost never above 20%. naive performance gap recovered on the proprietary ChatGPT reward-modeling task. Reported as self reported, Section 3.2 and Figure 3, PMLR pages 4974-4975. Evidence base: model-size pairings shown in Figure 3. Check it →

What remains open

Which properties of an error make it easy for a stronger model to imitate?
Can recovery be estimated without ground-truth labels or repeated access to the final test?
Do the results survive generative tasks, downstream optimization pressure and strategic behavior?
Can an independent group reproduce the task-level results with open models and data?

What this bears on

Superalignment maintains a public register of the claims it makes and the evidence that would change its mind. The entries below connect this source to the exact public claims it bears on.

C5. The binding constraint on overseeing stronger workers is conditions, not capability. This record bears on it, indirectly. Holding the supervisor fixed while changing the training objective materially changes recovered task performance. This shows that supervision conditions matter, but it does not identify the method as the sole or binding constraint. See the claim and what would change our mind →

How it sits against other work

responds to Supervising strong learners by amplifying weak experts

Source audit and review status

What was checked, and against what

Field	Result	Checked	By	Against
`title`	exact	2026-08-17	codex-primary-source-review	source
`authors`	exact	2026-08-17	codex-primary-source-review	source
`date`	exact	2026-08-17	codex-primary-source-review	source
`venue`	exact	2026-08-17	codex-primary-source-review	source
`full_text`	exact	2026-08-17	codex-primary-source-review	source

Review status prototype.
Program collection seminal v1; synthetic stress test; wave 1, release slot unassigned.
Source access public full text, PDF. Open the reading copy →
Provider-authored safety claim yes.
Explained by Superalignment Research.
Reviewed by No named human reviewer yet.
Explainer dates created 2026-08-16; updated 2026-08-17.
AI assistance AI assisted with source discovery, full-text extraction, claim checking, first-pass prose and implementation. The page remains a prototype until a named human review is recorded.
Rights and access The published PMLR paper and the arXiv manuscript are openly available from their source pages.
Corrections Read the correction policy or report an error.

Provenance

First seen 2026-08-16, via hand-written example, cycle 1.
Work id work:weak-to-strong-generalization, which groups manifestations of the same intellectual work.
Record id arxiv:2312.09390, the natural key for this catalog manifestation.
2026-08-16 created as the annotated-tier worked example
2026-08-16 added the first schematic interactive figure
2026-08-17 full ICML paper re-read; metadata corrected; unsupported claims narrowed; source-valued Assumption Switch, outside lens, open questions and visible prototype status added

Full audit data, including this record under id arxiv:2312.09390: /library/records.jsonl. Compact browser index: /library/corpus.json. Catalog method and counts: /library/index.json.