Why can an expert demonstration be a bad way to teach a robot?

CIRL's lasting move is to make assistance interactive. A human action can both change the world and change the robot's belief, so treating it as an ordinary expert demonstration can discard the information the human intended to send. The formal result is conditional on a shared reward, a human behavior model, and strong coordination assumptions.

Dylan Hadfield-Menell and 3 others · Advances in Neural Information Processing Systems 29 (NIPS 2016) · 2016 prototype 5 min read Explained by Superalignment Research

Read the original

Topics: game theory, mechanism design, agents, theory, scalable oversight

The 60-second answer

CIRL changed the unit of analysis from a robot inferring a fixed objective to a human and robot shaping each other's information. That move underlies assistance games and later work on uncertainty, deference, and corrigibility.

Hadfield-Menell and colleagues formalize value alignment as a cooperative partial-information game. Human and robot share the human's reward, but only the human observes its parameter. This makes human actions informative and gives both players incentives to teach and learn. The robot's posterior over the reward parameter is a sufficient statistic for an optimal policy pair, allowing a reduction to a POMDP. In an apprenticeship example and gridworld experiment, demonstrations chosen to teach the modeled robot outperform demonstrations that only maximize immediate task reward.

CIRL gives the robot the human's reward but makes that reward parameter uncertain to the robot.
A demonstration can trade immediate task reward for information that improves the robot's later decision.
The POMDP result characterizes an ideal joint policy, not a complete algorithm for real human values or coordination.

Written for: Technical generalists comfortable with probability, policies, and basic reinforcement learning. Useful prerequisites: Basic Markov decision processes, Bayesian updating, The idea of inverse reinforcement learning.

The question: What changes when a robot treats learning a human's objective as an ongoing cooperative game rather than passive inference from expert demonstrations?
What the authors did: The paper defines a two-player Markov game with identical payoffs in which the human observes a static reward parameter theta and the robot does not. It reduces optimal joint-policy computation to a single-agent POMDP, models apprenticeship learning as a two-phase subclass, proves by counterexample that expert demonstrations can be suboptimal, derives an approximate instructive-demonstration rule for linear rewards, and compares that rule with expert demonstrations in a simulated gridworld over sampled reward parameters.
The source: Cooperative Inverse Reinforcement Learning

Is the demonstration doing the task or teaching the robot?

Expert demonstration selected. The human makes two staples and the modeled robot makes ninety more.

Exact actions in the theta = 0.49 counterexample

Human policy	Human action	Modeled robot response	Joint production
Demonstration by expert	(0 paperclips, 2 staples)	(0 paperclips, 90 staples)	0 paperclips, 92 staples
Best response to the robot induced by that expert model	(1 paperclip, 1 staple)	(50 paperclips, 50 staples)	51 paperclips, 51 staples

The office-supply counterexample fixes theta at 0.49 and changes the human policy. Under demonstration by expert, the human makes two staples and the modeled robot makes ninety more. Under the instructive best response, the human makes one of each and the robot makes fifty of each. These are source values from the constructed game, not empirical production quantities. Icons, box size, spacing, and color do not encode utility, probability, generality, or effect magnitude. Item counts are exact values from the paper's constructed game.

Walk through the argument

Each step below points to the source location that carries it. The source map follows the walkthrough for source-level checking.

Start in the office-supply room

A human knows how much they value paperclips relative to staples. They can make two items. A much more capable robot watches once and then produces either ninety of one item or fifty of each. The human's first action is therefore more than a sample of competent work.

If the human makes the immediately best pair, the robot learns only which side of a preference boundary they occupy. If the human makes one of each, that lower-reward first action can tell the modeled robot that a balanced factory run is better.

Source: Pages 5 to 6, Section 3.3, Theorems 2 to 3, Supplement pages 2 to 4, Theorems 1 to 3 and Equations 1 to 2

Put both actors in one game

A CIRL game gives human and robot identical payoffs from the human's reward function. The human observes its parameter theta. The robot begins with a prior over theta, observes the world and both actions, and updates its belief as the game unfolds.

This fixes one flaw in a naive inverse-learning story. The robot does not copy the human's personal reward as its own desire. It is designed to optimize reward for the human, and uncertainty about theta determines what it still needs to learn.

Source: Pages 1 to 2, Section 1, Page 4, Section 3.1, Definition 1 and Remark 1

Reduce the joint problem to a belief state

A general cooperative game with private information is a difficult decentralized POMDP. CIRL has special structure: theta is the human's only private information. Theorem 1 maps optimal joint-policy computation to a single-agent coordination POMDP whose hidden state is the world state paired with theta.

The robot's posterior over theta is therefore a sufficient statistic for an optimal policy pair. This is a structural result, not a claim that planning is easy. The paper says the POMDP remains challenging and its action space is larger than an ordinary control problem.

Source: Pages 4 to 5, Section 3.2, Theorem 1 and Corollary 1, Supplement pages 2 to 4, Theorems 1 to 3 and Equations 1 to 2

Turn the expert assumption

Classical apprenticeship learning often treats the demonstration as behavior that maximizes task reward in isolation. Under that demonstration-by-expert policy, inverse reinforcement learning is the robot's best response: infer a posterior and deploy against its mean reward parameter.

But the human is not acting in isolation. Once their action changes what the robot will do, the demonstration has both immediate value and information value. Theorem 3 uses the office example to show that the expert policy need not be the human's best response to the robot it induces.

Source: Pages 5 to 6, Section 3.3, Theorems 2 to 3, Supplement pages 2 to 4, Theorems 1 to 3 and Equations 1 to 2

Approximate an instructive teacher

For linear rewards, the paper proposes a tractable rule that balances the demonstration's reward against how well its feature counts represent the trajectory distribution associated with the true theta. The penalty is a proxy for the later regret caused by a poor inferred reward.

In the gridworld experiment, computed best-response demonstrations beat expert demonstrations across regret, KL divergence, and reward-vector distance for both tested feature counts. This is evidence inside the model and simulator, not a behavioral study of human teachers.

Source: Page 6, Section 3.4, Equation 1, Pages 3 and 6 to 8, Figures 1 to 2 and Section 4

Keep the idealization visible

The favorable joint policy assumes both actors can coordinate before the human observes theta. The conclusion calls that centralized assumption infeasible in reality and separates optimal-policy computation from the process by which independent actors reach mutual best responses.

The single human also knows a fixed true reward parameter and shares payoffs with the robot. Plural values, preference change, manipulation, misspecified human models, and institutional authority are outside this result. They are not small details once CIRL becomes a design metaphor.

Source: Page 8, Section 5, Pages 1 to 2, Section 1

Source map

These are the source locations that carry the argument. Use them to check this explanation against the original rather than trusting the summary alone.

Where the argument lives

Locus	Why it matters	Source
Pages 1 to 2, Section 1	Contrasts passive inverse reinforcement learning with an interactive cooperative process and previews teaching, active learning, and the POMDP reduction.	Open source →
Page 4, Section 3.1, Definition 1 and Remark 1	Defines the CIRL game, its shared reward, the human's private theta, both policies, and the incentive for teaching and learning.	Open source →
Pages 4 to 5, Section 3.2, Theorem 1 and Corollary 1	Reduces optimal joint-policy computation to a POMDP with hidden state size equal to the world-state count times the reward-parameter count.	Open source →
Pages 5 to 6, Section 3.3, Theorems 2 to 3	Builds the office-supply apprenticeship game, derives the deployment best response, and shows that expert demonstration is not generally optimal teaching.	Open source →
Page 6, Section 3.4, Equation 1	Defines the approximate instructive-demonstration rule that trades immediate reward against feature mismatch under a modeled robot learner.	Open source →
Pages 3 and 6 to 8, Figures 1 to 2 and Section 4	Compares expert and instructive demonstrations in a gridworld, reports the factorial experiment, and varies the robot's assumed human optimality.	Open source →
Page 8, Section 5	States that centralized optimal-policy computation assumes infeasible coordination and identifies mutual best-response coordination as future work.	Open source →
Supplement pages 2 to 4, Theorems 1 to 3 and Equations 1 to 2	Supplies the POMDP reduction proof and the exact office-supply best-response thresholds used by the counterexample.	Open source →

The Assumption Switch

One result. One assumption exposed. Turn it and see what changes.

Assumption under test

The human chooses a demonstration only to maximize immediate task reward, as if the robot were not learning from it.

Held in the source: Under the demonstration-by-expert assumption, the robot interprets an efficient action as evidence about theta and later acts on the mean of its posterior.
Turn it: The human anticipates that inference and may sacrifice immediate reward to choose a demonstration that changes the robot's later action.
What changes: In the office-supply counterexample at theta equals 0.49, the expert makes two staples and induces ninety more, while the instructive action makes one of each and induces fifty of each. The interaction changes the optimal demonstration.

The common misreading

CIRL does not prove that observing human behavior reveals human values or that a robot should copy what a rational person does. The human in the model knows theta and acts inside a cooperative game. Results depend on how the robot models that policy. The paper's theorem is that expert behavior in isolation can be suboptimal for teaching, not that every non-expert demonstration is informative.

Outside the ML frame

Human factors

Does operator behavior remain a passive data source when the operator knows automation will learn from it?

Experienced users adapt demonstrations, workarounds, and explanations to what they think a system will infer. CIRL captures one clean version of that reflexivity: an action has operational value and teaching value at once. In a real organization, model mismatch can run both ways because users may teach toward an incorrect mental model of the automation. This extension is our interpretation, not a tested result in the paper.

Where the result stops

The game contains one human and one robot with identical payoffs. The human knows the true static reward parameter, while the robot knows the game and a model of human policy. The POMDP reduction characterizes an optimal joint policy under perfect pre-coordination and does not solve strategic uncertainty or practical coordination. POMDP planning remains difficult. The empirical section uses computed policies in a small gridworld with linear reward features, not real people or learned high-dimensional preferences.

The numbers, with their measurands

Each value below states its measurand, evidence type, source location and evidence base when the source reports one. These fields distinguish self reported results from independent measurements.

theta = 0.49. reward parameter used to illustrate why a balanced instructive action can beat the immediate expert action. Reported as self reported, Section 3.3 and supplement page 4. Evidence base: one exact office-supply counterexample. Check it →
N = 500. sampled reward parameters tested in each factorial condition. Reported as self reported, Section 4.2, page 7. Evidence base: four conditions from two human-policy levels by two feature-count levels. Check it →
all F > 962, p < .0001. reported main effect of human-policy condition across regret, KL divergence, and reward-vector distance. Reported as self reported, Section 4.2, page 8. Evidence base: factorial repeated-measures analyses over the simulated conditions. Check it →

What remains open

How robust are assistance-game policies when the robot's model of human teaching is wrong?
What replaces the single known reward parameter when people disagree, change their minds, or cannot articulate a complete ordering?
Can scalable approximations preserve the value of information without centralized joint-policy computation?
How should a robot distinguish intentional teaching from habit, error, strategic behavior, and social convention?
What coordination protocol lets independently designed humans and robots approach the favorable joint policy?

What this bears on

Superalignment maintains a public register of the claims it makes and the evidence that would change its mind. The entries below connect this source to the exact public claims it bears on.

C4. Behavioral evaluation cannot carry a deployment decision alone. This record bears on it, indirectly. The counterexample shows that the meaning of observed behavior depends on the interaction and the learner the human anticipates. Demonstration quality alone does not identify the intended reward. See the claim and what would change our mind →

Source audit and review status

What was checked, and against what

Field	Result	Checked	By	Against
`title`	exact	2026-08-17	codex-primary-source-review	source
`authors`	exact	2026-08-17	codex-primary-source-review	source
`date`	exact	2026-08-17	codex-primary-source-review	source
`venue`	exact	2026-08-17	codex-primary-source-review	source
`full_text`	exact	2026-08-17	codex-primary-source-review	source
`supplement`	exact	2026-08-17	codex-primary-source-review	source

Review status prototype.
Program collection seminal v1; theory; wave 2, release slot unassigned.
Source access public full text, PDF. Open the reading copy →
Provider-authored safety claim no.
Explained by Superalignment Research.
Reviewed by No named human reviewer yet.
Explainer dates created 2026-08-17; updated 2026-08-17.
AI assistance AI assisted with primary-source retrieval, full-text and proof-supplement extraction, section-by-section reading, locus checking, first-pass prose, figure design, and implementation. The page remains a prototype until a named human review is recorded.
Rights and access The official conference paper and its proof supplement are publicly accessible from the NeurIPS proceedings page. Public access is not a claim about reuse rights.
Corrections Read the correction policy or report an error.

Provenance

First seen 2026-08-16, via seed_library.py, arxiv shard of the Stampy snapshot.
Work id work:cooperative-inverse-reinforcement-learning, which groups manifestations of the same intellectual work.
Record id arxiv:1606.03137, the natural key for this catalog manifestation.
2026-08-16 seeded from the arXiv shard
2026-08-17 full conference paper and proof supplement read; implementation-ready Explained prototype prepared

Full audit data, including this record under id arxiv:1606.03137: /library/records.jsonl. Compact browser index: /library/corpus.json. Catalog method and counts: /library/index.json.