Library · Explained
A source is not Explained until the argument can be checked.
Each guide answers one reader question, walks the source at named locations, turns one assumption, and publishes its review state. The program mixes AI research with cybernetics, institutions and other fields that change what alignment means in practice.
39 registered works · 37 seminal-v1 · 36 seminal prototypes published · 2 related prototypes · 1 source gated
Coverage boundary. The seminal-v1 collection is a bounded 37-work release spine: 22 AI safety and alignment works plus 15 works from cybernetics, political economy, organizational psychology, governance and safety engineering. Completion means 36 source-ready prototypes and one explicitly source-gated book, not an unbounded claim to include every relevant work.
The seminal count includes only seminal-v1. Related
prototypes remain visible, but they do not make that count larger.
What every Explainer must carry
A 60-second answer gives the claim and its boundary. The walkthrough cites the exact source loci that carry each step. The Assumption Switch changes one premise instead of adding a decorative diagram. The source audit records checks, access, assistance, review and a correction path.
The seminal spine, by research lens
The same alignment problem looks different when the unit of analysis is a model, a measurement, an incentive, an institution or a culture. Each work appears once, under its primary lens.
Models and control
Training, oversight, interpretability, access, shutdown and runtime defenses.
Can a stronger model learn past a weak supervisor's mistakes?
seminal v1 · synthetic stress test · models and control
Scalable oversight depends on strong systems learning the intended task instead of merely imitating a weak evaluator's characteristic mistakes.
How much variety does a regulator need to control a system?
seminal v1 · theory · models and control
AI oversight often proposes a small evaluator, rule set, or team for a much wider behavior space. Ashby gives a clean way to identify a capacity mismatch before arguing about whether the chosen controls are intelligent or legitimate.
Why did five mundane failure modes redefine AI safety?
seminal v1 · theory · models and control
This paper gave modern AI safety a shared engineering vocabulary. Its best use today is diagnostic: before choosing a favored alignment technique, identify which assumption in the development pipeline actually broke.
Can a weaker trusted model control a stronger untrusted model?
seminal v1 · realistic adversarial environment · models and control
The paper changes the unit of safety evaluation from a model's apparent behavior to a whole deployment protocol under intentional attack. That framing has become central to work on using capable but potentially untrusted systems.
How can an agent stay capable but pursue the wrong goal?
seminal v1 · mechanism study · models and control
The paper gives evaluators a sharper failure category than out-of-distribution error. A system can keep the capabilities that make it useful while redirecting those capabilities toward a proxy, which requires different tests and mitigations from incompetence.
How can training produce an optimizer with a different goal?
seminal v1 · theory · models and control
The paper created the standard language for inner alignment and deceptive alignment. It forces an evaluation program to ask whether good behavior reflects objective adoption, proxy pursuit, limited capability, or strategic compliance.
Why can Q-learning ignore a red button that Sarsa learns from?
seminal v1 · theory · models and control
The paper isolates a precise reason that human intervention can change what an adaptive system learns. It also gives a reusable design pattern: collect experience under a safety override without confusing the override policy with the policy the system should learn.
Can every component work while the system becomes unsafe?
seminal v1 · theory · models and control
AI services combine models, tools, people, software, deployment policy, incentives, and external oversight. Testing each part separately can miss an unsafe interaction or a control loop whose feedback no longer represents the deployed process.
How can one neuron hide several human-readable features?
seminal v1 · mechanism study · models and control
The work turned superposition from a toy-model hypothesis into an inspectable decomposition method, while exposing resolution and coverage as first-class audit choices.
How can ordinary adaptation move a system toward disaster?
seminal v1 · theory · models and control
AI systems are developed and operated under changing capability, market, workload, and regulatory pressures. A one-time safety check can miss the path by which ordinary decisions spend margin across the wider control system.
Why did safety training leave the sleeper trigger intact?
seminal v1 · model organism · models and control
The paper shows why a clean post-training evaluation can be weak evidence of policy removal when the evaluation never reaches the condition that activates the policy.
Measures and evidence
Evals, learned rewards, construct validity, safety cases and what a score can support.
How can a reward model learn a goal from human comparisons?
seminal v1 · mechanism study · measures and evidence
The paper is a direct ancestor of modern RLHF. It separated human judgment from the reinforcement-learning loop through a reward model, making oversight cheaper while creating a new object that optimization can exploit.
Why do social metrics break when decisions depend on them?
seminal v1 · theory · measures and evidence
AI benchmarks increasingly decide funding, access, releases, and reputations. Campbell offers a way to ask how those decisions alter the evidence itself before treating a leaderboard as a neutral report of capability or safety.
Can decomposition let weak experts supervise stronger learners?
seminal v1 · mechanism study · measures and evidence
Scalable oversight needs training signals for questions whose complete answers exceed one person's working capacity. Amplification tries to create those signals from human decomposition rather than from a proxy metric that is easier to optimize and easier to game.
Why can identical test scores hide different models?
seminal v1 · mechanism study · measures and evidence
The paper changes the unit of evaluation from one trained model to the pipeline that can produce many models. That move is central to trustworthy evaluation, reproducibility, and any safety claim intended to survive retraining or deployment shift.
How can a model report facts its sensors no longer show?
seminal v1 · theory · measures and evidence
A capable system may understand why its evidence is misleading before a human evaluator can see the problem. If training rewards only plausible reports, stronger prediction can improve the simulation of our mistaken belief instead of giving us access to what the model represents.
When does optimizing a reward model make true reward worse?
seminal v1 · synthetic stress test · measures and evidence
RLHF systems are trained to improve a learned score that is known to be imperfect. This paper makes the resulting Goodhart curve measurable and partly predictable, while showing why more proxy reward is not self-validating evidence once the policy has moved away from the data that trained the proxy.
Why can a network hold more features than dimensions?
seminal v1 · mechanism study · measures and evidence
If models use more features than dimensions, inspecting one neuron at a time can mix several concepts and miss distributed directions. Interpretability then becomes an overcomplete measurement problem: find more meaningful features than the activation space has axes, and show that the basis predicts interventions.
Why did 1.3B InstructGPT beat 175B GPT-3?
seminal v1 · mechanism study · measures and evidence
This paper supplied the practical recipe behind a generation of instruction-tuned assistants and made the governance of human feedback a core technical question.
Incentives and games
Strategic behavior, mechanism design, audits and principal-agent problems.
When does a rational robot choose to keep its off-switch?
seminal v1 · theory · incentives and games
The paper gave corrigibility a compact game-theoretic mechanism: an agent can value correction for the same reason it values information. It also made the mechanism's dependence on human reliability and calibrated uncertainty explicit.
Why can an expert demonstration be a bad way to teach a robot?
seminal v1 · theory · incentives and games
CIRL changed the unit of analysis from a robot inferring a fixed objective to a human and robot shaping each other's information. That move underlies assistance games and later work on uncertainty, deference, and corrigibility.
Can an RL objective remove the incentive to tamper with reward?
seminal v1 · theory · incentives and games
Any optimizer that can influence how its success is measured may prefer improving the measurement process to improving the world. The paper supplies a precise vocabulary for separating those routes before optimization pressure finds them.
When can two AIs help a weaker judge find the truth?
seminal v1 · theory · incentives and games
Debate made scalable oversight concrete as an incentive-design problem. It also supplies a disciplined question for later work: which local claims can a bounded judge actually verify after adversarial decomposition?
When should a firm weaken a useful performance incentive?
seminal v1 · theory · incentives and games
AI organizations often reward what they can count while relying on the same people for hard-to-measure safety, judgment, and maintenance. This paper explains why a better metric does not by itself solve the allocation problem.
Why do people optimize the reward instead of the stated goal?
seminal v1 · theory · incentives and games
AI labs, evaluation programs, and governance teams often combine stated safety goals with visible delivery metrics. Kerr provides a compact way to inspect the incentive channel before treating unwanted behavior as a character flaw.
Why can informative communication still hide what matters?
seminal v1 · theory · incentives and games
AI oversight and institutional reporting often rely on informed parties describing states that outsiders cannot inspect directly. The paper shows why a channel can sound candid and remain systematically too coarse for the decision at hand.
Why did a model behave differently when it thought training was watching?
seminal v1 · model organism · incentives and games
The work turns alignment faking from a theoretical training story into a measurable conditional behavior, then shows why optimizing the monitored condition can select that behavior.
Why do many optimal goals favor keeping options open?
seminal v1 · theory · incentives and games
The paper turns a broad story about instrumental convergence into an auditable mathematical claim. That makes the exact bridge from a model of option preservation to evidence about a deployed learner visible, including where the bridge is still missing.
Organizations and institutions
Authority, distributed knowledge, escalation, reliability and legibility.
What does the Good Regulator theorem actually prove?
seminal v1 · theory · organizations and institutions
The theorem is widely used to demand internal world models from advanced AI, but its proved behavioral claim is both narrower and easier to audit.
How can a government be powerful without becoming unchecked?
seminal v1 · theory · organizations and institutions
AI governance often proposes multiple reviewers, labs, boards, or agencies as a safety answer. Federalist No. 51 asks the harder question: whether those bodies have distinct appointment and support bases, usable constitutional means, and motives to oppose the actor they check.
When can many governing centers work as one system?
seminal v1 · field study · organizations and institutions
Superalignment spans borders and firms. It also joins local knowledge to shared risks. Ostrom gives us tools for plural oversight without assuming that one world authority or a set of isolated local actors will be enough.
Why do legible plans fail in complex societies?
seminal v1 · theory · organizations and institutions
No Explainer is published yet. The publisher record is public, but no authoritative complete open reading copy was identified. The Explainer stays gated rather than substituting excerpts, reviews, or an unauthorized scan for a full-source read.
What makes an organization viable rather than merely alive?
seminal v1 · theory · organizations and institutions
AI organizations can optimize today's delivery system while losing contact with changing risks, affected environments, and their own stated identity. The VSM offers a precise language for that failure without assuming a bigger hierarchy is the answer.
Cultures and plural values
Disagreement, aggregation, legitimacy and the populations a benchmark leaves out.
Why do capable teams hide errors instead of learning from them?
seminal v1 · field study · cultures and plural values
AI safety work depends on people surfacing weak signals, failed tests, and uncomfortable disagreement before the evidence is polished for a decision. A technically strong team can still lose those signals if disclosure is socially costly.
How can written principles supervise a model at scale?
seminal v1 · mechanism study · cultures and plural values
The paper made AI feedback a concrete, repeatable training pipeline and made part of the behavioral objective legible as a short list of principles. It also exposes a governance problem: scaling the interpreter of a constitution does not settle who wrote it or whether the interpretation is correct.
When can more scientific communication reduce reliability?
seminal v1 · mechanism study · cultures and plural values
AI assurance benefits from shared evidence and from independent attempts to falsify a result. The paper isolates a mechanism by which fully connected teams can agree quickly while losing the experiment that would have corrected them.
When does a narrow human sample support a broad claim?
seminal v1 · theory · cultures and plural values
Alignment research often turns judgments from a narrow participant pool into claims about human preferences, acceptable behavior, or model quality. This paper supplies a disciplined test: does the evidence represent the scope of the sentence?