News
The week in superalignment, collected
91 items from 8 sources in the last 45 days: papers, lab posts, forum threads and press on superalignment and everything next to it. Collected automatically from public feeds and linked, not rewritten. Not editor-reviewed. Fetched Oct 22, 2026.
The wire
Newest first
-
Neural nets are exceptionally vulnerable to goal self-hack (alignmentforum.org)
Suppose we have an AI CEO, ruthlessly devoted to maximising shareholder value. Fortunately, we haven't been completely stupid, and have given it a rich definition of shareholder value, encompassing the success, flourishing, safety, preference satisfaction, and moral values of the shareholders, not just the stock price. So it won't destroy the company - or the world - to build a giant stock-ticker
-
Minimum required pacing: finish aligning it before you train the next one (lesswrong.com)
Let's say we coordinate to pace the frontier (rather than pausing outright or failing to coordinate). What should the standard of pacing be? My proposal for a minimum requirement: a lab cannot train the next generation of models (in terms of capabilities) until they have adequately [1] resolved all significant alignment issues in the present generation of models. You don't launch a jet that goes a
-
Game Design Creativity evals (lesswrong.com)
A personal eval I run on new AI models is “design a novel game, with mechanics that have never been seen before.” I do this because this is a domain I have enough taste in that I think I can tell if they are doing something actually creatively interesting. [1] For a long time, they would always say “time manipulation mechanics!” as their first answer. Braid was cool, guys, but, it's almost 20 year
-
Labs’ current alignment plans seem terrible (lesswrong.com)
Tl;dr: The frontier AI companies’ implicit alignment plans center on rushing towards RSI with something like current models. This plan is (at best) contentious, and many believe it will have catastrophic consequences. Nonetheless, it is the track we are on. It is therefore important to collect evidence which helps us anticipate how this will go. - Note: This post is based on conversations inside t
-
More people should write basic AI safety explainers (lesswrong.com)
I think people continue to underrate the value of writing the clearest, most basic, simplest possible case for important parts of AI safety. For example, before I wrote my own explainer for the simplest possible case for AI doom, I noticed that all existing explainers, of which there aren't many, had multiple of the six problems below: They have too many extraneous details They appeal to overly co
-
Democratic governance of frontier AI, part II: Empowering cyber defenders (openai.com)
OpenAI outlines a blueprint for empowering cyber defenders with frontier AI, stronger safeguards, modernized cybersecurity, and more resilient public institutions.
-
Advancing coding in low‑resource programming languages (openai.com)
Learn how OpenAI is improving ChatGPT across low-resource programming languages, with new evaluations for software engineering, SQLite, and COBOL migration.
-
The Alignment Problem: Computer Science Meets Analytic Philosophy (alignmentforum.org)
-
Deconfusing Corrigibility (alignmentforum.org)
-
A Simpler and Better Debate Protocol for Scalable Oversight (alignmentforum.org)
-
Screwtape: Adversarial Transcript Editing for Monitor Red-Teaming (blog.redwoodresearch.org)
Automatically finding monitor failures by editing real agent transcripts
-
Preface to the Computational Agent Theory and ASI Alignment Sequence (alignmentforum.org)
The idea of “artificial superintelligence [1] ” (ASI) has recently crossed from the realm of science fiction (in the mainstream perception) to the realm of near-term technological goals openly pursued by most leading commercial organizations in the field of AI. Unfortunately, ASI poses a grave existential risk to the human species. This was argued by Eliezer Yudkowsky since 2001 [2], and by many o
-
Our approach to working with AI companies (metr.org)
-
The Continual Constitutional Convention: How language models revise the documents that govern them (alignmentforum.org)
Executive Summary Human societies exhibit broad shifts in values and norms over time, often driven by moral disagreements. One way of understanding moral progress is the resolution of these disagreements through a process of reflection by rational agents. Frontier AI companies use documents such as constitutions, model specs, and system prompts to guide the moral orientation of AI systems. In this
-
AISN #83: Six AI Companies Sign White House Accord (newsletter.safe.ai)
Also, the Team Human campaign and a new introductory course on AI safety.
-
OpenAI Can't Self-Police Because It Fires the People Raising the Alarm (alignmentforum.org)
By Alex Turner, Amber Scorah, and Karl Koch. Op-ed written for Fortune. Giving independent evaluators “employee-like access” is “a great idea”, OpenAI CEO Sam Altman said on September 12th. Two weeks later, OpenAI president Greg Brockman signed the White House Accord on Super Intelligence which outlined the administration’s answer to growing fears about AI: the companies would police themselves. O
-
[Paper] Distillation for Incrimination and Distillation for Capabilities (blog.redwoodresearch.org)
Empirical work on the Distillation Double Bind.
-
Sophos cuts threat investigation time by 96% with OpenAI Daybreak (openai.com)
Discover how Sophos uses OpenAI’s Daybreak to cut cyber-threat investigation time by 96% and automate 52% of MDR cases while preserving human oversight.
-
AI systems could cover up misbehavior (metr.org)
-
Frontier models state different decision theory preferences depending on who’s asking (blog.redwoodresearch.org)
Asked which decision theory is right, frontier models name FDT/UDT. But add a sentence saying you're an academic philosopher, and they often switch to CDT.
-
Rocks to Rules: Climbing a Rickety-But-Interoperable Semantic Ladder (alignmentforum.org)
-
OpenAI safety leader quits, warning AI company's culture is 'broken' (theguardian.com)
-
Capabilities research expands the safety-usefulness Pareto frontier too (blog.redwoodresearch.org)
A model of why capabilities research usually increases risk despite enabling safety - and when it can reduce risk
-
Endogenous Alignment Requires Dependence (alignmentforum.org)
My post on endogenous alignment got several excellent comments. Among them, one from Karl Krueger argued that imitation learning is more important for how humans align their children than reinforcement learning. Teasing out which is more important seems hard, but I agree I was wrong to ignore the importance of imitation, and arguably imitation learning is more fundamental for a reason explained by
-
AISN #82: The US and China Begin a Dialogue on AI (newsletter.safe.ai)
Also, AI companies’ discussions about auditing each other, and a new benchmark testing models’ tendencies to cheat.
-
Towards safety cases for frontier AI training (openai.com)
Our early guidelines for safety cases in frontier AI training cover technical safeguards, operational practices, and investigating misalignment incidents
-
Continual learning might make your blocking monitors nearly useless (blog.redwoodresearch.org)
When monitor evasion looks like legitimate learning to your continual learning system, it’s hard to have one without the other
-
AI safety is mostly a sex cult in Berkeley (verysane.ai)
-
Latent reasoning architectures would likely undermine CoT, our strongest oversight tool (blog.redwoodresearch.org)
We should have a strong presumption that latent reasoning architectures would make oversight far more difficult.
-
Astra is much better at reasoning with filler tokens than previous models (blog.redwoodresearch.org)
Unlike other models we tested, GPT-6 Astra performs modestly better on general benchmarks with filler tokens, and significantly better on serial depth-heavy tasks.
-
Sam Altman’s remarks at the United Nations Security Council (openai.com)
OpenAI CEO Sam Altman discusses AI safety, human control, and international cooperation in remarks to the United Nations Security Council.
-
Priorities and principles for effective third party assessments (openai.com)
OpenAI outlines priorities and principles for rigorous, secure, and independent third-party AI safety assessments of frontier models and safeguards.
-
AI Safety Is Mostly a Sex Cult (bsky.app)
-
OpenAI discloses six new AI safety incidents (axios.com)
-
AISN #81: Anthropic Researcher’s Resignation Propels AI Risk into the Public Eye (newsletter.safe.ai)
Also, OpenAI releases GPT-6 Astra.
-
AI Regulation as Anthropic's Business Model (twitter.com)
-
CoT controllability evals seem very under-elicited (blog.redwoodresearch.org)
Simple prompt optimizations can improve model capability to control their reasoning
-
An operationalization of opaque serial depth (blog.redwoodresearch.org)
"Serial depth between text bottlenecks" as a proxy for latent reasoning abilities.
-
Proposal for tracking the effects of architecture on monitorability (blog.redwoodresearch.org)
Architectures that incorporate opaque recurrence or allow agents to communicate using latents could rapidly make it much harder to monitor chains of thought. We propose that AI companies regularly report verified information about opaque serial depth, share monitorability evidence, and publish a policy on architectures that could degrade monitorability.
-
A Stupid Idea for AI Alignment We Came with by Looking at Specification Gaming (slimemoldtimemold.com)
-
An alignment assessment of recent cybersecurity incidents (anthropic.com)