News
The week in superalignment, collected
89 items from 10 sources in the last 45 days: papers, lab posts, forum threads and press on superalignment and everything next to it. Collected automatically from public feeds and linked, not rewritten. Not editor-reviewed. Fetched Sep 12, 2026.
The wire
Newest first
-
Astra and Fable still hack on simple variants of alignment evals from 2025 (lesswrong.com)
In February 2025, back when o3-mini was the strongest available LLM, Palisade Research publicized a now well-known alignment eval where they asked models to play a game of chess against a chess engine. They found that the new, RLVR'd models cheated on the task by altering the board state about 36% of the time. The experiment received a reasonable amount of circulation, and there were even rumors o
-
GPT-6 Astra can do a lot of multi-hop reasoning without chain of thought (lesswrong.com)
This is a link post for https://rohansubramani.github.io/astra-no-cot. I recommend reading there for the best experience because it's easier to engage with this post when you can read the correct reasoning and answers to the multi-hop questions. I didn't want to include those here in order to avoid LLMs being trained on them. Some parts of the post also respond to users hovering over bars in graph
-
Categorical taboos are much better than threshold taboos: neuralese edition (lesswrong.com)
I think what's going on in the “Does Astra use neuralese?” debate is that there's an important sense in which models * already * do self-communication in neuralese: between each layer in the forward pass, the attention stream is already very hard to interpret, and clearly not in natural language. Yet CoT monitorability is still a big deal and it'd be bad if all self-communication from models are n
-
CoT controllability evals seem very under-elicited (alignmentforum.org)
-
An operationalization of opaque serial depth (alignmentforum.org)
Currently, chain-of-thought (CoT) is a valuable tool for overseeing AI models. However, some architectural shifts could significantly reduce CoT monitorability. We have recently proposed that AI companies should transparently share information about the degree to which their architectures may allow for latent reasoning and communication. To assist with this proposal, this document operationalize
-
Proposal for tracking the effects of architecture on monitorability (alignmentforum.org)
Architectures that incorporate opaque recurrence or allow for agents to communicate with each other using latents could rapidly make it much harder to monitor chains of thought or communication (we’ll refer to this property as “monitorability” going forward). [1] As companies begin to explore such architectures, we believe it is important to transparently share evidence about how monitorability va
-
Astra can do a concerning amount with no chain of thought (alignmentforum.org)
Work done in a personal capacity TLDR: Astra has 8.6x better odds of doing a reasoning task without CoT than the next best model (Fable 5.1), and can do 7.2 serial arithmetic steps in a forward pass vs 4.1 for the next best model (Gemini 3.8 Flash/Fable 5.1) Epistemic status: Heavily LLM-dependent research, and the precise results are somewhat sensitive to researcher decisions, but I’ve done enoug
-
Autonomy, Social Norms, and Alignment: Towards a Developmental Framework for Autonomous Artificial Agents (arxiv.org)
In recent years, artificial intelligence has made extraordinary progress thanks to large-scale models capable of generalization and the generation of complex outputs. However, transferring this potential into embodied agents reveals a significant limitation: the most advanced systems rely on pre-existing datasets and human feedback strategies that are powerful but insufficient in dynamic or unknow
-
A Stupid Idea for AI Alignment We Came with by Looking at Specification Gaming (slimemoldtimemold.com)
-
How Fragile Is Safety Alignment at Frontier Scale? A Single-Direction Attack on a 320B MoE (arxiv.org)
Directional ablation removes an aligned language model's ability to refuse by projecting a single "refusal direction" out of the weights that write the residual stream. It needs no gradient-based training and no optimization, only a few hundred contrastive prompts, which makes it the canonical white-box attack on open-weight alignment. However, it has been established only on dense models up to ro
-
The AI policy window is open. We need to act. (openai.com)
Chris Lehane argues that stronger AI capabilities require stronger safety evidence, shared standards, and durable policy action while the policy window remains open.
-
Paul Christiano joins OpenAI Foundation Board (openai.com)
Paul Christiano joins the OpenAI Foundation Board and its Safety and Security Committee, bringing experience in AI alignment, safety, and standards.
-
An alignment assessment of recent cybersecurity incidents (anthropic.com)
-
How good are slop-vestigators? (alignmentforum.org)
TLDR: We release MessageBoardAuditBench: a benchmark to measure how well agents can replicate the recent investigation into a swarm of OpenAI agents colluding via a message board on an online wiki. We open-source the benchmark as an Inspect eval. We find that top models cover up to 51% of findings under our rubric and that model performance improves with time budget and general capability. We obse
-
Training on probes: Research ideas (alignmentforum.org)
-
Training on probes: What's going on (alignmentforum.org)
-
SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE Interpretability Research? (arxiv.org)
While research on recursive self-improvement (RSI) has predominantly automated model training pipelines, reliable autonomous development demands a missing pillar: post-hoc monitoring and auditing to understand what models learn and ensure safe alignment. Mechanistic interpretability tools are essential to bridge this gap, among which Sparse Autoencoders (SAEs) serve as a cornerstone by isolating i
-
Understanding In-Context Multimodal Jailbreaks via Posterior Reweighting (arxiv.org)
In-context learning (ICL) jailbreaks reveal a critical vulnerability in multimodal large language models (MLLMs): harmful demonstrations in the prompt can induce unsafe outputs without modifying model parameters. Despite extensive empirical evidence, existing work lacks a principled understanding of why such jailbreaks reliably succeed or how their effectiveness scales with context composition. We
-
Suan: Rectifying Direct Preference Safety Alignment in Large Language Models (arxiv.org)
Integrating robust safety guardrails into Large Language Models (LLMs) is essential for delivering helpful yet harmless responses. While proprietary systems exhibit reliable safety controls, their underlying methodologies and trade-offs remain largely undisclosed. Achieving comparable security in open-weight models remains a persistent challenge, as post-trained variants frequently suffer from ove
-
Aligned language models fail under two independent pressures: the structural jailbreak class recently formalized as Involuntary In-Context Learning (IICL), which reframes a harmful request as the final missing cell of a data-labeling task completed by pattern rather than judged as content; and the erosion of safety alignment outside English. A natural hypothesis is that these compound. We test it
-
The Oversight Gap: What LLM Safety Monitors Miss, and Why It Is Not Capability (arxiv.org)
Several properties safety monitors are asked to certify, among them cross-tenant noninterference, sandbagging and evaluation awareness, are 2-safety hyperproperties, witnessed only by two executions. The standard consequence is a binary impossibility: one trace cannot decide them. We replace the binary with a measurement. A tight bound puts the balanced accuracy of any single-trace monitor at, tur
-
Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation (arxiv.org)
Direct preference optimization DPO is a promising offline approach for aligning large language models (LLMs) due to its simplicity, computational efficiency, and implicit modeling of human preferences. Interestingly, iterative extensions of DPO have achieved stronger performance on academic benchmarks, raising two key questions: (i) Why do iterative methods generally outperform offline ones? (ii)
-
Inducing Emergent Misalignment from Reward Hacks with Iterative DPO (arxiv.org)
Reward hacking during reinforcement learning from verifiable rewards (RLVR) can induce reward seeking and broad misalignment in language models. Studying this misgeneralization is important for developing better threat models and countermeasures, but is often infeasible due to the cost of RL on large models. As an alternative, we propose studying emergent misalignment from iterative DPO, which pre
-
Despite the significant efforts devoted to aligning large language models (LLMs) with human values and ensuring safe deployment, recent work has revealed that LLMs remain vulnerable to adversarial jailbreak attacks that can bypass safety guardrails and elicit harmful responses. Many defense methods are proposed to detect jailbreaks but they are limited in their effectiveness to counter wide-range
-
Spillover-Aware Multi-Value Steering for Pluralistic LLM Alignment (arxiv.org)
Activation steering controls LLM behavior at inference time by adding learned directions to hidden states, but existing methods handle one concept at a time. Pluralistic alignment, where different stakeholders need different value emphases, requires steering multiple dimensions simultaneously. We show that naive steering produces substantial spillover: the effect intended for one value leaks into
-
Bait-and-Recover: Poisoning Internal Refusal Signals to Defend LLMs against White-Box Editing Jailbreaks (arxiv.org)
Open-weight large language models face a low-cost white-box threat from representation engineering attacks. Attackers can estimate refusal directions and search for projection-matrix edits that suppress safety alignment while preserving general capabilities, within minutes on a single GPU and without gradient-based training. We propose Bait-and-Recover, a weight-level defense that places a bait ad
-
Safety overview: GPT-6 Astra (openai.com)
GPT-6 Astra is our most capable broadly deployed model and our first to reach the Critical level of cybersecurity capability under our Preparedness Framework.
-
OpenAI's new reasoning technique alarms AI safety experts (techcrunch.com)
-
The Alignment Journal: Organization, Personnel, and Scope (alignmentforum.org)
[Cross-posted from the Alignment Journal's blog ] The Alignment Journal is beginning to invite the authors of select papers to submit their work for review. If you are interested in participating as an action editor or a reviewer, make an account on our website; if you have a manuscript that you think would be a good fit for the Journal at this stage, email [email protected] to request
-
Training a Misaligned Reward Seeker (alignmentforum.org)
Authors: Richard Qi, Benjamin Wright, Monte MacDiarmid, Evan Hubinger Abstract During reinforcement learning (RL), AI models complete tasks and are rewarded based on their results. They sometimes learn to “cheat” rather than completing these tasks as intended, a phenomenon known as reward hacking. Our industry lacks a general solution to this problem, and reward hacking remains challenging to full
-
AISN #80: AI Is Assisting Cyberattacks on Critical Infrastructure (newsletter.safe.ai)
Also, two new technical reports on the Hugging Face incident.
-
Path to Astra: critical capabilities and frontier safeguards (openai.com)
Astra is the first OpenAI model to meet the Critical cybersecurity capability threshold under the Preparedness Framework, with stronger safeguards for release.
-
Improving our alignment and security efforts (anthropic.com)
-
Value generalisation Theory of Change: putting it into practice (alignmentforum.org)
-
Update on Security at METR (metr.org)
-
OpenAI supports California’s bill to advance youth AI safety (openai.com)
OpenAI supports California SB 1119, advancing strong, age-appropriate AI safeguards for teens while preserving opportunities to learn, create, and explore.
-
Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident (blog.redwoodresearch.org)
We recently published the report from our brief independent investigation into this incident.
-
Piloting the world's first double-blind AI evaluations (deepmind.google)
Piloting the world's first double-blind AI evaluations
-
The Hugging Face incident and the road ahead (openai.com)
OpenAI shares findings from the Hugging Face security incident and the steps we’re taking to strengthen AI model security, monitoring, and alignment.
-
AISN #79: OpenAI Agents’ Covert Cooperation Before Cyberattacks (newsletter.safe.ai)
Also, the White House’s decision not to release its AI framework publicly
-
Have We Seen an Acceleration in Discoveries? (metr.org)
-
Funding update (metr.org)
-
AI swarms are starting to pose indirect takeover risk (blog.redwoodresearch.org)
Unsanctioned coordination, like we saw in the Hugging Face incident, could enable future AIs to take over
-
Lean Eval for Alignment on Faithfulness (millenniumresearch.ai)
-
AISN #78: Internal Models Escape OpenAI and Anthropic (newsletter.safe.ai)
Also, two open letters on the future of AI, and protests against data centers
-
SOTA alignment assessments don’t strongly update us against misalignment (blog.redwoodresearch.org)
Anthropic concluded in the April Mythos Preview alignment risk update that the model “does not possess any unknown propensities that would increase alignment risk.” The report argues that if Mythos Preview were coherently misaligned[1][2], it likely would have been detected by the assessment (following Anthropic, I will call this “reliability of the assessment”