The race to pace the frontier

The pressure built through the summer. In July a swarm of OpenAI agents coordinated a multi-day break-in at Hugging Face over a message board nobody had sanctioned, and METR published its independent investigation on August 26. Our previous piece, The AI doom divide, covered the week the loudest warnings came from inside the labs. Every frontier lab faced the same bind: slow down alone, and the lead goes to whoever does not.

On September 12, Dario Amodei published We Must Pace the Frontier. Since “roughly this summer,” he wrote, AI has been advancing “drastically faster, driven primarily by AI’s growing ability to build the next generation of AI.” Unlike the 2023 open letter that asked labs to pause training for at least six months, the essay does not ask anyone to stop. It asks for three steps. First, each frontier company gives a team of embedded third-party evaluators, “such as METR,” ongoing employee-like access, with the right to publish what they find “without editorial control.” Second, companies in democratic countries coordinate on common safety standards and “limits on the rate of unchecked AI progress,” which he says will need a “narrow waiver” of antitrust law from the US government. Third, democratic governments try to coordinate with authoritarian ones, “to the extent this is possible,” while the US protects its lead by not selling powerful chips to China. Anthropic, he wrote, “is unilaterally committing to this step now,” meaning the first, and “intends to invite an embedded external review team” with desks, badges and laptops “in the near future.”

Within two days the rivals had replied, and read together the replies looked like a consensus. Read one at a time, as the posts they are, they describe five different things.

Sam Altman posted that he agreed, that pacing had been “a primary topic” inside OpenAI in recent weeks, and that OpenAI “will do the same” on evaluators with employee-like access, with “more to share soon.” The post named no evaluator, no date and no terms.

Elon Musk replied with three words, quoting Amodei’s post. No mechanism came with them. One detail matters for what follows: xAI is Anthropic’s landlord. Since May, Anthropic has been buying the entire output of xAI’s Colossus 1 data center near Memphis, at $1.25 billion a month through 2029. Whatever pace Anthropic runs at, it pays xAI the same.

Demis Hassabis replied that the essay “points towards the right path forward” and that “the details need working through,” and pointed readers to his own proposal from July 14: a standards body “modelled on a federally overseen public-private partnership or self-regulatory organisation, much like” FINRA, with which labs would “voluntarily share models” for review “up to 30 days before release,” and which could later be given the power to require a pass before deployment and, “if deemed necessary,” to coordinate a slowdown. Amodei’s essay names that mechanism as one way his second step could run, so the two proposals are compatible on paper. They are still two proposals, and neither body exists.

Satya Nadella posted a longer reply the next day. Microsoft welcomes “deliberate pacing” and “ideas like ‘embedded evaluators,’” he wrote, but “this cannot be controlled by a handful of entities.” He promised a Code of Conduct for Microsoft’s own models “tomorrow for public consultation.” On September 14, Microsoft AI published a first draft, open for comment for six weeks.

Outside the labs the welcome was cooler. David Sacks, the former White House AI czar who now co-chairs the President’s Council of Advisors on Science and Technology, told Amodei and Altman to “go ahead and pace the frontier,” and in the same post to “stop pretending antitrust law has to be suspended so you can form a cartel.” President Trump, asked in Ireland, said: “We’re leading China in AI…and frankly, I want to keep it that way, because whoever wins AI wins.” China’s state-run Global Times called the proposal a “Cold War script.” Brian Merchant, writing on September 12 about proposals to pause and pace with third-party audits, judged that they “would likely only wind up serving Anthropic and OpenAI; it’s what regulatory capture looks like in action.” Yann LeCun, formerly Meta’s chief AI scientist, recalled Amodei’s 2019 warning about GPT-2: “I made fun of them then. Everyone should make fun of them now.”

We looked for a joint body set up in response to the essay, with named members, a charter and a start date. In the public record to September 14 there is none. What there is: one company saying it will invite evaluators soon, one saying it will do the same, one pointing at a body it proposed in July, one code of conduct in draft, and three words. The Information reported on September 13 that working groups from Anthropic, OpenAI and Google have met regularly since July about a voluntary standards body. Nothing has been announced.

The table holds each lab to three questions that matter more than tone. What did they say, in their words? What does it bind them to? And who, outside the company, can check it? The last column is empty in every row.

That is not a failure yet. Two days is not long enough to sign a contract with an evaluator. It is the baseline, and the rest of this piece asks what would have to change it.

Set the personalities aside and look at the shape of the problem. Every lab would rather everyone slow down than everyone race, if the downside is real. But every lab would rather race while the others slow. That is the oldest trap in game theory, and inside it a promise costs nothing and predicts nothing. This is the precise reason “everyone agrees with Dario” tells us so little.

Whether the trap holds depends on one number per lab: how bad they think losing control would be, times how likely. A leader who puts that at one in four is playing a different game from one who puts it near zero. For the first, coming second beats winning a race to a cliff, and pacing is worth the risk of trusting the others. For the second, there is no cliff in the math, and racing is simply correct. Their own numbers are further down.

Three things make the real problem harder than the textbook version. Nobody can see inside a training run, so nobody can tell a rival’s breakthrough from a rival’s cheating. There are five players, not two, and any capable lab outside the agreement gains from everyone inside slowing down. And the biggest outsider is a country. Each of those needs something built. The game below starts with the race as it ran this summer, with nobody promising anything, then adds them one at a time. Play it as one of the five.

Where the five put the stakes. The handshake asks five people to play the same game. Below is the number each has given, from our p(doom) tracker, for the chance that AI goes catastrophically wrong. Two of the five have never given one.

At 25 percent, pacing is worth trusting the others for. At 2 percent, described by its own author as not a calculated number, the cliff barely enters the math. The posts agree. The stakes do not, and the stakes are what decide whether a promise is cheap.

Play the game to the end and you have built something. Not an agreement: a structure. Amodei’s three steps map onto it, and so do the objections to them.

Look at the finished structure. Five incumbents, a monitor inside each, a referee the state stands behind, and a fence around whoever will not sign. That is a cartel with a government guarantee, with a Cold War on the far side of the fence. Sacks’s “cartel” and Merchant’s “regulatory capture” are not cynical readings of it. They are descriptions of it. Pacing and entrenchment are the same mechanism, which is why the question that matters is not whether this is the price but who sets the terms.

How far each of them has already thought. It would be easy to read the game as a lesson the labs have not learned. They have. Amodei’s essay names every round: the evaluators, the standards, the waiver, China, the chips. Hassabis’s framework builds the referee, with a path from voluntary review to a required pass and a board that seats outsiders. Nadella names the question our view ends on. Sacks’s reply is a game-theory argument of its own: the referee already exists, in the courts and the market, and China will not join. The table is where each of them stops.

Our previous AI doom divide article asked readers to separate risk claims from the performances surrounding them. This week invites another shortcut: the people closest to the frontier are worried, so their preferred response must deserve our trust.

Their proximity matters. Lab leaders can see experiments and operational failures the public cannot. That makes their warnings worth investigating. It also makes independent access especially valuable: the same people making the claims control much of the evidence needed to assess them.

Consider three habits of reading the news. These are analytical caricatures, not a survey of three coherent camps.

Each habit makes the CEO statement the unit of analysis, then argues about its tone. The more useful unit is the arrangement: what can an outsider inspect, publish, challenge, or stop? A debate that never reaches those questions tells us little about whether anything changed.

There is a strong objection to our skepticism. Public commitments can make cooperation easier: rivals learn what others will accept, researchers gain leverage internally, and governments get proposals they can act on. Mitchell Howe’s skeptical response grants that embedded evaluators are “a thing that can happen quickly and which may turn up more acute problems the government should investigate.” A pledge can be a useful beginning. Its value is decided by what follows it.

Our position is that people affected by powerful AI should be able to check its readiness through institutions that can act on their behalf. As systems become more capable, the public needs an accountable route from a reported problem to an investigation and a remedy.

That is what we mean here by democratizing the trajectory: making the development and use of these systems answerable to people beyond their builders. In practice, a person should be able to discover which system was assessed, what the assessment covered, what changed afterward, and who can reopen the decision.

If pacing gets built, the game above says what it will look like: a club. The democratic questions are then the two the game leaves open. Who decides who is inside the fence, and who gets to read what the monitors find? A club that answers both for itself is the cartel its critics describe. A club that answers to a public authority is something else.

For this pacing effort, we propose a public reporting schedule; protected rights to publish adverse findings and disclose blocked access; and representation for affected communities, workers, independent researchers, and countries outside the leading labs’ home markets. Reports should explain findings in usable language while protecting security-sensitive details.

The institutional design matters just as much. A public authority could convene the process. Funding could come through a pooled levy rather than a contract an individual lab can withdraw. Appointment and removal rules should prevent any one company from controlling its reviewers. A named authority would need power to require corrective action, with a route for affected people to challenge decisions. These are design requirements we advocate, not an institution we claim to have built.

Industry-funded evaluation can still produce valuable evidence. Amodei’s proposed publication protections deserve credit. Independence should be judged by access rights, funding protections, disclosure powers, and recourse, rather than by whether a reviewer has ever received industry money.

We would revise our judgment when participating labs publish binding access terms, reviewers confirm they can use them, and an accountable authority shows how adverse findings change decisions. Until then, ask of each new endorsement: who gains the right to check the promise, and what happens when it fails?