Superalignment

Wiki · Updated September 8, 2026

Evals

Tests for AI systems. Short for evaluations.

By the Superalignment Team · Last reviewed September 8, 2026

In plain terms

Evals are how AI systems get tested: score the behavior on a set of cases, because exact answers cannot be asserted the way ordinary software tests do. They are the load-bearing evidence in every deployment decision and every safety framework, and their two structural limits, a score says nothing about what is in the failures, and a capable model may not show what it can do, are now research fields of their own.

Evals are tests for AI systems, short for evaluations. Because a model's output is not a fixed value, an eval cannot assert equality the way ordinary software tests do; it scores behavior instead: on a set of cases, how often is the answer acceptable. That one change, from asserting to scoring, makes evals genuinely useful and quietly dangerous, and as deployment decisions and safety cases came to rest on them, their limitations became a research field of their own.

What an eval is

An eval bundles three choices: a set of cases, a scoring rule, and a threshold someone will act on. Each choice smuggles in an assumption. The cases stand in for the deployment distribution; the scoring rule, increasingly a model grading a model, stands in for human judgment; the threshold converts a distribution into a yes. The bundle works exactly as far as those stand-ins hold, which is why an eval is best read as an observation made inside an envelope: this model version, these cases, this grader, this date.

Capability evals ask what a system can do (coding, reasoning, agentic tasks, dangerous capabilities); alignment and propensity evals ask what it tends to do; safety evals ask whether specific harms are reachable. The distinction matters because the inference each supports is different, and the failure of each is different.

The ecosystem

Evaluation became institutional infrastructure between 2023 and 2026. METR built the canonical third-party autonomy evaluations and the field's most-cited capability trendline: Measuring AI Ability to Complete Long Tasks found the length of software task a frontier model can complete autonomously doubling roughly every seven months over 2019 to 2025, with a faster fit in the most recent window: METR's Time Horizon 1.1 update (January 2026) put the post-2023 doubling time at about four months and the doubling time since 2024 at about three, with the best model measured at a 50 percent horizon of roughly four to five hours of human working time. The UK AI Security Institute open-sourced its Inspect evaluation framework and ran pre-deployment tests across dozens of frontier models. The first cross-lab exercise, OpenAI and Anthropic evaluating each other's models, ran in August 2025, and the International Network of AI Safety Institutes reached consensus on automated evaluations in 2026. Every frontier lab's safety framework, and the International AI Safety Report 2026, routes its load-bearing claims through evals.

What evals cannot show

Two structural limits define responsible use.

A score is not the failures. An eval that returns 94 percent says nothing about whether the 6 percent contains the case that ends you. Averages are the wrong summary for deployment questions, because critical blockers do not average: one catastrophic failure mode in the residue changes the decision type, not the score. Ask what is in the failures, never what the average is.

Evals demonstrate capability, never its absence. A model that performs a task has the capability; a model that does not perform it under test conditions has not been shown to lack it. Elicitation matters (better prompting, tools, or fine-tuning routinely surface capability earlier tests missed), and the model itself may not cooperate. Sandbagging, strategic underperformance on evaluations, was demonstrated by van der Weij et al. (2024), and evaluation awareness, models recognizing test contexts and behaving differently in them, is documented well enough that mitigation results carry it as a standing confound: the anti-scheming training reductions note the improvement may partly reflect models recognizing evaluations, and evaluation designs like LURE exist specifically to reduce evaluation awareness by replaying live usage. See sandbagging.

Benchmarks add their own pathologies: contamination (test data in training data), saturation, and the pattern Safetywashing documented, safety benchmarks that track capability and compute, so capability gains score as safety progress.

How to read an eval, and how not to

The practiced reading is scope-first: which model version, which cases, which grader, which date, and what would count as the distinguishing observation the eval did not make. The failure reading, common enough that this site named it, is treating a green eval as readiness: the proxy subtype of false convergence when the score detaches from the target, the envelope subtype when a lab result is carried into a deployment it never covered. In our vocabulary, an eval is evidence, and evidence is bounded by its envelope and spends down with time and action.

FAQ

Are evals enough to justify deploying an AI system?

No, and the strongest evaluation organizations say so themselves. Evals demonstrate what a system did under test conditions: they support inability arguments while capability is genuinely absent, and they inform, but do not constitute, readiness. A deployment decision also needs the questions evals do not answer: what is in the failures, whether the test envelope matches the deployment, what independent check exists at runtime, and when the evidence expires.

Why can a model pass an eval and fail in production?

Because the eval and the deployment differ in distribution, in elicitation, or in observation. The production case may sit in the untested residue; production prompting and tools may elicit capability or failure the eval never surfaced; and a model that distinguishes test from deployment contexts may behave differently across them, which is the evaluation-awareness problem. Each is a documented mechanism, and none requires anything exotic to occur.

Sources