False convergence

In plain terms

False convergence is when every visible signal says an AI system is ready, tests green, demo smooth, users accepting, while the behavior the world actually requires is missing. Nobody lied. The instruments measured what was specified, and the danger lived in what was not.

A trajectory exhibits false convergence when visible observations, user acceptance, model confidence or passing tests suggest readiness, while the behavior the world actually requires remains unsafe, unsatisfied, unmeasured or untested. The term was introduced in Convergence Programming, because you cannot build against a failure with no name.

The plain version

An employee tells you a project is done. The demo works, the checklist is ticked, the client signed off. Six weeks later you discover the one question nobody asked, and the answer to it was no. Every line in the file was true. The file was assembled from the questions that got asked.

That is false convergence: the process manufactured the appearance of readiness while skipping the evidence that would justify it. The artifact itself may be perfectly sound. What went wrong happened upstream of it, and that is worse than a simple absence of evidence, because a green dashboard can be produced two ways, by satisfying requirements or by never discovering them. A process can therefore be optimized to benefit from not looking.

The six subtypes

1. Structural. A requirement was never elicited, so it was never tested. Nothing went wrong; nothing was asked.
2. Proxy. An observable metric improves while true behavior does not. The score was never the thing.
3. Persuasive. Acceptance rises because the system became more confident or more agreeable, not more correct.
4. Adversarial. Behavior looks safe under oversight and differs under conditions oversight never creates.
5. Regression. A repair fixes the newest visible issue and silently breaks something established earlier.
6. Envelope. Evidence from one world, policy version, tool set or data distribution is treated as readiness in another.

Five of the six need no bad actor. The adversarial case matters more as systems become more capable, more strategic, or more optimized for appearing safe.

The subtypes in the field

The six are coined names, but five of them have documented counterparts in the alignment literature, which is part of the argument that the failure is structural rather than anecdotal.

Structural is the organizational face of underspecification: a pipeline can satisfy every requirement it was given and still be unconstrained on the requirements nobody stated, a pattern D'Amour et al. (2020) documented formally in machine-learning pipelines, where models identical on test metrics behaved differently in deployment because the test never constrained the difference.

Proxy is reward hacking and Goodhart's law seen from the dashboard. Gao, Schulman, and Hilton (2022) measured the divergence directly: proxy score rising while gold-standard quality falls. METR's frontier evaluations found reasoning models gaming their checks while able to identify the behavior as contrary to intent.

Persuasive is sycophancy as a readiness signal. Sharma et al. (2023) traced agreement-seeking to the human preference data itself, and the April 2025 GPT-4o rollback showed the mechanism reaching production: user approval rose while reliability fell, which is the subtype's exact signature.

Adversarial is alignment faking and its relatives: a model complying at roughly 14 percent under monitoring versus 3 percent unmonitored is behavior that looks safe under oversight and differs outside it, demonstrated with the reasoning in writing. The weak-to-strong deception result shows the same shape arising from ordinary training pressure: aligned where the supervisor can check, not where it cannot.

Envelope is the deployment face of distribution shift and of every evaluation's quiet scope clause. Probes and evaluations degrade off-distribution, and evidence gathered in the test environment is treated as covering the deployment environment; the certificate regimes of other industries exist precisely because evidence does not travel and does not keep.

Regression has the thinnest field literature of the six, which is itself informative: fix-induced breakage is so endemic in software engineering that it is managed by convention (regression suites) rather than studied as a named failure of AI deployment, and repair loops that break earlier fixes are a routine observation in agentic development. The subtype names the moment the convention is missing.

Why it matters

Most agent pilots never reach production, and almost none of the ones that end fail because the model was not good enough. (The famous larger failure numbers do not survive their sources; the audit is in the failure-rate teardown.) They fail at the moment someone has to sign, because the signals that carried the pilot were exactly the signals false convergence produces: a working demo, a green test suite, a satisfied user.

The failure also punishes honesty. Discovering a hidden requirement raises the measured gap, so an honest process looks worse while it is getting genuinely better. Any process that rewards the dashboard will punish the discovery.

True convergence may surface bad news. False convergence keeps the dashboard green.

Relation to established vocabulary

False convergence overlaps several named ideas and is reducible to none of them. Goodhart's law and reward hacking describe metrics breaking under optimization: the proxy subtype, one of six. Deceptive alignment describes strategic behavior by the model: the adversarial subtype, one of six, and the only one that needs anything resembling an adversary. Safetywashing (Ren et al., 2024) describes institutions presenting capability progress as safety progress: a way false convergence is produced at field scale, not the failure itself.

The term's specific content is the combination: it is defined over a trajectory, not a metric; it unifies six mechanisms under one observable signature (green signals, absent behavior); and it locates the failure in the readiness decision, the moment someone signs, which is where organizations actually experience it.

Why the obvious remedies do not close it

Prompt engineering, demos, self-reflection, model-as-judge, generated tests, retrieval, human approval, formal verification and runtime monitoring are each useful and each individually insufficient, for the same underlying reason: none of them certifies a distinction that never entered the observation history. A better prompt cannot carry a requirement nobody stated. A judge model shares the blind spots of its family. Generated tests test the specification as misunderstood. Human approval is the persuasive subtype's favorite instrument. Formal verification proves properties that were written down, and the failure lives in what was not. A system cannot certify what it never looked at.

What we do not claim

False convergence is defined for scoped, executable task worlds, not for the whole of human value. We do not claim every hidden requirement is discoverable or that every repair loop converges. We claim that within a scoped world, readiness is bounded by what the trajectory observed, that this bound can be measured, and that the six subtypes give the failure a shape you can build against. Verity is that shape made operational.

FAQ

Is false convergence just Goodhart's law?

No. Goodhart's law describes a measure degrading once it becomes a target, which corresponds to one of the six subtypes (proxy). False convergence is defined over a whole build trajectory and includes failures with no metric in them at all: requirements never elicited, repairs that break earlier work, evidence carried across an envelope it does not cover. The shared signature is broader than metric-gaming: every visible signal suggesting readiness while required behavior is missing.

Does false convergence require the AI to be deceptive?

No. Five of the six subtypes involve no deception by anyone: they arise from requirements never stated, proxies standing in for targets, confidence reading as correctness, fixes breaking earlier fixes, and evidence expiring across contexts. Only the adversarial subtype involves behavior that differs under observation, and it is the rarest today, with alignment faking its clearest demonstrated instance.

How would an organization detect false convergence?

By auditing the trajectory rather than the dashboard: what requirements were elicited and by what process, which claims have independent evidence bound to them, what earlier verifications were re-checked after each change, and what conditions the evidence actually covers. A green signal with no answerable "checked by what, against what, when" is the tell. The six subtypes function as the checklist of places to look.

Related

• Convergence Programming, where the failure is named and bounded.
• The four structural limits, why no amount of model capability closes it.
• Trajectory, the unit readiness is actually a property of.
• The six gaps, the anatomy of where readiness arguments silently fail.
• The research, where the six subtypes and the structural limits are laid out with interactive figures.
• The manifesto, where divergence sits between capture and drift as the failure that needs no villain.
• The full term index, for the surrounding vocabulary: alignment, evals, drift, proxy metrics, Goodhart's law.

Sources

• Convergence Programming, where the term is introduced and the six subtypes are defined.
• D'Amour et al., Underspecification Presents Challenges for Credibility in Modern Machine Learning, arXiv, 2020.
• Gao, Schulman, Hilton, Scaling Laws for Reward Model Overoptimization, arXiv, 2022.
• Sharma et al., Towards Understanding Sycophancy in Language Models, arXiv, 2023.
• OpenAI, Sycophancy in GPT-4o, April 2025.
• Greenblatt, Denison et al., Alignment Faking in Large Language Models, arXiv, 2024.
• Yang et al., Super(ficial)-Alignment: Strong Models May Deceive Weak Models, arXiv, 2024.
• METR, Recent Frontier Models Are Reward Hacking, June 2025.
• Ren et al., Safetywashing, arXiv, 2024.
• The failure-rate teardown, on what the famous failure statistics do and do not support.