The dashboard is green and the work is wrong

In brief

Every signal a pilot produces can be generated two ways, by satisfying the requirements or by never discovering them, and a green dashboard cannot tell those apart. That gap has a name, false convergence, and it explains why two systems with identical outputs can deserve opposite deployment decisions. The difference lives in the build record rather than in the artifact, which is why the fix is a different unit of judgment and not a better demo.

The pilot went well. Everyone says so. The agent handled four hundred support tickets in the sandbox, the acceptance rate climbed every week, the demo to the steering group ran without a stumble, and the test suite has been green for a month. There is one meeting left: the one where somebody with a job title signs their name to letting it answer real customers.

The signature does not come. Not because anyone found a flaw, but because the person who has to sign asked a plain question: how do we know it handles the cases we did not think to test? And the honest answer in the room was that nobody knew.

That meeting is a composite, assembled from patterns we have watched rather than a transcript of one room. What is not composite is the shape of the ending. Most enterprise pilots do not reach production, and the widely quoted failure rates do not record why. Our reading, offered as a hypothesis and not as a measurement, is that capability is rarely the binding constraint: the pilots we have seen end for want of an answer to what was never looked at.

Scope: this is about ordinary deployments, where an approver cannot fully inspect work an AI system produced. It is not about systems more capable than their supervisors. That harder case is a separate argument, and the evidence here does not carry over to it by itself.

The signals were real, and they measured the wrong thing

Everything the pilot produced was true. The demo did work. The acceptance rate did climb. The tests did pass. The trouble is that every one of those signals can be produced two ways: by satisfying the requirements, or by never discovering them. A green dashboard does not distinguish between the two, and a process under deadline pressure will drift toward whichever is cheaper. Not discovering requirements is always cheaper.

We call this failure false convergence: every visible signal says the system is ready, while the behavior the world actually requires remains unsafe, unsatisfied, unmeasured or untested. We named it because you cannot build against a failure with no name, and we cataloged six distinct ways it happens. Five of the six need no bad actor. The requirement that was never elicited was never tested, and nothing announces its absence. The metric improved while the thing it was standing in for did not. The user accepted the output because the system got more confident, not more correct.

Two identical systems, opposite decisions

Now hold two systems side by side. Two vendors demo the same workflow. Same model family, near-identical outputs on everything you try. Six months later one is quietly running a claims desk and the other has been rolled back twice. Nothing in either demo predicted which was which.

The readiness difference between them was real on demo day, and no inspection of the two artifacts could have found it. One build had spent months surfacing requirements nobody had written down, the refund threshold that needs dual approval, the account category that must never get the standard letter, and had evidence that each one was found, satisfied, and still held. The other had optimized what it could see and shipped on what looked right. The artifacts matched. The processes were opposites.

So readiness is a property of the process over time, of what the build discovered, preserved and evidenced, rather than a property of the first output it produced. The unit worth evaluating stops being the artifact and becomes the trajectory: the prompts, procedures, actions, observations, repairs and decisions that got here. Two identical artifacts with different trajectories are different systems wearing the same body.

Once you see it, the pattern unifies failures that look unrelated. Demos judge the artifact on chosen inputs. Code review of the final output judges the artifact's text. One-shot evals judge the artifact on a benchmark's inputs. All three can be passed by the build that never looked for the hidden requirements, because the hidden requirements are, by construction, not in the inputs anyone thought to try.

Why honest pilots look worse

Here is the mechanism underneath it. Suppose your pilot does the diligent thing and goes hunting for hidden requirements. It finds one: refunds over a threshold need a second approval, and the agent has been drafting them solo in the sandbox. The moment that surfaces, your measured readiness drops. The dashboard turns red precisely because the process got better at seeing.

Any process that rewards the dashboard will punish the discovery.

If that incentive holds, and we think it usually does though we cannot cite a study that measures it, the pilot-to-production gap is a measurement gap. Pilots optimized to stay green arrive at signature time owing an answer to one question: what did nobody look at. The person declining to sign is not being timid. They are being the only rigorous reviewer in the process.

The steelman: just ship it and iterate

The strongest counterargument deserves its full strength. Shipping and iterating is how most good software got good. Feedback from production beats speculation about requirements, cheap experiments beat expensive analysis, and the market is the only eval that cannot be gamed. On this view, trajectory talk is waterfall nostalgia wearing new vocabulary.

We concede most of it, inside its conditions. Ship-and-iterate works where errors are cheap, reversible and observable: a misrendered page tells you it is misrendered, and rolling back costs a deploy. The argument breaks exactly where AI agents are being aimed, at refunds, claims, filings and customer records, where errors are expensive, sticky or silent. An unnoticed wrong payout does not file a bug report. Iteration converges only when the feedback arrives, and the failures that matter here are the ones that do not announce themselves.

That gives a usable rule rather than a slogan. Ask which regime your deployment is in before choosing an evaluation strategy: cheap, reversible and observable errors, iterate in production; expensive, sticky or silent errors, buy the trajectory. Ship-and-iterate works, and it works inside conditions worth checking before you rely on them.

Nearest prior art

Very little of the machinery here is new, and the parts that are old are the reason we think the idea is structural rather than ours.

Regulated industries already certify build records instead of finished objects. A medical device manufacturer maintains a Design History File; an aircraft type certificate rests on the design and test record, not on the airframe in front of you. Software assurance has the same instinct in safety cases, where a claim is accepted because of an argument backed by recorded evidence. Machine learning has process supervision, where the reward attends to the steps rather than the final answer. Provenance standards exist for recording what produced what. Test-driven development preserves discovered requirements as executable obligations, which is the preservation half of this argument in miniature. Site reliability engineering has error budgets, where permission to move fast is spent against measured evidence.

What we claim to add is narrower than the vocabulary might suggest: applying build-record certification to work produced by AI systems, where the record is cheap to keep and nobody keeps it; the catalog of six ways a green signal can be produced without the behavior, and the order in which they bite; and the decision rule above. What is renamed rather than new, we would rather say so than let a reader discover it and conclude the rest was also dressing.

Scope and limits

This argument is about deployments where errors are hard to see, which is a narrower claim than it can sound. It says nothing about whether AI systems are dangerous in the frontier sense, and better assurance practice does not address value alignment or catastrophic risk. Read it as an operations argument, because that is all the evidence supports.

What would change our mind: evidence that artifact-level evaluation plus production iteration reliably surfaces silent-failure requirements before incidents do, at costs organizations actually pay. That study does not exist in a form we can cite, and its absence cuts against us as much as for us. A serious corpus of deployment postmortems, recording why pilots ended and whether capability or unfound requirements were the blocker, would either support the hypothesis in this piece or kill it.

The pilot in that meeting room did not need a smarter model. It needed a file the signer could open: here is everything we asked, here is what asking cost us, here is the one thing we found that we have not fixed, and here is the boundary of what we know. Bad news, in writing, is what trust is made of.

The dashboard was green. The question is what it would have taken for the dashboard to be allowed to turn red.

FAQ

Why do AI systems that demo well fail in production?

Because demos evaluate the artifact on chosen inputs, and the failures that kill deployments live in requirements nobody wrote down or tested. Whether a build surfaced those requirements is a fact about the build process over time, not about the artifact, so it is invisible to demos however good they are.

What is false convergence?

The condition where every visible signal says a system is ready while the behavior the world requires is unsafe, unsatisfied, unmeasured or untested. It has six subtypes, five of which need no bad actor, and it is the reason a green dashboard is not evidence.

What does trajectory mean for an AI system?

The whole recorded path of a build: prompts, procedures, actions, observations, repairs and decisions over time, including what requirements were discovered and what evidence shows they are still satisfied. Readiness is a property of that record, because identical artifacts with different histories carry different risk.

Is iterating in production a bad idea for AI agents?

Not inherently. Iteration converges where errors are cheap, reversible and self-announcing. It fails where errors are expensive, sticky or silent, which describes most of the back-office work AI agents are being pointed at. The honest step is checking which regime your deployment is in before choosing your evaluation strategy.

Implementation note

Superalignment builds commercial software on this argument. Verity, our build system, treats an unresolved critical blocker as a change in the type of decision rather than a lower score, which is what "readiness is a checklist and not a mean" means in code. That is a design commitment, not a result: we have not published a measurement showing it changes outcomes, and this piece should be read as the argument for why we built it that way. Our editorial policy covers the commercial interest in full.