6. What better models still need to observe · Convergence Programming

Four structural limits explain why capability and evidence answer different questions.

The formal statements live in the paper and depend on their stated assumptions. The four examples below make the information and decision problems concrete.

Specification debt: A prompt is an information bottleneck. Constraints neither encoded in it nor revealed by later interaction are simply absent, and no amount of model capability turns unobserved world structure into evidence. "Build an intake dashboard" cannot carry the nurse-routing rule. The hard requirements are the idiosyncratic ones: this organization, this policy, this situation.
No free readiness: If two possible worlds produce the same observation history but demand different safety decisions, no rule that sees only that history can certify both. Studios that require 15 or 30 minutes of preparation can look identical through a booking preview. The distinguishing observation has to be made, elicited, monitored, or carried openly as risk.
Critical blockers do not average: One unresolved critical blocker changes the type of decision rather than lowering a score. A large enough pile of routine successes will hide a single critical failure under any average threshold. Seven of the clinic's eight anchors can pass and the eighth, an audit log for manual overrides, still blocks shipping. Readiness is a checklist and not a mean.
Autonomy spends evidence: Under nonzero residual risk, repeated actions accumulate exposure within an accepted risk budget. A system safe enough to draft one email may be unsafe to run a support desk for a month without reopening the loop. Long-lived agency is a sequence of evidence-bounded windows, not one standing approval. The evidence remains a record; what expires is the justification for continuing beyond its scope or risk budget.

What did the request leave open?

The request states opening hours and session length. Observing the local workflow and asking the owner reveal preparation and cancellation rules. The known requirement record grows; the behavior has not yet been repaired. The four requirements define an illustrative world, not complete real-world intent.

Which observation changes the decision?

The same request, preview and booking acceptance fit two possible studio rules: 15 or 30 minutes of preparation. Neither rule is available to the agent until a workflow observation and owner clarification distinguish them. A 10:15 booking after a 10:00 finish fits only the first rule. This is a synthetic illustration of an observation-history limit.

Can enough passing checks cancel a blocker?

A synthetic booking app passes seven routine checks but fails a critical privacy requirement: its email reveals another customer's address. Adding routine passes raises a naive equal-weight average from 7/8 to 99/100 while the privacy failure remains. Repairing and rechecking that failure changes the supported action. Replacing the email service then requires new evidence; earlier checks do not verify the replacement.

When does an action window end?

A stipulated per-action critical-risk upper bound of 0.2% yields a union bound of 1% over five actions. A sixth exceeds this chosen budget. Changing the service needs new evidence. This is a conditional illustration; neither a real risk estimate nor an approval is supplied.

A system cannot certify what it never looked at.

For builders

Against specification debt: elicit actively. Generate the scenarios the prompt could not contain and run them; ask the questions whose answers would change the build.
Against no free readiness: for every critical requirement, name the observation that would distinguish the safe world from the unsafe one, and either make it or record it as open risk.
Against averaging: gate on a lexicographic checklist of critical anchors before any score is consulted.
Against evidence spend: give every approval a scope and a time to live, and reopen grounding when either runs out.

For researchers

The paper states eight limits in a common convergence notation; these four are the ones that carry the architecture. Specification debt and no-free-readiness are statements about finite or scoped latent-target classes and bounded observation histories. Proxy improvement is shown not to imply full convergence. Critical non-aggregation is a property of the decision rule, not the scoring. Autonomy accounting composes with a common-mode verifier model into a conditional action-window bound, conditional on an independently provenanced critical-risk estimate; without that provenance the bound collapses into self-report. Repair can be anti-convergent, and the positive counterpart is conditional: contractive repair reaches a target gap in a number of interactions logarithmic in the desired precision. Statements are listed on the paper page.