Superalignment

Borrowed from security, and the single most useful question to ask about any automation. Not how likely is it to be wrong, but what happens when it is. A system that drafts is nearly harmless. A system that sends, pays, deletes or promises has a radius, and it should be sized deliberately rather than discovered.

Why it matters

The one question that reliably separates a safe pilot from an expensive one.