Can every component work while the system becomes unsafe?

A system can lose safety without a component failing in isolation. Reliable parts can interact under an unsafe command, an obsolete model, missing feedback, or a constraint that nobody owns. Component reliability remains useful, but Leveson's framework expands the causal question to how safety constraints are enforced across technical, human, management, and regulatory control loops over time.

Nancy G. Leveson · MIT Press · January 13, 2012 prototype 5 min read Explained by Superalignment Research

Read the original

Topics: safety science, systems theory, control theory, organizational design, safety cases, assurance, governance

The 60-second answer

AI services combine models, tools, people, software, deployment policy, incentives, and external oversight. Testing each part separately can miss an unsafe interaction or a control loop whose feedback no longer represents the deployed process.

Safety is an emergent system property enforced through constraints, not the same property as component reliability. STAMP models a sociotechnical system as nested control loops in which controllers issue actions, receive feedback, and act through process models. Accidents can follow from missing or unsafe control actions, bad timing, coordination failures, inadequate feedback, or process models that do not match reality. STPA uses this model prospectively, while CAST uses it to analyze losses retrospectively.

Safety is an emergent constraint-control problem and cannot be inferred from component reliability alone.
A controller can create danger through an unsafe action, omission, timing error, or stale process model without a simple component failure.
STAMP is the causality framework, STPA is prospective analysis, and CAST is retrospective analysis, none of which is self-validating.

Written for: Technical generalists building or governing complex software and sociotechnical systems. Useful prerequisites: Feedback control, The difference between component reliability and system safety, Basic hazard analysis.

The question: How should engineers analyze accidents that emerge from unsafe interactions, control, and organizational decisions even when no component simply breaks?
What the authors did: Leveson develops a systems-theoretic accident model called STAMP, then derives methods for prospective hazard analysis, safety-guided design, operations, and retrospective accident analysis. The book combines conceptual argument, control diagrams, worked examples, historical cases, and method demonstrations across technical and sociotechnical systems.
The source: Engineering a Safer World: Systems Thinking Applied to Safety

Can an operational controller issue an unsafe action?

Model matches. Feedback says the door is open, so the controller commands power off.

Process-model state and unsafe control categories

Condition	Controller sees	Possible result
Model matches	Relevant process state through usable feedback	Control can enforce the stated constraint.
Model stale	An earlier or incorrect process state	An unsafe action can be provided even while components operate.
Feedback missing	No update about a material state change	The model can stay stale and support the wrong action.
Four source categories	Action provided, omitted, mistimed, or applied for the wrong duration	Each can violate a safety constraint in context.

The switch reconstructs Leveson's process-model mechanism. With accurate feedback, the controller's model matches the process. Missing or delayed feedback can make the model stale, so the controller issues an unsafe action even though the controller and process still operate. This does not imply that component failure is irrelevant. Box sizes, positions, colors, arrow thickness, and state labels are illustrative. They do not encode likelihood, severity, organizational authority, delay duration, reliability, or empirical effect.

Walk through the argument

Each step below points to the source location that carries it. The source map follows the walkthrough for source-level checking.

Separate safety from reliability

Reliability asks whether components behave as specified. Safety asks whether the system avoids defined losses. A perfectly reliable component can still execute a hazardous command, and a failed component can be contained by an effective safety control.

Leveson does not discard reliability. She argues that it is one contributor inside a wider causal model that must also cover interactions, software, humans, organizations, and change.

Source: Author manuscript PDF pages 15 to 18, Chapter 1, Why Do We Need Something Different?, Author manuscript PDF pages 19 to 62, Chapter 2

Model safety as constraint enforcement

STAMP begins with losses and hazards, then identifies safety constraints that must hold. A hierarchy of technical and organizational controllers is responsible for enforcing those constraints through control actions and feedback.

The hierarchy can include software, operators, management, regulators, and legislators. Control here does not require rigid command. It means that responsibilities, authority, actions, and feedback form an effective loop.

Source: Author manuscript PDF pages 63 to 72, Chapter 3, Author manuscript PDF pages 77 to 100, Chapter 4, especially Sections 4.1 to 4.4 and Figures 4.4 to 4.8

Put a process model inside every controller

A controller needs a model of the process it controls. A thermostat's model is tiny. An operator or manager may need a far richer account of state, transitions, constraints, and downstream effects.

If feedback is missing, delayed, or misleading, the internal model can diverge from reality. The controller may then omit a required action, issue an unsafe one, act too early or late, or apply control for the wrong duration.

Source: Author manuscript PDF pages 87 to 89, Section 4.3 and Figure 4.6, Author manuscript PDF pages 191 to 224, Chapter 8, especially Sections 8.1 to 8.4

Use STPA before the loss

STPA first identifies control actions that could be unsafe in context. It then examines how each action could occur by tracing process-model flaws, feedback, control logic, coordination, and degradation over time.

The method can begin before a detailed design exists and guide design around constraints. Its output depends on the analyst's system boundary, hazard definitions, control structure, and domain knowledge.

Source: Author manuscript PDF pages 191 to 224, Chapter 8, especially Sections 8.1 to 8.4

Use CAST after the loss

CAST documents the accident as a dynamic process across the sociotechnical control structure. It asks which constraints were not enforced at each level, what information and models shaped decisions, and how coordination and controls weakened over time.

The goal is learning rather than finding one root cause or stopping at the nearest human error. CAST is an analysis method, not an accident investigation procedure that gathers all needed evidence by itself.

Source: Author manuscript PDF pages 307 to 341, Chapter 11, especially Sections 11.1 to 11.5 and 11.10 to 11.12

Keep the control structure alive

A safety analysis can age out as software, organizations, incentives, and environments change. Later chapters connect design artifacts to operations, management, culture, audits, and incident learning so that constraints remain enforced over a system's life.

For AI, that suggests testing whether deployment feedback updates the right process models and whether responsibility for system-level constraints survives organizational change. It does not make STAMP an automatic safety case.

Source: Author manuscript PDF pages 191 to 224, Chapter 8, especially Sections 8.1 to 8.4, Author manuscript Chapters 12 to 14, beginning at PDF pages 342, 363, and 386

Demand comparative evidence

The book explains a broad framework through examples and applications. That supports understanding and use, but it does not yield a common effect size or prove superiority over every modern alternative.

A strong evaluation would compare hazard coverage, analyst effort, reproducibility, decision impact, and prevented losses under clearly defined conditions.

Source: Author manuscript PDF pages 19 to 62, Chapter 2, Author manuscript PDF pages 191 to 224, Chapter 8, especially Sections 8.1 to 8.4, Author manuscript PDF pages 307 to 341, Chapter 11, especially Sections 11.1 to 11.5 and 11.10 to 11.12

Source map

These are the source locations that carry the argument. Use them to check this explanation against the original rather than trusting the summary alone.

Where the argument lives

Locus	Why it matters	Source
Author manuscript PDF pages 15 to 18, Chapter 1, Why Do We Need Something Different?	Motivates a new safety model for increasing complexity, software, sociotechnical coupling, change, and limits of adding more local reliability.	Open source →
Author manuscript PDF pages 19 to 62, Chapter 2	Separates safety from reliability and critiques simple event chains, single causes, blame, static models, and treatments of software and human error.	Open source →
Author manuscript PDF pages 63 to 72, Chapter 3	Introduces emergence, hierarchy, communication, control, requisite control conditions, and process models as the systems-theory foundation.	Open source →
Author manuscript PDF pages 77 to 100, Chapter 4, especially Sections 4.1 to 4.4 and Figures 4.4 to 4.8	Defines safety constraints, hierarchical safety control structures, process models, and the STAMP account of inadequate control.	Open source →
Author manuscript PDF pages 87 to 89, Section 4.3 and Figure 4.6	Shows how a mismatch between a controller's process model and the controlled process can support an unsafe command even when the controller and components operate.	Open source →
Author manuscript PDF pages 191 to 224, Chapter 8, especially Sections 8.1 to 8.4	Defines STPA's goals and two main steps for identifying unsafe control actions and the scenarios that can produce them before a loss occurs.	Open source →
Author manuscript PDF pages 307 to 341, Chapter 11, especially Sections 11.1 to 11.5 and 11.10 to 11.12	Defines CAST as retrospective analysis of a dynamic accident process across constraints and control levels rather than a search for one blameworthy cause.	Open source →
Author manuscript Chapters 12 to 14, beginning at PDF pages 342, 363, and 386	Extends the framework to operations, change, safety management, culture, and the SUBSAFE case rather than stopping at design-time analysis.	Open source →

The Assumption Switch

One result. One assumption exposed. Turn it and see what changes.

Assumption under test

A controller's process model matches the relevant state of the controlled process and feedback updates it in time.

Held in the source: The controller can choose and time a control action that enforces the applicable safety constraint.
Turn it: Feedback is missing, delayed, or misleading, so a human or automated controller acts on a process model that no longer matches reality.
What changes: The controller can issue an unsafe command, omit a required action, act at the wrong time, or stop too soon even when its components remain operational.

The common misreading

STAMP is not a claim that components never fail or that reliability engineering should be discarded. It says failure chains are not a complete accident model for complex sociotechnical systems. STAMP is the causality framework, STPA is a prospective hazard-analysis method, and CAST is a retrospective accident-analysis method. Treating those names as synonyms hides what each one does.

Outside the ML frame

AI system safety and governance

Which safety constraint can fail even when every model component passes its local test?

The framework suggests mapping the AI system, operators, deployment controls, feedback, management, and external governance as one safety-control structure. It asks who owns each constraint and what process model guides their action. This is a transfer of systems-safety concepts, not evidence that applying STPA alone establishes AI safety.

Where the result stops

The book is a comprehensive framework and methods text, not one comparative trial proving that STAMP, STPA, or CAST outperforms every alternative. Its examples vary from illustrations to detailed accident analyses and applications, so they should not be pooled as a common effect estimate. Results depend on how analysts define hazards, boundaries, control structures, process models, and constraints. The loci use the free author manuscript, whose PDF pagination differs from the published MIT Press edition.

What remains open

Which comparative studies can distinguish the coverage and practical value of STPA from other modern hazard-analysis methods?
How should analysts represent learned components whose behavior and operating envelope change after deployment?
What evidence shows that a documented safety control structure remains effective as incentives, software, organizations, and environments change?
How can process-model assumptions be tested when important system state is latent or strategically reported?
Who should define and review system-level losses and constraints when affected groups disagree about acceptable risk?

What this bears on

Superalignment maintains a public register of the claims it makes and the evidence that would change its mind. The entries below connect this source to the exact public claims it bears on.

C4. Behavioral evaluation cannot carry a deployment decision alone. This record bears on it, indirectly. Leveson argues that safe component behavior and reliability evidence do not establish system safety because unsafe interactions and inadequate control can create loss. The book does not study AI behavioral evaluations or deployment decisions directly. See the claim and what would change our mind →

Source audit and review status

What was checked, and against what

Field	Result	Checked	By	Against
`title`	exact	2026-08-17	codex-primary-source-review	source
`authors`	exact	2026-08-17	codex-primary-source-review	source
`date`	exact	2026-08-17	codex-primary-source-review	source
`venue`	exact	2026-08-17	codex-primary-source-review	source
`full_text`	minor variant	2026-08-17	codex-primary-source-review	source

Review status prototype.
Program collection seminal v1; theory; wave 4, release slot unassigned.
Source access publisher full text, Open-access book PDF. Open the reading copy →
Provider-authored safety claim no.
Explained by Superalignment Research.
Reviewed by No named human reviewer yet.
Explainer dates created 2026-08-17; updated 2026-08-17.
AI assistance AI assisted with primary-source retrieval, full-manuscript extraction, manifestation checking, locus mapping, first-pass prose, figure design, and implementation. The page remains a prototype until a named human review is recorded.
Rights and access MIT Press publishes the complete book as open access under CC BY-NC-ND 4.0. The free author manuscript was read for pagination and locus checks; it is a distinct manifestation and should not be presented as the typeset publisher edition.
Corrections Read the correction policy or report an error.

Provenance

First seen 2026-08-17, via cross-disciplinary seminal-source survey and full-source review.
Work id work:leveson-engineering-safer-world, which groups manifestations of the same intellectual work.
Record id doi:10.7551/mitpress/8179.001.0001, the natural key for this catalog manifestation.
2026-08-17 full author manuscript read, publisher edition and license verified, and implementation-ready Explained prototype prepared

Full audit data, including this record under id doi:10.7551/mitpress/8179.001.0001: /library/records.jsonl. Compact browser index: /library/corpus.json. Catalog method and counts: /library/index.json.