cleaned up, or trying to spread—and especially not with greater intelligence than the humans trying
to contain it.16 But the power-seeking agents just described would be trying, in sophisticated ways, to
undermine our efforts to stop them. If such agents are sufﬁciently capable, and/or if sufﬁciently many
of such failures occur, humans could end up permanently disempowered, relative to the power-seeking
systems we’ve created.
This difference—between the usual sort of damage that occurs when a piece of technology malfunc-
tions, and the type that occurs when you lose control over strategically sophisticated, power-seeking
agents whose objectives conﬂict with yours—marks a key distinction not just between worries about
AI vs. other sorts of risks, but also between the speciﬁc type of AI-related worry I focus on in what
follows, and the more inclusive set of worries that sometimes go under the heading of “AI alignment.”
Is Power-Seeking AI an Existential Risk?