if/when humans try to stop them. Hence, it’s much harder to see why humans would fail to notice,
contain, and correct the problem before it reaches an existential scale.
This, then, is the backdrop picture underlying the speciﬁc arguments I’ll examine: non-human agents,
much more intelligent than humans, would likely be potent forces in the world; and if their objectives
are problematic, some of them may, by default, seek to gain and maintain power. Building such
agents therefore seems like it could risk human disempowerment, at least in principle. Let’s look
more closely at whether to expect this in practice.
Is Power-Seeking AI an Existential Risk?