more).
A second objection (in possible tension with the ﬁrst) is: humans (or, some humans) may be power-
seeking, but this is a product of a speciﬁc evolutionary history (namely, one in which power-seeking
was often directly selected for), which AI systems will not share.82 Some versions of this objection
simply neglect to address the instrumental convergence argument above83 (and note, regardless, that
some proposed ways of training AI systems resemble evolution in various respects).84 But we can see
stronger versions as pointing at a possibility similar to the one I mentioned above: namely, perhaps it
just isn’t that hard to train APS systems not to seek power in unintended ways, across a large enough
range of inputs, if you’re actively trying to do so (evolution wasn’t). Again, I discuss possibilities and
difﬁculties in this regard in the next section.
Ultimately, I’m sympathetic to the idea that we should expect, by default, to see incentives towards
Is Power-Seeking AI an Existential Risk?