or deploy, a strategically-aware AI agent that will end up seeking power in unintended ways?
It’s an important question, and one I’ll look at in some detail. In particular, I think these considerations
suggest that we should be less worried about practically PS-misaligned agents that are so unreliably
well-behaved (at least externally) that they aren’t useful, and more worried about practically PS-
misaligned agents whose abilities (including their abilities to behave in the ways we want, when it’s
useful for them to do so) make them at least superﬁcially attractive to use/deploy—because of e.g.
the proﬁt, social beneﬁt, and/or strategic advantage that using/deploying them affords, or appears to
afford. My central worry is that it will be substantially easier to build that type of agent than it will
be to build agents that are genuinely practically PS-aligned—and that the beliefs and incentives of
Is Power-Seeking AI an Existential Risk?