60Indeed, I’m skeptical that it will generally be well-deﬁned, even for agentic planners, what an arbitrarily
capable system pursuing that agent’s objectives looks like (for example, I’m skeptical that there’s a single version
of “omnipotent Joe”—and still less, “omnipotent MuZero,” “omnipotent Coca-Cola company,” “omnipotent
version of my mom’s dog” and so forth). If we want to talk about alignment properties that are robust to
improvements in capability, I think we should talk about what sorts of behavior will result from a particular
process for improving that system’s capabilities (e.g,. a particular type of retraining, a particular way of scaling
up its cognitive resources, and so forth). Thanks to Paul Christiano, Ben Garﬁnkel, Richard Ngo, and Rohin
Shah for discussion.
Is Power-Seeking AI an Existential Risk?