Ensuring safety and reliability requires understanding a system well enough to predict its behavior.
But plausibly, this is uniquely challenging in the context of a strategically aware agentic planner
whose cognitive capabilities signiﬁcantly exceed those of humans in a given domain. That is, the
thinking and strategic decision-making of such an agent will likely reach a quite alien and opaque
level of sophistication, which humans may be poorly positioned to anticipate and understand at the
level required for ensuring PS-alignment. For example, it may consider many options humans never
would; it may understand physical and social dynamics that humans do not; and so forth.116
This issue seems especially salient in the current, machine-learning dominated AI paradigm, in which
our ability to create an AI system that can perform some task (e.g., predicting text) often far exceeds
our ability to understand how the system does what it does. We set various key high-level variables
Is Power-Seeking AI an Existential Risk?