to any inputs compatible with basic physical conditions of our universe (I’ll call these “physics-
compatible” inputs).56 By “inputs,” I mean information the system receives via the channels intended
by its designers (an input to GPT-3, for example, would be a text prompt).57 I am not including
processes that intervene in some other way on the internal state of the system—for example, by
directly changing the weights in a neural network (analogy: a soldier’s loyalty need not withstand
arbitrary types of brain surgery).58
Sometimes, something like full alignment is framed by asking whether we would be OK with a
system’s objectives being pursued with ~arbitrary amounts of capability.59 This isn’t the concept I
Is Power-Seeking AI an Existential Risk?