PS-alignment is a problem, we will see and correct various forms of misaligned power-seeking
in comparatively weak (but still strategically aware) systems (indeed, we may well devote a lot
of the energy to trying to trigger tendencies towards misaligned power-seeking, but in contained
environments we’re conﬁdent we can control). Let’s call these “warning shots.”
Warning shots should get us very worried. If early, strategically-aware AI agents show tendencies to
try to e.g. lie to humans, break out of contained environments, get unauthorized access to resources,
manipulate reward channels, and so forth, this is important evidence about the degree of PS-alignment
the techniques used to develop such systems achieve—and plausibly, of the probability of signiﬁcant
PS-alignment problems more generally. Receiving such evidence should make us sit up straight.
Indeed, precisely because warning shots provide such tangible evidence, it seems preferable, other
Is Power-Seeking AI an Existential Risk?