55See Mikulik (2019) for related discussion of “2-D robustness.”
56Thus, it is physics-compatible for a randomly chosen bridge in Indiana to get hit, on a randomly chosen
millisecond in May 2050, by a nuclear bomb, a 10 km-diameter asteroid, and a lightning bolt all at once; but not
for the laws of physics to change, or for us all to be instantaneously transported to a galaxy far away. Obviously
the scope here is very broad: but note that misaligned behavior is a different standard than “bad” or even
“catastrophic” behavior. It will always be possible to set up physics-compatible inputs where a system makes a
mistake, or gets deceived, or acts in a way that results in catastrophic outcomes. To be misaligned, though, this
behavior needs to arise from problems with the system’s objectives in particular. Thus, for example, if Bob is a
paper-clip maximizer, and he builds Fred, who is also a paper-clip maximizer, Fred will (on my deﬁnition) be
Is Power-Seeking AI an Existential Risk?