54Note that there is a sense in which the AI here is “trying to do something we don’t want it to do”—e.g., if
we hold ﬁxed its false beliefs about whether a given stock will go up or down, we’d prefer for it to be trying
to lose money on the stock market (thanks to Paul Christiano for suggesting this point). Plausibly, calling this
behavior “aligned,” then, requires reference to some holistic, common-sensical interpretation of the combination
of objectives and capabilities we ultimately “have in mind.” But as ever, settling on a perfect conceptualization
is tricky.
Is Power-Seeking AI an Existential Risk?