information about that behavior, or by manipulating my preferences.88
This general problem—that optimizing for a proxy correlated with but not identical to some intended
outcome can break the correlation in question—is familiar from human contexts.89 And we already
see analogous problems in existing AI systems. Thus:
Is Power-Seeking AI an Existential Risk?