119And deﬁning these distinctions may prove challenging.
120Bostrom (2014) calls this a “treacherous turn.”
121Muelhauser (2021) describes a rudimentary form of this behavior occurring in simulated evolution ex-
periments: “As part of a project studying the evolution of (simulated) organisms, computer scientist Charles
Ofria wanted to limit the replication rate of a digital organism. So, he programmed the system to pause after
each mutation, measure the mutant’s replication rate in an isolated test environment, and delete the mutant if
it replicated faster than its parent. However, the organisms evolved to recognize when they were in the test
environment and ‘play dead’ (pause replication) so they would not be eliminated and instead be kept in the
population where they could continue to replicate outside the test environment. Once he discovered this, Ofria
randomized the inputs of the test environment so that it couldn’t be so easily detected, but the organisms evolved
Is Power-Seeking AI an Existential Risk?