now (after generations of trial and error), for bridges or planes—much higher, indeed, than we use
for approximately anything (this is one key reason to never, ever create such a virus). For example,
bridges need not be robust to nuclear attack; but the storage facility for such a virus should be—such
attacks just aren’t sufﬁciently unlikely.
We might view the threat of PS-misaligned behavior from sufﬁciently capable APS systems in similar
terms. And human track records of ensuring safety and security in our highest-stakes contexts—BSL-
4 labs, nuclear power plants, nuclear weapons facilities—seem very far from comforting.122
Is Power-Seeking AI an Existential Risk?