standards, and so forth. And we can imagine cases in which sufﬁciently scary and/or high-proﬁle
failures trigger extreme and globally-coordinated responses (for example, large-scale bans on certain
automated systems) that would seem out of the question in less dire circumstances.
Indeed, humans have some track record of eschewing and/or coordinating to avoid/limit technologies
that a naive incentives analysis might’ve predicted we’d pursue more vigorously. Thus, for example,
various high-proﬁle nuclear accidents contributed to signiﬁcant (indeed, plausibly net-harmful)
reductions in the use of nuclear power in countries like the US; I expect that in 1950, I would have
predicted greater proliferation and use of nuclear weapons by 2021 than we’ve in fact seen; and we
eschew human cloning, and certain types of human genetic engineering, centrally for ethical rather
than technological reasons.162 Corrective feedback loops in response to PS-alignment problems might
Is Power-Seeking AI an Existential Risk?