converge on167—are more complicated to evaluate.168 And we can imagine other routes to optimism
as well—related, for example, to hypotheses about the default consciousness, pleasure, preference
satisfaction, or partial alignment of the AI systems that disempowered humans.
I’m not going to dig in on this much. I do, though, want to reiterate that my concern here is with
the unintentional disempowerment of humanity. That is, sharing power with AI agents—especially
conscious and cooperative ones—may ultimately be the right path for humanity to take. But if so, we
want it to be a path we chose, on purpose, with full knowledge of what we were doing and why: we
don’t want to build AI agents who force such a path upon us, whether we like it or not.
I think the moral situation here is actually quite complex. Suitably sophisticated AI systems may be
moral patients; morally insensitive efforts to use, contain, train, and incentivize them risk serious
Is Power-Seeking AI an Existential Risk?