AI vs. AGI vs. Consciousness vs. Super-Intelligence vs. Agency
&gt; Almost all AGI doomsayers assume AGI will have agency. They have this vision of the machine deciding it’s time to end civilization.<p>No. Agency is not a necessary condition for AI to do massive damage. I don&#x27;t believe agency is really well-defined either.<p>An AI merely needs to be hooked up to enough physical systems, have sufficiently complex reaction mechanisms, and some way of looping to do a lot of damage. For the first everyone seems to be rushing as fast as possible to hook up <i>everything they possibly can</i> to AI. For the second, we&#x27;re already seeing AI do all sorts of things we didn&#x27;t expect it to do.<p>And for the third, again everyone seems eager to create looping&#x2F;recursive structures for AIs as soon as possible.<p>Once you have all of this, all it takes a cascade of sufficiently inscrutable and damaging reactions from the AI to do serious harm.<p>See e.g. <a href="https:&#x2F;&#x2F;www.lesswrong.com&#x2F;posts&#x2F;kpPnReyBC54KESiSn&#x2F;optimality-is-the-tiger-and-agents-are-its-teeth" rel="nofollow">https:&#x2F;&#x2F;www.lesswrong.com&#x2F;posts&#x2F;kpPnReyBC54KESiSn&#x2F;optimality...</a>