The Myth of a Superhuman AI
I am not believer in superintelligence, but for a different reason than author. I assume the following about superintelligence:<p>- It somehow needs to be distributed, that is, composed of smaller computing parts, because there is a physical limit what you can do in unit of space.<p>- It needs to change to adapt to environment (learn), and so all the parts need to potentially change.<p>From this follows that the parts will be subject to evolution, even if they don&#x27;t reproduce. And so the existence of the parts will depend on their survival. This, in my opinion, inevitably leads to evolution of parts that are &quot;interested&quot; in their own survival, at the expense of the &quot;superintelligent&quot; whole. And it leads to conflict, which can eventually eat up all the improvements in the intelligence.<p>Look at humans. Humanity (or biosphere in general) didn&#x27;t become a superintelligent whole, capable of following some single unified goal. Instead, we became fighting factions of different units, and most of the actual intelligence is spent on arms races.<p>Anyhow, even if superintelligence is possible, I believe the problem of friendly AGI has a simple solution. We simply need to make sure that the AGI doesn&#x27;t optimize anything, but instead takes the saying &quot;all things in moderation&quot; to its heart. That means, every once in a while, AGI should stop whatever goals it pursues and reflect on purpose of those goals, if it is not, by some measure, going too far.<p>You can argue that we don&#x27;t actually know how to make AI to stop and think. I would respond, AI that cannot do that, and only pursues some pre-programmed optimum mindlessly, is not really general.