The impossibility of intelligence explosion
<i>&quot;An individual brain cannot implement recursive intelligence augmentation.  An overwhelming amount of evidence points to this simple fact: a single human brain, on its own, is not capable of designing a greater intelligence than itself. This is a purely empirical statement: out of billions of human brains that have come and gone, none has done so. Clearly, the intelligence of a single human, over a single lifetime, cannot design intelligence, or else, over billions of trials, it would have already occurred.&quot;</i><p>The same argument could have been about human flight (or any other invention).  Billions of human beings over millions of years had come and gone, and yet none had been able to fly.  Until they did.<p>There&#x27;s also a point to be made about humans not having adequate tools for introspection or self-modification.  We can not simply look in our brains&#x2F;minds and read our source code, nor easily tweak it to see what would happen without great risks to our lives.  A computer could.  Furthermore, a computer could potentially run billions or trillions such experiments in the course of a single human lifetime.<p><i>&quot;no human, nor any intelligent entity that we know of, has ever developed anything smarter than itself.&quot;</i><p>What about humans simply having smarter children?  In many ways, humanity&#x27;s creation of AI could be seen as analogous to giving birth to a child that&#x27;s smarter than its parents.<p><i>&quot;Beyond contextual hard limits, even if one part of a system has the ability to recursively self-improve, other parts of the system will inevitably start acting as
bottlenecks. Antagonistic processes will arise in response to recursive self-improvement and squash it ... a brain with smarter parts will have more trouble coordinating
them ... Exponential progress, meet exponential friction.&quot;</i><p>The problem is the author is looking at human-level problems with human-level intelligence.  A superintelligence may have no such problems, or be able to easily resolve them.  Once over the initial speed bump of creating such an intelligence it could be smooth sailing from then on.  There&#x27;s no way to tell, really, what its limits will be until it exists.  Besides, even if it can&#x27;t exponentially increase its own intelligence forever, even a relatively slight increase of intelligence over humanity could be a massive gamechanger.