So everyone keeps saying this but I don&#x27;t see any evidence that this will actually be the case.<p>Consider that the first artificial intelligence probably won&#x27;t be (exactly) designed at all. It will be a neural network or some other construct of an advanced algorithm that&#x27;s solving some machine learning problem, and once it emerges no single human being (or possibly even group of humans) will fully understand why it works. Lets say that this AI is, against all odds, smarter than any human being. Chances are that NO human being will understand how the AI works, but why would you think that the AI understands itself? Why would we be certain that an AI can understand it&#x27;s own neural network better than humans can, and furthermore, be able to quickly iterate on it, especially if a chaotic and complex process produced it in the first place?<p>I suspect that machine intelligence will arise in a much more similar fashion to messy organic evolution than you think, and it will be subject to all the same disadvantages - lots of random chance, dead ends and very slow advancement by trial and error.