It is, in some sense, a very simple problem. When and if we advance to the point that AI algorithms are themselves capable of writing AI algorithms, what happens then? Quantify (albeit vaguely) the rate at which AI algorithms can improve AI algorithms. If that number turns out to be roughly as slow as human beings, there&#x27;s no significant problem. If it turns out that number is greatly larger than human beings, that&#x27;s a problem.<p>It is not particularly clear than the number is large; hard problem is hard. No sarcasm; arguably &quot;general intelligence&quot; is the hardest possibly engineering problem. Adding a bit more intelligence to the problem when it will already be getting worked on by a lot of humans may not move the needle much to speak of at all. But it is not particularly clear that that rate is small either, though; humans are smart, yet at the same time, really dumb in a lot of ways. We have terrible working memory (7 +&#x2F;- 2 items is absurdly small). We have lots of irrational biases. We have lots of things to do in our lives that are not &quot;thinking&quot; (eating, sleeping, etc.), and the vast majority of our brain is dedicated to those problems, not rational thought. We are terrible at manipulating vast symbol systems without taking immense shortcuts, which inevitably completely color our manipulations. What happens when something that lacks those restrictions is turned loose on the problem of writing AI algorithms?<p>Heck if I know. I&#x27;m a human too.<p>I honestly think that those people who are utterly convinced the rate must necessarily be slow are just as wrong as those who are utterly convinced it must necessarily be fast. We really don&#x27;t know, <i>but</i> it is hardly an invalid concern.<p>So far, AI algorithms have not written very many AI algorithms. I&#x27;ve seen some toys where one AI algorithm is hooked up to tune another one, but I&#x27;m not sure if any of them have ever been practically useful. (All I know is all the ones I&#x27;ve personally seen have amounted to toys.) And the system as a whole is generally constrained by what the final AI algorithm can output anyhow; if the domain and range of the AI function are immutably fixed, there&#x27;s not much the AI-meta-system can do to &quot;escape&quot; and do anything terribly nasty.<p>I do think we&#x27;re still a ways away from this being an issue. AI is currently still very much constrained to problems a great deal simpler than &quot;writing AI algorithms&quot;, which is right up there with the hardest possible things that <i>human</i> intelligences are currently capable of, and that only to a rare few highly trained <i>and</i> highly talented individuals. We&#x27;re easily decades away from this problem. But when the day comes, well, I wouldn&#x27;t <i>bet</i> on the self-improving AI experiencing multiple orders-of-magnitude improvement in mere minutes, but I&#x27;d sure hate to bet the future of the species that it&#x27;s not possible. It is not unreasonable to be concerned that the knee could be very steep. We&#x27;ll know more as we get closer.