Thoughts on OpenAI, reinforcement learning, and killer robots
Maybe I&#x27;m missing something here.<p>First of all, I know a lot of the popular perception of artificial &quot;general intelligence&quot; is overly simplistic - I don&#x27;t believe in the nerd rapture, and I agree with a lot of what was written in the three articles tweeted by Chollet that have been linked in this thread. And yet, I still don&#x27;t see how AI is not a plausible existential threat.<p>Unless you believe that some magical business is going on in the human mind that isn&#x27;t subject to the normal laws of physics, then I don&#x27;t see how you can believe that there&#x27;s anything our brains can do that another machine can&#x27;t. Even if no cognitive skill can be increased to infinity, we have no good reason to believe that our brains represent the maximum performance of all possible cognitive modes. That&#x27;s an appeal to the discredited idea that evolution has a &quot;ladder&quot;, upon which we stand at the apex as the finished product. Natural selection doesn&#x27;t optimize for intelligence, and it is not &quot;finished&quot;. So, if our brains are machines (albeit highly complex ones that we only partially understand), and if they probably don&#x27;t represent the maximum potential performance of cognition, then how can we say with any confidence that it is not possible to create another machine with higher cognitive performance across all (or nearly all) modes of cognition? And if that is possible, how can we say with confidence that such machines could pose no existential threat to us? Sure, maybe they won&#x27;t, or maybe we&#x27;ll never figure out how to build them. But how is it implausible? How is it something to laugh out of the room?<p>Furthermore, AI need not surpass us in all modes of cognition in order to be an existential threat. As AI gets better at accomplishing a wide variety of tasks, it becomes an ever more powerful lever for those who own it. The near-term threat from AI is socioeconomic: the replacement of vast numbers of jobs with AI&#x2F;robotics controlled by a small number of people who receive all the profits of their &quot;labor&quot;. It doesn&#x27;t take much imagination to see how thie could be, at the least, an existential threat to our current society if it is not addressed with adequate forethought.<p>All in all, I just do not see how AI is not an existential threat worth thinking about and sinking some money into researching how we can make it safer. The tired old argument about the need to spend those resources on more urgent matters doesn&#x27;t hold water. There are seven billion of us. We can specialize - indeed, it&#x27;s arguably our greatest strength! We can - and must! - devote resources and talent to a great many urgent issues, such as poverty, conflict, disease, and illiteracy. But I think we would be very unwise not to put a little of our wealth and time into researching how best to mitigate the long-term threats that don&#x27;t seem urgent yet and might not even come to pass. If we don&#x27;t, then chances are someday one of them will in fact come to pass and we&#x27;ll wish we had worked on it sooner.