There's No Fire Alarm for Artificial General Intelligence
I worry talking about AGI is like going to the early industrial revolution and worrying about man building superhuman biology. A reasonable critic would point at the many aspects of biology we have little hope of replicating, like growth, self reparation, and general robustness.<p>But history has never been about competing on the same playing field. We don&#x27;t build cars that perform like poor horses, we build cars that are 99% inferior to biology and 1% far, far superior. When we find something that looks like an existential threat, it isn&#x27;t the mostly-general superhuman robot terminator, it&#x27;s the tool that&#x27;s that-much-superhuman on 0.01% of tasks: nuclear fusion.<p>I see no reason to bet against this same argument for AI. AlphaGo isn&#x27;t 130% of a human Go master, it&#x27;s 1,000x at a tiny sliver of the game. And the first AI that poses an existential threat won&#x27;t need to have super- or even near-human levels of each piece of mental machinery, and I don&#x27;t even have much reason to believe it will look like an entity at all. It could very well be something, some <i>system</i>, that achieves massive superintelligence on <i>just enough</i> to break the foundations of society.<p>Our world isn&#x27;t designed to be robust against superhuman adversaries, even if those adversaries are mostly idiot. If we have hope of a fire alarm, it&#x27;s that things will break faster and far worse than people expect.