How Far to AI Utopia?
There are two different types of discussions that happen about AI. They are very different but often confused for each other.<p>The first is the very near future, 10-20 years or so. Machine learning is rapidly advancing and we are getting all kinds of cool toys. Automating things that previously weren&#x27;t automatable. 90% of jobs probably could be automated soon, and there will be huge societal consequences. That&#x27;s what this article is about, mostly.<p>The second is far more interesting. Strong AI. The distant future of 30 years, and maybe sooner. I think it&#x27;s very likely this will happen in our lifetime, though obviously it&#x27;s impossible to know for sure.<p>This isn&#x27;t just replacing a few jobs with machines. This is replacing <i>all</i> humans with machines. When we get AIs that are just as smart as humans. And eventually <i>far</i> smarter than humans. AIs which are so far beyond us that we look like chimps.<p>We would not be able to compete with the AIs. They would be able to get whatever they want. Humans would no longer be in control. It&#x27;s very, very important that we make sure the AIs we build want what we want. Then we can get a utopia so advanced we can&#x27;t even imagine it. But if we get it wrong we will just be replaced or destroyed.<p>And we haven&#x27;t the slightest idea how to do this, or even if it&#x27;s possible. Creating an AI which wants to do what it&#x27;s human creator wants it to do is a remarkably hard problem. Especially if that AI is much more intelligent than it&#x27;s human creator. This is the so called &quot;control problem&quot;.