We want to make a utopia. But no one knows or agrees what utopia would look like. And it s very easy to get it wrong and get stuck in a bad or suboptimal system.<p>I think that (superhuman) AI will help us figure this out. It could model us well and figure out &quot;what we really want&quot;. It could propose questions and ideas we wouldn&#x27;t think of, and help us work out what the answer Is.<p>However the problem is getting to that point. We don&#x27;t know how to make an AI that will follow instructions at all. Most possible A Is do everything within their power to take over the world.