The tl;dr is that we will eventually have AIs which are smarter than humans. <i>Much</i> smarter than humans, to the point that we are to them what chimpanzees are to us. They would have godlike powers and abilities. And this could happen very rapidly, as the first smart AIs design even better AIs and better computers, which then design even better AIs, and so on.<p>The second point is that they would have no reason to keep us around. They would do whatever we programmed them to do, to the point of absurdity. An AI programmed to collect as many paperclips as possible, would convert the mass of the solar system into paperclips. An AI programmed to solve the Riemann hypothesis would build the biggest computer possible to do more calculations. An AI programmed to value self preservation would try to destroy anything that was even a tiny percent chance of being a threat to it. It would try to preserve as much energy and mass as possible to last through the heat death of the universe. Etc.<p>The only AIs which actually do things we want them to do would have to be explicitly programmed to want to do that. And we have no idea how to do that. We can&#x27;t just hope that arbitrary AIs will happen to value humans instead of something else. We need to figure out how to control them, which requires solving something called &quot;the control problem&quot;.