Adding to this, imagine you were part of an ancient alien species checking in on earth every so often. Around 4.5 billion years ago, Earth was formed. 500 million years later, the first self-replicators appeared. Another half billion years (now 3.5 billion years ago) for things to settle down and create the most recent ancestor from which all life descends from. Over the next 3 billion years nothing much happens, until the Phanerozoic Eon is reached around 500 million years ago where more &quot;interesting&quot;, larger life starts to develop. But again for another half a billion years nothing that much happens other than evolution continuing playing its game of optimizing for gene replication in more and more interesting ways. Around one hundred thousand years ago, an interesting branch of a species has been developing in ways that indicate the rise of general intelligence (their few pounds of squishy matter in their heads are pretty neat: <a href="http:&#x2F;&#x2F;www.yudkowsky.net&#x2F;singularity&#x2F;power&#x2F;" rel="nofollow">http:&#x2F;&#x2F;www.yudkowsky.net&#x2F;singularity&#x2F;power&#x2F;</a>). But they don&#x27;t seem to be doing all that much until about 10-20 thousand years ago. And even then, less than 10 thousand years ago, they built some interesting structures like the pyramids, but their population is still tiny and their global influence is limited. And it wasn&#x27;t until about 500 years ago when things really started picking up in population growth and world influence (some mark that as the discovery of a formal scientific process to build and record more and more truths about the world that are actually true). Just around 100 years ago, this species developed air travel. In about 60 years after that, they went beyond their planet to their moon. This species at present could wipe out almost all life from the planet (if it wanted, which it doesn&#x27;t) in multiple ways, to really get every last bit of life exterminated would take some doing but in a large enough time span doesn&#x27;t seem impossible.<p>An alien species would have to be observing this planet at very frequent intervals to be able to catch any of these recent developments -- because if millions of years ago they determined nothing interesting was happening, and decided to only look in every five hundred thousand years or so, it&#x27;s very likely the last time they looked our species wasn&#x27;t even around.<p>Now you take the development of an AI that is roughly human level intelligent with a given amount of resources. If you give it the same amount of resources again, it can &quot;breed&quot;, copy itself perfectly onto them, and now we have a &quot;population&quot; of two AIs whose combined intelligence should roughly match that of two twin humans. Maybe it doesn&#x27;t even copy perfectly, maybe it switches some things about to see if it can create something smarter without modifying its own code directly. In any case its breeding is only restricted by its ability to breed and desire to breed in the first place and the compute and power resources it needs to run a new copy, and these can always get lower than initially. What does the long-term look like? Even if the initial resources for the first one are astronomical (e.g. all computing power in use on the planet as of 2016), I would still bet you that left to breed without any other restriction it would take the AI family far less time than 100 thousand years to reach 7 billion instances. So you&#x27;re looking at many, many more AIs than non-AIs, plus each with human level intelligence that isn&#x27;t distributed normally (assuming soft eugenics doesn&#x27;t become widespread in non-AI populations) but is mainly all the same with perhaps increases here and there (assuming a soft takeoff), each coordinating with the same goals in mind. This situation without further specification can be either extremely good for the non-AIs or extremely bad for the non-AIs. The question of whether this will happen (and whether it happens in that exact form, personally I think hard takeoff from a single AI is likely) in the next 100 years or the next hundred thousand years (perhaps a near-extinction-level event occurs forcing our species to basically start over but with an even harder battle to survive since many low-hanging-fruit resources have been depleted) might help you determine whether to worry now about doing actions that make the extremely-good more likely than the extremely-bad, but if one thinks human-level AI can&#x27;t happen at all in any amount of time, that there&#x27;s something special about our squishy brains or something inherently limiting about our so-far-general intelligence such that we can&#x27;t solve the engineering problem of creating another intelligence directly and must always go through breeding, the argument needs to take place at a different level.<p>The paperclip maximizer argument (not sure if Bostrom repeats it in his book) is yet another level of argument but it&#x27;s meant for those who already buy the premise of human level AI and self-improving AI but who also think increased intelligence will naturally converge to a human-loving benevolent AI that sees how self-evidently obvious it is to love all life or whatever, that we don&#x27;t have to worry about all the messy details around Friendliness because they&#x27;ll just fall out of the entity naturally. No one is actually worried about Clippy tiling the solar system with himself, its probability is epsilon. But a lot of people just see this one argument out of context and infer the arguer believes it to be a relevant possibility to worry about and write off the whole group as crazy...