Answers.Body: <p>It's not just Hawking, you hear variations on this refrain from a lot of people.  And given that they're mostly very smart, well educated, well informed people (Elon Musk is another, for example), it probably shouldn't be dismissed out of hand.</p>
  
  <p>Anyway, the basic idea seems to be this: If we create "real" artificial intelligence, at some point, it will be able to improve itself, which improves it's ability to improve itself, which means it can improve it's ability to improve itself even more, and so on... a runaway cascade leading to "superhuman intelligence".  That is to say, leading to something that more intelligent than we area.</p>
  
  <p>So what happens if there is an entity on this planet which is literally more intelligent than us (humans)? Would it be a threat to us?  Well, it certainly seems reasonable to speculate that it <em>could</em> be so.   OTOH, we have no particular reason, right now, to think that it <em>will</em> be so. </p>
  
  <p>So it seems that Hawking, Musk, etc. are just coming down on the more cautious / fearful side of things.  Since we don't <em>know</em> if a superhuman AI will be dangerous or not, and given that it could be unstoppable if it were to become malicious (remember, it's smarter than we are!), it's a reasonable thing to take under consideration.</p>
  
  <p>Eliezer Yudkowsky has also written quite a bit on this subject, including come up with the famous "AI Box" experiment.  I think anybody interested in this topic should read some of his material.</p>
  
  <p><a href="http://www.yudkowsky.net/singularity/aibox/" rel="noreferrer">http://www.yudkowsky.net/singularity/aibox/</a></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-02T16:00:04.297Z
Answers.Text: I think the AI Box is actually a bad introduction to AI risk; it targets a specific small part of the explanation, which is "how could anyone ever be convinced to let an AI out of the box?". To someone who thinks that AI is worth using, they've already decided it's worth letting out of the box. The Orthogonality Thesis is probably the best place to start, I think.
Answers.UserDisplayName: Matthew Gray
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-02T16:04:37.090Z
Answers.Text: Fair enough.  I just happen to be more familiar with Yudkowsky's writings.  I certainly don't mean to diminish the importance of other works on this topic.
Answers.UserDisplayName: mindcrime
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-02T15:56:10.167Z
Answers.LastActivityDate: 2016-08-02T15:56:10.167Z
Answers.UserDisplayName: mindcrime
Answers.Body: <p>Because he did not yet know how far away current AI is... Working in an media AI lab, I get this question a lot. But really... we are still a long way from this. The robots still do everything we detailledly describe them to do. Instead of seeing the robot as intelligent, I would look to the human programmer for where the creativity really happens.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-02T15:54:29.263Z
Answers.LastActivityDate: 2016-08-02T15:54:29.263Z
Answers.UserDisplayName: dorien
Answers.Body: <p>As Andrew Ng <a href="http://www.theregister.co.uk/2015/03/19/andrew_ng_baidu_ai/" rel="nofollow noreferrer">said</a>, worrying about such threat from AI is like worrying about of overpopulation on Mars. It is science fiction. </p>
  
  <p><a href="https://i.sstatic.net/m6jnl.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/m6jnl.png" alt="enter image description here"></a></p>
  
  <p>That being said, given the rise of (much weaker) robots and other (semi-)autonomous agents, the fields of the law and ethics are increasingly incorporating them, e.g. see <a href="https://en.wikipedia.org/wiki/Roboethics" rel="nofollow noreferrer">Roboethics</a>.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-02T15:57:19.303Z
Answers.LastActivityDate: 2016-08-02T15:57:19.303Z
Answers.UserDisplayName: Franck Dernoncourt
Answers.Body: <p>There are a number of long resources to answer this sort of question: consider Stuart Armstrong's book <a href="http://rads.stackoverflow.com/amzn/click/B00IB4N4KU" rel="nofollow">Smarter Than Us</a>, Nick Bostrom's book <a href="http://rads.stackoverflow.com/amzn/click/B00LOOCGB2" rel="nofollow">Superintelligence</a>, which grew out of this <a href="http://www.nickbostrom.com/views/superintelligence.pdf" rel="nofollow">edge.org answer</a>, <a href="http://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html" rel="nofollow">Tim Urban's explanation</a>, or <a href="https://aisafety.wordpress.com/" rel="nofollow">Michael Cohen's explanation</a>.</p>
  
  <p>But here's my (somewhat shorter) answer: intelligence is all about decision-making, and we don't have any reason to believe that humans are anywhere near close to being the best possible at decision-making. Once we are able to build an AI AI researcher (that is, a computer that knows how to make computers better at thinking), the economic and military relevance of humans will rapidly disappear as any decision that could be made by a human could be made better by a computer. (Why have human generals instead of robot generals, human engineers instead of robot engineers, and so on.)</p>
  
  <p>This isn't necessarily a catastrophe. If the Vulcans showed up tomorrow and brought better decision-making to Earth, we could avoid a lot of misery. The hard part is making sure that what we get are Vulcans who want us around and happy, instead of something that doesn't share our values.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-02T15:58:13.970Z
Answers.LastActivityDate: 2016-08-02T15:58:13.970Z
Answers.UserDisplayName: Matthew Gray
Answers.Body: <blockquote>
    <p>To put it simply in layman terms, what are the possible threats from AI? </p>
  </blockquote>
  
  <p>Currently, there are no threat. </p>
  
  <p>The threat comes if humans create a so-called ultraintelligent machine, a machine that can surpass all intellectual activities by any human. This would be the last invention man would need to do, since this machine is better in inventing machines than humans are (since that is an intellectual activity).  However, this could cause the machine to invent machines that can destruct humans, and we can't stop them because they are so much smarter than we are.</p>
  
  <p>This is all hypothetical, no one has even a clue of what an ultraintelligent machine looks like. </p>
  
  <blockquote>
    <p>If we know that AI is so dangerous why are we still promoting it? Why is it not banned?</p>
  </blockquote>
  
  <p>As I said before, the existence of a ultraintelligent machine is hypothetical. Artificial Intelligence has lots of useful applications (more than this answer can contain), and if we develop it, we get even more useful applications. We just have to be careful that the machines won't overtake us. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-02T15:54:26.937Z
Answers.LastActivityDate: 2016-08-02T15:54:26.937Z
Answers.UserDisplayName: wythagoras
Answers.Body: <p>He says this because it can happen. If something becomes smarter than us, why would it continue to serve us? The worst case scenario is that it takes over all manufacturing processes and consumes all matter to convert it into material capable of computation, extending outward infinitely until all matter is consumed.</p>
  
  <p>We know that AI is dangerous but it doesn't matter because most people don't believe in it. It goes against every comfort religion has to offer. Man is the end-all-be-all of the universe and if that fact is disputed, people will feel out of place and purposeless.</p>
  
  <p>The fact is most people just don't acknowledge it's possible, or that it will happen in our lifetimes, even though many reputable AI experts put the occurrence of the singularity within two decades. If people truly acknowledged that AI that was smarter than them was possible, wouldn't they be living differently? Wouldn't they be looking to do things that they enjoy, knowing that whatever it is they do that they dread will be automated? Wouldn't everyone be calling for a universal basic income?</p>
  
  <p>The other reason we don't ban it is because its promise is so great. One researcher could be augmented by 1,000 digital research assistants. All manual labor could be automated. For the first time, technology offers us real freedom to do whatever we please.</p>
  
  <p>But even in this best case scenario where it doesn't overtake us, humans still have to adapt and alter their economic system to one where labor isn't necessary. Otherwise, those who aren't technically-trained will starve and revolt.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-03T01:57:29.823Z
Answers.Text: Do you have a sauce for this -> _"AI experts put the occurrence of the singularity within two decades"_, please? ;)
Answers.UserDisplayName: Daniel
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-03T14:25:05.593Z
Answers.Text: I can't seem to find the original survey, but these sites come close:
  http://aiimpacts.org/predictions-of-human-level-ai-timelines/
  – http://aiimpacts.org/ai-timeline-surveys/  –
  http://techemergence.com/artificial-intelligence-risk/
Answers.UserDisplayName: zavtra
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-02T15:57:48.363Z
Answers.LastActivityDate: 2016-08-02T16:46:21.237Z
Answers.LastEditDate: 2016-08-02T16:46:21.237Z
Answers.UserDisplayName: zavtra
Body: <p>This <a href="https://www.independent.co.uk/life-style/gadgets-and-tech/news/stephen-hawking-artificial-intelligence-could-wipe-out-humanity-when-it-gets-too-clever-humans-could-become-ants-being-stepped-a6686496.html" rel="nofollow noreferrer">quote by Stephen Hawking</a> has been in headlines for quite some time:</p>
  <blockquote>
  <p>Artificial Intelligence could wipe out humanity when it gets too clever as humans will be like ants.</p>
  </blockquote>
  <p>Why does he say this? To put it simply: what are the possible threats from AI (that Stephen Hawking is worried about)? If we know that AI is so dangerous, why are we still promoting it? Why is it not banned?</p>
  <p>What are the adverse consequences of the so-called <a href="https://en.wikipedia.org/wiki/Technological_singularity" rel="nofollow noreferrer">Technological Singularity</a>?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-24T15:21:23.427Z
Comments.Text: Well, if we see hierarchy of evolution, lesser intelligent species are consumed one way or another by higher intelligent species, so grass consume nutrients, insects consume grass, birds consume insects and human consume birds (you get the point even if the sequence is vague). But imagine if human themselve evolve better then they are now, dont you think they would want to press for dominance?
Comments.UserDisplayName: user0193
Tags: <agi><superintelligence><singularity><ai-safety><ai-takeover>
Title: Why does Stephen Hawking say "Artificial Intelligence will kill us all"?
UserDisplayName: Soham
ThreadText: # Why does Stephen Hawking say "Artificial Intelligence will kill us all"?
  
  (Asked by: username_0 on 2016-08-02)
  
  This [quote by Stephen Hawking](https://www.independent.co.uk/life-style/gadgets-and-tech/news/stephen-hawking-artificial-intelligence-could-wipe-out-humanity-when-it-gets-too-clever-humans-could-become-ants-being-stepped-a6686496.html) has been in headlines for quite some time:
  
  > Artificial Intelligence could wipe out humanity when it gets too clever as humans will be like ants.
  
  Why does he say this? To put it simply: what are the possible threats from AI (that Stephen Hawking is worried about)? If we know that AI is so dangerous, why are we still promoting it? Why is it not banned?
  
  What are the adverse consequences of the so-called [Technological Singularity](https://en.wikipedia.org/wiki/Technological_singularity)?
  
  **Tags:** agi, superintelligence, singularity, ai-safety, ai-takeover
  > **Comments:**
  > **username_1**: Well, if we see hierarchy of evolution, lesser intelligent species are consumed one way or another by higher intelligent species, so grass consume nutrients, insects consume grass, birds consume insects and human consume birds (you get the point even if the sequence is vague). But imagine if human themselve evolve better then they are now, dont you think they would want to press for dominance?  
  
  ---
  
  ## 6 answers
  
  ### Answer by username_2 on 2016-08-02. Score: 5
  
  It's not just Hawking, you hear variations on this refrain from a lot of people. And given that they're mostly very smart, well educated, well informed people (Elon Musk is another, for example), it probably shouldn't be dismissed out of hand.
  
  Anyway, the basic idea seems to be this: If we create "real" artificial intelligence, at some point, it will be able to improve itself, which improves it's ability to improve itself, which means it can improve it's ability to improve itself even more, and so on... a runaway cascade leading to "superhuman intelligence". That is to say, leading to something that more intelligent than we area.
  
  So what happens if there is an entity on this planet which is literally more intelligent than us (humans)? Would it be a threat to us? Well, it certainly seems reasonable to speculate that it *could* be so. OTOH, we have no particular reason, right now, to think that it *will* be so.
  
  So it seems that Hawking, Musk, etc. are just coming down on the more cautious / fearful side of things. Since we don't *know* if a superhuman AI will be dangerous or not, and given that it could be unstoppable if it were to become malicious (remember, it's smarter than we are!), it's a reasonable thing to take under consideration.
  
  Eliezer Yudkowsky has also written quite a bit on this subject, including come up with the famous "AI Box" experiment. I think anybody interested in this topic should read some of his material.
  
  <http://www.yudkowsky.net/singularity/aibox/>
  
  > **Comments:**
  > **username_3**: I think the AI Box is actually a bad introduction to AI risk; it targets a specific small part of the explanation, which is "how could anyone ever be convinced to let an AI out of the box?". To someone who thinks that AI is worth using, they've already decided it's worth letting out of the box. The Orthogonality Thesis is probably the best place to start, I think.  
  > **username_2**: Fair enough.  I just happen to be more familiar with Yudkowsky's writings.  I certainly don't mean to diminish the importance of other works on this topic.  
  
  ---
  
  ### Answer by username_4 on 2016-08-02. Score: 4
  
  Because he did not yet know how far away current AI is... Working in an media AI lab, I get this question a lot. But really... we are still a long way from this. The robots still do everything we detailledly describe them to do. Instead of seeing the robot as intelligent, I would look to the human programmer for where the creativity really happens.
  
  
  ---
  
  ### Answer by username_5 on 2016-08-02. Score: 4
  
  As Andrew Ng [said](http://www.theregister.co.uk/2015/03/19/andrew_ng_baidu_ai/), worrying about such threat from AI is like worrying about of overpopulation on Mars. It is science fiction.
  
  [![enter image description here](https://i.sstatic.net/m6jnl.png)](https://i.sstatic.net/m6jnl.png)
  
  That being said, given the rise of (much weaker) robots and other (semi-)autonomous agents, the fields of the law and ethics are increasingly incorporating them, e.g. see [Roboethics](https://en.wikipedia.org/wiki/Roboethics).
  
  
  ---
  
  ### Answer by username_3 on 2016-08-02. Score: 3
  
  There are a number of long resources to answer this sort of question: consider Stuart Armstrong's book [Smarter Than Us](http://rads.stackoverflow.com/amzn/click/B00IB4N4KU), Nick Bostrom's book [Superintelligence](http://rads.stackoverflow.com/amzn/click/B00LOOCGB2), which grew out of this [edge.org answer](http://www.nickbostrom.com/views/superintelligence.pdf), [Tim Urban's explanation](http://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html), or [Michael Cohen's explanation](https://aisafety.wordpress.com/).
  
  But here's my (somewhat shorter) answer: intelligence is all about decision-making, and we don't have any reason to believe that humans are anywhere near close to being the best possible at decision-making. Once we are able to build an AI AI researcher (that is, a computer that knows how to make computers better at thinking), the economic and military relevance of humans will rapidly disappear as any decision that could be made by a human could be made better by a computer. (Why have human generals instead of robot generals, human engineers instead of robot engineers, and so on.)
  
  This isn't necessarily a catastrophe. If the Vulcans showed up tomorrow and brought better decision-making to Earth, we could avoid a lot of misery. The hard part is making sure that what we get are Vulcans who want us around and happy, instead of something that doesn't share our values.
  
  
  ---
  
  ### Answer by username_6 on 2016-08-02. Score: 3
  
  > To put it simply in layman terms, what are the possible threats from AI?
  
  Currently, there are no threat.
  
  The threat comes if humans create a so-called ultraintelligent machine, a machine that can surpass all intellectual activities by any human. This would be the last invention man would need to do, since this machine is better in inventing machines than humans are (since that is an intellectual activity). However, this could cause the machine to invent machines that can destruct humans, and we can't stop them because they are so much smarter than we are.
  
  This is all hypothetical, no one has even a clue of what an ultraintelligent machine looks like.
  
  > If we know that AI is so dangerous why are we still promoting it? Why is it not banned?
  
  As I said before, the existence of a ultraintelligent machine is hypothetical. Artificial Intelligence has lots of useful applications (more than this answer can contain), and if we develop it, we get even more useful applications. We just have to be careful that the machines won't overtake us.
  
  
  ---
  
  ### Answer by username_7 on 2016-08-02. Score: 2
  
  He says this because it can happen. If something becomes smarter than us, why would it continue to serve us? The worst case scenario is that it takes over all manufacturing processes and consumes all matter to convert it into material capable of computation, extending outward infinitely until all matter is consumed.
  
  We know that AI is dangerous but it doesn't matter because most people don't believe in it. It goes against every comfort religion has to offer. Man is the end-all-be-all of the universe and if that fact is disputed, people will feel out of place and purposeless.
  
  The fact is most people just don't acknowledge it's possible, or that it will happen in our lifetimes, even though many reputable AI experts put the occurrence of the singularity within two decades. If people truly acknowledged that AI that was smarter than them was possible, wouldn't they be living differently? Wouldn't they be looking to do things that they enjoy, knowing that whatever it is they do that they dread will be automated? Wouldn't everyone be calling for a universal basic income?
  
  The other reason we don't ban it is because its promise is so great. One researcher could be augmented by 1,000 digital research assistants. All manual labor could be automated. For the first time, technology offers us real freedom to do whatever we please.
  
  But even in this best case scenario where it doesn't overtake us, humans still have to adapt and alter their economic system to one where labor isn't necessary. Otherwise, those who aren't technically-trained will starve and revolt.
  
  > **Comments:**
  > **username_8**: Do you have a sauce for this -> _"AI experts put the occurrence of the singularity within two decades"_, please? ;)  
  > **username_7**: I can't seem to find the original survey, but these sites come close: http://aiimpacts.org/predictions-of-human-level-ai-timelines/ – http://aiimpacts.org/ai-timeline-surveys/  – http://techemergence.com/artificial-intelligence-risk/  
  
  ---
  
  