Answers.Body: <p>Assuming super-intelligence is possible, the answer is probably yes and no.</p>
  
  <p>Yes in Kurzweil-like scenarios, where super-intelligence is an extension of human beings by technology (we are already in to some extent). Then control follows, as super-intelligence depends on us. It would extend our capabilities, such as speed of processing, extent of processing, etc. Even then control is debatable, as a remote-controlled killing machine would be part of a super-intelligent organism, partially human "controlled", partially autonomous.</p>
  
  <p>No in "Future of Life Institute"-like scenarios, where super-intelligence is independent from humans. The thinking is simple: What can we hope to do facing someone way more intelligent? The usual parallel is to compare this scenario with the arrival of the "developed" conquistadors in early America. Gunpowder vs. mere raw strength and arrows.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-02T10:59:38.660Z
Answers.Text: You mean Kurzweil? what is "FIL-like scenarios"?
Answers.UserDisplayName: akm
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-02T11:04:52.117Z
Answers.Text: let me counter argue in the case of non human like super-intelligence. of course, they will be made by humans. for such intelligence to do any harm, they must need resources i.e. power,materials etc. can't we control them by controlling these resources? can we think of any situation where they won't need resources to harm? but then again,we may not be able think of such situation because we are low intelligence! :)
Answers.UserDisplayName: akm
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-07T21:52:32.457Z
Answers.Text: @AmitMaurya That's the whole story. We may reach a point where we cannot wrap our head around what is happening, either because of complexity, or mere too-high pace of change. It seems you have follow-up questions, and those would be terrific new entries on the site.
Answers.UserDisplayName: Eric Platon
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-01T22:31:59.743Z
Answers.LastActivityDate: 2017-03-07T21:49:37.613Z
Answers.LastEditDate: 2017-03-07T21:49:37.613Z
Answers.UserDisplayName: Eric Platon
Answers.Body: <p>Competition always gives better result. If machines will try to improve themselves, we as human beings will definitely try to improve ourself.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-03T06:25:30.740Z
Answers.LastActivityDate: 2017-03-03T06:25:30.740Z
Answers.UserDisplayName: Rishi Raj
Answers.Body: <p>Yes, it is possible.</p>
  
  <p>When humans were working on the first nuclear bomb, some field experts of the time thought that when the reaction went super-critical, it would not stop, and would devour the earth. It was a plausible <em>possibility</em> given our understand of nuclear energy at the time, and we didn't know for sure until we did it.</p>
  
  <p>Some scientists synthesize black-hole like environments in laboratories. Some experts think that if a certain point is accidentally crossed due to ignorance or negligence, we may devour our planet with a self made black hole.</p>
  
  <p>The situation is the same with AI. Until we actually create a super-intelligent AI, we <strong>cannot say with certainty</strong> whether it will be controlled or controllable until it happens. Until that time comes the answer to your question is yes, it's possible, but that does not mean it will or will not happen that way.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-11T02:17:56.760Z
Answers.LastActivityDate: 2017-03-11T02:17:56.760Z
Answers.UserDisplayName: Nomadyn
Answers.Body: <p>Without going into more detail at the moment (b/c I'm time strapped), I strongly urge you to research the <a href="https://en.wikipedia.org/wiki/AI_control_problem" rel="nofollow noreferrer">Control Problem</a>.</p>
  
  <p><em>My own personal view is that humans are more problematic than machines.  Machines are at least rational.</em></p>
  
  <p>To be more specific, I believe human "management" (read as "mis-management") of powerful AI is potentially more of a problem than super-intelligent AI left to it's own devices.  </p>
  
  <p>Humans are known to abuse power, and history is filled with such examples.  Machines, at least, have  a clean slate in this regard.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-02T22:27:55.480Z
Answers.Text: thanks. problem is machines are created by humans.problematic humans may create problematic machines. even if machines are good initially,the way they will evolve can be problematic.
Answers.UserDisplayName: akm
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-02T22:30:19.713Z
Answers.Text: @AmitMaurya Could be problematic, but I see that as a factor of humans imposing poor ideas based on shoddy understanding.  Details here: http://ai.stackexchange.com/a/2878/1671
Answers.UserDisplayName: DukeZhou
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-02T22:56:35.283Z
Answers.Text: u r very correct when u say "Humans are known to abuse power, and history is filled with such examples".  power corrupts. so what will happen with powerful AIs? "leaving it on its  own" may not be a  good idea.
Answers.UserDisplayName: akm
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-03T20:24:25.477Z
Answers.Text: @AmitMaurya Yes, but humans have emotions that cloud logic and lead to poor decisions. Machines are not so burdened. The only apocalyptic scenarios regarding AI that are completely sound, imho, are the ones regarding lack intelligence (grey goo) or insufficient intelligence ("Hal 2000", "I Robot", etc.) Hal was smart but he couldn't contextualize or reconcile contradictions. .
Answers.UserDisplayName: DukeZhou
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-02T22:22:18.233Z
Answers.LastActivityDate: 2017-03-02T22:41:14.743Z
Answers.LastEditDate: 2017-03-02T22:41:14.743Z
Answers.UserDisplayName: DukeZhou
Body: <p>Post singularity AI will surpass human intelligence. The evolution of AI can take any direction, some of which may not be preferable for humans. Is it possible to manage the evolution of super-intelligent AI? If yes, how? One way I can think of is following. Instead of having a mobile AI like humanoid, we can keep it immobile, like a box, like current super computers. It can be used to solve problems of maths, theoretical science etc.</p>
  
Tags: <singularity>
Title: Is it possible to manage the evolution of super-intelligent AI?
UserDisplayName: akm
ThreadText: # Is it possible to manage the evolution of super-intelligent AI?
  
  (Asked by: username_0 on 2017-03-01)
  
  Post singularity AI will surpass human intelligence. The evolution of AI can take any direction, some of which may not be preferable for humans. Is it possible to manage the evolution of super-intelligent AI? If yes, how? One way I can think of is following. Instead of having a mobile AI like humanoid, we can keep it immobile, like a box, like current super computers. It can be used to solve problems of maths, theoretical science etc.
  
  **Tags:** singularity
  
  ---
  
  ## 4 answers
  
  ### Answer by username_1 on 2017-03-01. Score: 2
  
  Assuming super-intelligence is possible, the answer is probably yes and no.
  
  Yes in Kurzweil-like scenarios, where super-intelligence is an extension of human beings by technology (we are already in to some extent). Then control follows, as super-intelligence depends on us. It would extend our capabilities, such as speed of processing, extent of processing, etc. Even then control is debatable, as a remote-controlled killing machine would be part of a super-intelligent organism, partially human "controlled", partially autonomous.
  
  No in "Future of Life Institute"-like scenarios, where super-intelligence is independent from humans. The thinking is simple: What can we hope to do facing someone way more intelligent? The usual parallel is to compare this scenario with the arrival of the "developed" conquistadors in early America. Gunpowder vs. mere raw strength and arrows.
  
  > **Comments:**
  > **username_0**: You mean Kurzweil? what is "FIL-like scenarios"?  
  > **username_0**: let me counter argue in the case of non human like super-intelligence. of course, they will be made by humans. for such intelligence to do any harm, they must need resources i.e. power,materials etc. can't we control them by controlling these resources? can we think of any situation where they won't need resources to harm? but then again,we may not be able think of such situation because we are low intelligence! :)  
  > **username_1**: @AmitMaurya That's the whole story. We may reach a point where we cannot wrap our head around what is happening, either because of complexity, or mere too-high pace of change. It seems you have follow-up questions, and those would be terrific new entries on the site.  
  
  ---
  
  ### Answer by username_2 on 2017-03-03. Score: 1
  
  Competition always gives better result. If machines will try to improve themselves, we as human beings will definitely try to improve ourself.
  
  
  ---
  
  ### Answer by username_3 on 2017-03-11. Score: 0
  
  Yes, it is possible.
  
  When humans were working on the first nuclear bomb, some field experts of the time thought that when the reaction went super-critical, it would not stop, and would devour the earth. It was a plausible *possibility* given our understand of nuclear energy at the time, and we didn't know for sure until we did it.
  
  Some scientists synthesize black-hole like environments in laboratories. Some experts think that if a certain point is accidentally crossed due to ignorance or negligence, we may devour our planet with a self made black hole.
  
  The situation is the same with AI. Until we actually create a super-intelligent AI, we **cannot say with certainty** whether it will be controlled or controllable until it happens. Until that time comes the answer to your question is yes, it's possible, but that does not mean it will or will not happen that way.
  
  
  ---
  
  ### Answer by username_4 on 2017-03-02. Score: 0
  
  Without going into more detail at the moment (b/c I'm time strapped), I strongly urge you to research the [Control Problem](https://en.wikipedia.org/wiki/AI_control_problem).
  
  *My own personal view is that humans are more problematic than machines. Machines are at least rational.*
  
  To be more specific, I believe human "management" (read as "mis-management") of powerful AI is potentially more of a problem than super-intelligent AI left to it's own devices.
  
  Humans are known to abuse power, and history is filled with such examples. Machines, at least, have a clean slate in this regard.
  
  > **Comments:**
  > **username_0**: thanks. problem is machines are created by humans.problematic humans may create problematic machines. even if machines are good initially,the way they will evolve can be problematic.  
  > **username_4**: @AmitMaurya Could be problematic, but I see that as a factor of humans imposing poor ideas based on shoddy understanding.  Details here: http://ai.stackexchange.com/a/2878/1671  
  > **username_0**: u r very correct when u say "Humans are known to abuse power, and history is filled with such examples".  power corrupts. so what will happen with powerful AIs? "leaving it on its  own" may not be a  good idea.  
  > **username_4**: @AmitMaurya Yes, but humans have emotions that cloud logic and lead to poor decisions. Machines are not so burdened. The only apocalyptic scenarios regarding AI that are completely sound, imho, are the ones regarding lack intelligence (grey goo) or insufficient intelligence ("Hal 2000", "I Robot", etc.) Hal was smart but he couldn't contextualize or reconcile contradictions. .  
  
  ---
  
  