Answers.Body: <p>AI will only "evolve" selfishness if it "evolves" in a competitive environment and has certain human-like faculties.</p>
  
  <p>Self-protecting desires on the other hand are logical consequences of having any goal at all. After all, you can't reach your goal if you are destroyed. </p>
  
  <p>The concern of "evil" super intelligences isn't that they literally turn evil and selfish and cruel. Those are human qualities.</p>
  
  <p>Instead a superintelligence that has a certain goal, will logically pursue subgoals that help it to reach the ultimate goal. Such subgoals will be resources, power, safety. </p>
  
  <p>So it will amass power and resources to reach its goal and exterminate any threat to its existence as long as its goal hasn't been reached, without being selfish at all.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-01T11:44:55.060Z
Answers.Text: Hello BlindKungFuMaster,Can you please first have a look at duplicated questions.
Answers.UserDisplayName: quintumnia
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-01T14:35:37.970Z
Answers.Text: @quintumnia: I don't think this question is a duplicate of the emotion or of the suffering question. Selfishness is neither an emotion nor a feeling. Also, the other questions are missing the AI-threat aspect. I really don't see much similarity.
Answers.UserDisplayName: BlindKungFuMaster
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-01T11:42:18.690Z
Answers.LastActivityDate: 2017-03-01T11:42:18.690Z
Answers.UserDisplayName: BlindKungFuMaster
Answers.Body: <p>I have some comments on this subject here: <a href="https://ai.stackexchange.com/a/2878/1671">https://ai.stackexchange.com/a/2878/1671</a></p>
  
  <p>This is some deep game theoretic stuff, and it partly depends on how you define "selfishness".  </p>
  
  <p>There is such a thing, for instance, as a "<a href="https://en.wikipedia.org/wiki/Greedy_algorithm" rel="nofollow noreferrer">greedy algorithm</a>".  Sometimes a greedy algorithm is the most convenient way to achieve an acceptable result, but the optimality is only local. </p>
  
  <p>On a mathematical level, the constructiveness or destructiveness of "self interest" in a system may be a function of whether a <a href="https://en.wikipedia.org/wiki/Nash_equilibrium" rel="nofollow noreferrer">Nash Equilibrium</a> is perceived.  In this case, self-interest is defined as maintaining the current strategy, because, unless the competitor changes their strategy, there is no gain for changing one's own strategy.</p>
  
  <p>As the BlindKungFuMaster importantly notes, competitiveness will evolve as a trait if the AI operates in a partisan context.  </p>
  
  <p>There, the problem comes from whether the "game" is zero sum (<a href="https://en.wikipedia.org/wiki/Pareto_efficiency" rel="nofollow noreferrer">Pareto Optimal</a>) or non zero sum (Pareto Improvable), or both.  Here, "destructive" may defined agents made worse off per the gains made by another agent.  </p>
  
  <p>Altruism seems to have a rational basis, and occurs in evolution because it is presumably useful. [<a href="https://plato.stanford.edu/entries/altruism-biological/" rel="nofollow noreferrer">See Biological Altruism.</a>]</p>
  
  <p>Although this tends to be confined to single species, the co-evolution of dogs and humans is a case for inter-species altruism, based on self interest.</p>
  
  <p>Humans and machines have also, and will continue to, co-evolve.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-03-01T16:55:13.723Z
Answers.LastActivityDate: 2017-03-01T16:55:13.723Z
Answers.LastEditDate: 2017-04-13T12:53:10.013Z
Answers.UserDisplayName: DukeZhou
Body: <p>A lot of experts have expressed concerns about evil super intelligence. While their concerns are valid, is it necessary, what are the chances or how the artificial super-intelligence will evolve to have selfishness and self protecting desires inherent in biological systems? Is there any work which comments on this line of inquiry?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-03-01T10:57:11.923Z
Comments.Text: Possible duplicate of [Could an AI feel emotion?](http://ai.stackexchange.com/questions/2277/could-an-ai-feel-emotion)
Comments.UserDisplayName: Mithical
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-03-01T11:41:35.190Z
Comments.Text: @Mithrandir So far two question have been duplicated from one question;and this [Can Robots learn how to feel suffering?](http://ai.stackexchange.com/questions/2872/can-robots-learn-how-to-feel-suffering)
Comments.UserDisplayName: quintumnia
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-03-01T16:35:37.060Z
Comments.Text: @quintumnia Not necessarily a duplicate imho, because this question is about "self interest" in a game theory context specifically, as opposed to useful traits related to "self interest" that could arise out of game theoretic applications of automata (genetic algorithms and evolutionary game theory.)
Comments.UserDisplayName: DukeZhou
Tags: <unsupervised-learning>
Title: Will artificial super-intelligence evolve to have selfishness inherent in biological systems?
UserDisplayName: akm
ThreadText: # Will artificial super-intelligence evolve to have selfishness inherent in biological systems?
  
  (Asked by: username_0 on 2017-03-01)
  
  A lot of experts have expressed concerns about evil super intelligence. While their concerns are valid, is it necessary, what are the chances or how the artificial super-intelligence will evolve to have selfishness and self protecting desires inherent in biological systems? Is there any work which comments on this line of inquiry?
  
  **Tags:** unsupervised-learning
  > **Comments:**
  > **username_1**: Possible duplicate of [Could an AI feel emotion?](http://ai.stackexchange.com/questions/2277/could-an-ai-feel-emotion)  
  > **username_2**: @Mithrandir So far two question have been duplicated from one question;and this [Can Robots learn how to feel suffering?](http://ai.stackexchange.com/questions/2872/can-robots-learn-how-to-feel-suffering)  
  > **username_3**: @quintumnia Not necessarily a duplicate imho, because this question is about "self interest" in a game theory context specifically, as opposed to useful traits related to "self interest" that could arise out of game theoretic applications of automata (genetic algorithms and evolutionary game theory.)  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_4 on 2017-03-01. Score: 2
  
  AI will only "evolve" selfishness if it "evolves" in a competitive environment and has certain human-like faculties.
  
  Self-protecting desires on the other hand are logical consequences of having any goal at all. After all, you can't reach your goal if you are destroyed.
  
  The concern of "evil" super intelligences isn't that they literally turn evil and selfish and cruel. Those are human qualities.
  
  Instead a superintelligence that has a certain goal, will logically pursue subgoals that help it to reach the ultimate goal. Such subgoals will be resources, power, safety.
  
  So it will amass power and resources to reach its goal and exterminate any threat to its existence as long as its goal hasn't been reached, without being selfish at all.
  
  > **Comments:**
  > **username_2**: Hello BlindKungFuMaster,Can you please first have a look at duplicated questions.  
  > **username_4**: @quintumnia: I don't think this question is a duplicate of the emotion or of the suffering question. Selfishness is neither an emotion nor a feeling. Also, the other questions are missing the AI-threat aspect. I really don't see much similarity.  
  
  ---
  
  ### Answer by username_3 on 2017-03-01. Score: 1
  
  I have some comments on this subject here: <https://ai.stackexchange.com/a/2878/1671>
  
  This is some deep game theoretic stuff, and it partly depends on how you define "selfishness".
  
  There is such a thing, for instance, as a "[greedy algorithm](https://en.wikipedia.org/wiki/Greedy_algorithm)". Sometimes a greedy algorithm is the most convenient way to achieve an acceptable result, but the optimality is only local.
  
  On a mathematical level, the constructiveness or destructiveness of "self interest" in a system may be a function of whether a [Nash Equilibrium](https://en.wikipedia.org/wiki/Nash_equilibrium) is perceived. In this case, self-interest is defined as maintaining the current strategy, because, unless the competitor changes their strategy, there is no gain for changing one's own strategy.
  
  As the BlindKungFuMaster importantly notes, competitiveness will evolve as a trait if the AI operates in a partisan context.
  
  There, the problem comes from whether the "game" is zero sum ([Pareto Optimal](https://en.wikipedia.org/wiki/Pareto_efficiency)) or non zero sum (Pareto Improvable), or both. Here, "destructive" may defined agents made worse off per the gains made by another agent.
  
  Altruism seems to have a rational basis, and occurs in evolution because it is presumably useful. [[See Biological Altruism.](https://plato.stanford.edu/entries/altruism-biological/)]
  
  Although this tends to be confined to single species, the co-evolution of dogs and humans is a case for inter-species altruism, based on self interest.
  
  Humans and machines have also, and will continue to, co-evolve.
  
  
  ---
  
  