Answers.Body: <p>Steve Omohudro wrote a paper called <a href="https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf" rel="noreferrer">Basic AI Drives</a> that steps through why we would expect an AI with narrow goals to find some basic, general concepts as instrumentally useful for their narrow goals. </p>
  
  <p>For example, an AI designed to maximize stock market returns but whose design is silent on the importance of continuing to survive would realize that its continued survival is a key component of maximizing stock market returns, and thus take actions to keep itself operational.</p>
  
  <p>In general, we should be skeptical of 'anthropomorphizing' AI and other code, but it seems like there <em>are</em> reasons to expect this beyond "well, humans behave this way, so it must be how all intelligence behaves."</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-04-22T20:46:24.443Z
Answers.Text: I assume Omohudru uses "maximize returns" in the context of minimax, where mitigation of downside is part of the equation.  From my perspective, "win else survive" is a basic extension of minimax in cyclic games.
Answers.UserDisplayName: DukeZhou
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-03T15:58:13.493Z
Answers.LastActivityDate: 2016-08-03T15:58:13.493Z
Answers.UserDisplayName: Matthew Gray
Answers.Body: <p>The concept of 'survival instinct' probably falls in the category of what Marvin Minsky would call a 'suitcase word', i.e. it packages together a number of related phenomena into what at first appears to be a singular notion. </p>
  
  <p>So it's quite possible that we can construct mechanisms that have the appearance of some kind of 'hard-coded' survival instinct, without that ever featuring as an explicit rule(s) in the design.</p>
  
  <p>See the beautiful little book <a href="https://mitpress.mit.edu/books/vehicles" rel="noreferrer">'Vehicles'</a> by the neuroanatomist Valentino Braitenberg for a compelling narrative of how such 'top down' concepts as 'survival instinct' might evolve 'from the bottom up'.</p>
  
  <p>Also, trying to ensure that intelligent artefacts place too high a priority on their survival might easily lead to a <a href="https://xkcd.com/1613/" rel="noreferrer">Killbot Hellscape</a>.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-03T15:45:41.770Z
Answers.LastActivityDate: 2016-08-03T15:45:41.770Z
Answers.UserDisplayName: NietzscheanAI
Answers.Body: <p>I will answer by quoting the book <em>The Myth of Sisyphus</em>: </p>
  
  <blockquote>
    <p>There is only one really serious philosophical question, and that is suicide. </p>
  </blockquote>
  
  <p>So, probably, we need <em>some</em> degree of a survival instinct if we don't want our AGIs to "terminate themselves" (whatever that means), whenever they get existential.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-04-22T04:23:39.397Z
Answers.LastActivityDate: 2019-11-22T23:30:12.630Z
Answers.LastEditDate: 2019-11-22T23:30:12.630Z
Answers.UserDisplayName: k.c. sayz 'k.c sayz'
Body: <p>This question stems from quite a few "informal" sources. Movies like <em>2001, A Space Odyssey</em> and <em>Ex Machina</em>; books like <em>Destination Void</em> (Frank Herbert), and others suggest that general intelligence <em>wants</em> to survive, and even learn the importance for it.</p>
  
  <p>There may be several arguments for survival. What would be the most prominent?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-15T04:47:35.947Z
Comments.Text: @DukeZhou Would you have public documents on the topic?
Comments.UserDisplayName: Eric Platon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-15T16:48:54.717Z
Comments.Text: Not as yet. *(It's going to be a long process as our development cycle is very slow.  I also probably shouldn't have said mentioned AGI in this context because what I'm really looking at is more of a generalized utility function that mitigates risk, not a super-intelligence!)*   Wanted to update you, but probably best if I delete my original comment for being overly-optimistic/misleading in this regard.
Comments.UserDisplayName: DukeZhou
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-18T03:08:53.723Z
Comments.Text: Essentially, my thought was that a general economic utility function would form a good basis for an AGI by providing a *prime motive*.  (The idea is that general intelligence is just a degree of capability re: an infinitely expanding set of problems.)  It would relate to survival instinct in the sense that the automata is driven toward optimal decisionmaking in the context of maintaining and bettering it's condition, *(though not to the point of mindless expansion which could be harmful to its "ecosystem)*, where every decision is ultimately regarded as economic.
Comments.UserDisplayName: DukeZhou
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-18T08:52:13.427Z
Comments.Text: (if you have an "online venue" where those topics can be discussed, that'd be great---I think we are getting kicked off this site)
Comments.UserDisplayName: Eric Platon
Tags: <philosophy><agi>
Title: Is there a strong argument that survival instinct is a prerequisite for creating an AGI?
UserDisplayName: Eric Platon
ThreadText: # Is there a strong argument that survival instinct is a prerequisite for creating an AGI?
  
  (Asked by: username_0 on 2016-08-03)
  
  This question stems from quite a few "informal" sources. Movies like *2001, A Space Odyssey* and *Ex Machina*; books like *Destination Void* (Frank Herbert), and others suggest that general intelligence *wants* to survive, and even learn the importance for it.
  
  There may be several arguments for survival. What would be the most prominent?
  
  **Tags:** philosophy, agi
  > **Comments:**
  > **username_0**: @DukeZhou Would you have public documents on the topic?  
  > **username_1**: Not as yet. *(It's going to be a long process as our development cycle is very slow.  I also probably shouldn't have said mentioned AGI in this context because what I'm really looking at is more of a generalized utility function that mitigates risk, not a super-intelligence!)*   Wanted to update you, but probably best if I delete my original comment for being overly-optimistic/misleading in this regard.  
  > **username_1**: Essentially, my thought was that a general economic utility function would form a good basis for an AGI by providing a *prime motive*.  (The idea is that general intelligence is just a degree of capability re: an infinitely expanding set of problems.)  It would relate to survival instinct in the sense that the automata is driven toward optimal decisionmaking in the context of maintaining and bettering it's condition, *(though not to the point of mindless expansion which could be harmful to its "ecosystem)*, where every decision is ultimately regarded as economic.  
  > **username_0**: (if you have an "online venue" where those topics can be discussed, that'd be great---I think we are getting kicked off this site)  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2016-08-03. Score: 6
  
  Steve Omohudro wrote a paper called [Basic AI Drives](https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf) that steps through why we would expect an AI with narrow goals to find some basic, general concepts as instrumentally useful for their narrow goals.
  
  For example, an AI designed to maximize stock market returns but whose design is silent on the importance of continuing to survive would realize that its continued survival is a key component of maximizing stock market returns, and thus take actions to keep itself operational.
  
  In general, we should be skeptical of 'anthropomorphizing' AI and other code, but it seems like there *are* reasons to expect this beyond "well, humans behave this way, so it must be how all intelligence behaves."
  
  > **Comments:**
  > **username_1**: I assume Omohudru uses "maximize returns" in the context of minimax, where mitigation of downside is part of the equation.  From my perspective, "win else survive" is a basic extension of minimax in cyclic games.  
  
  ---
  
  ### Answer by username_3 on 2016-08-03. Score: 5
  
  The concept of 'survival instinct' probably falls in the category of what Marvin Minsky would call a 'suitcase word', i.e. it packages together a number of related phenomena into what at first appears to be a singular notion.
  
  So it's quite possible that we can construct mechanisms that have the appearance of some kind of 'hard-coded' survival instinct, without that ever featuring as an explicit rule(s) in the design.
  
  See the beautiful little book ['Vehicles'](https://mitpress.mit.edu/books/vehicles) by the neuroanatomist Valentino Braitenberg for a compelling narrative of how such 'top down' concepts as 'survival instinct' might evolve 'from the bottom up'.
  
  Also, trying to ensure that intelligent artefacts place too high a priority on their survival might easily lead to a [Killbot Hellscape](https://xkcd.com/1613/).
  
  
  ---
  
  ### Answer by username_4 on 2017-04-22. Score: 2
  
  I will answer by quoting the book *The Myth of Sisyphus*:
  
  > There is only one really serious philosophical question, and that is suicide.
  
  So, probably, we need *some* degree of a survival instinct if we don't want our AGIs to "terminate themselves" (whatever that means), whenever they get existential.
  
  
  ---
  
  