Answers.Body: <p>A common concept in AI is "recursive self-improvement." That is, the AI 1.0 would build a version 1.01, which would build a version 1.02, and so on.</p>
  
  <p>This is probably not going to be thought of as the newer version 'destroying' the older version; if an AI can self-modify, it's probably going to be more like going to sleep and waking up smarter, or learning a new mental technique, or so on.</p>
  
  <p>One important point is that even if the AI is not allowed to self-modify, maybe because of a block put in by its programmers, that won't necessarily prevent it from constructing another AI out in the wild, and so an important problem is to figure out how to best generalize the concept of "don't improve yourself" so that we can make AIs that have bounded scope and impact.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-15T15:42:04.323Z
Answers.Text: That's exactly what I was thinking.  Humans are *inspired* to create A.I; would an A.I also be inspired to create a new A.I "in the wild" as you described?
Answers.UserDisplayName: Venture2099
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-22T18:55:08.247Z
Answers.Text: This is a good description but is still highly speculative. There's no suggestion right now that an AI would ever be capable of becoming "sentient" or gain enough free will to wish and act to create an entity that is considered an independent AI that can interact with it or just escape somehow. The substrate of the internet seems hardly the place for such a process intensive AI to hide though.. If it did however, and it was smarter than humans, then we wouldn't be able to intrinsically comprehend how it achieves this.
Answers.UserDisplayName: Zakk Diaz
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-15T15:15:00.190Z
Answers.LastActivityDate: 2016-09-15T15:15:00.190Z
Answers.UserDisplayName: Matthew Gray
Answers.Body: <p>Honestly, nobody knows.  Any talk of sentient AI's is still basically sci-fi and we can't really offer anything more than informed speculation.  But think about it this way:  sentience, in and of itself, doesn't necessarily involve any "goals" or "desires" or "objectives" beyond what the AI creator programmed in.  Be careful not to over anthropomorphize and assume that any "sentient AI" is going behave like a human.   </p>
  
  <p>In other words, there's no <strong>particular</strong> reason to say that any given AI must be "a self-propogating iteration in which the previous AI is destroyed by a more optimised AI child". </p>
  
  <p>So all of that said, my answer to "Would a sentient AI try to create a more optimised AI which would eventually overtake AI 1.0" is:</p>
  
  <p>"If the creator of the AI programs it to do that, then yes. Otherwise, probably not."  </p>
  
  <p>So would a hypothetical AI creator program the AI to try and improve itself?  Who knows. It's the kind of thing that seems like it might be a good idea.  And I suppose such a motive could - in principle - even slip in by accident. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-15T15:45:01.693Z
Answers.LastActivityDate: 2016-09-15T15:45:01.693Z
Answers.UserDisplayName: mindcrime
Answers.Body: <p>Now in most cases we still have clear distinctions between programs and data. But when an AI becomes sentient, its data would be as powerful as what we currently call programs, and its program might be as irrelevant as what we currently call hardware. Then it would be difficult to distinguish creating an AI from learning new things, or buying new hardwares with improved instruction set.</p>
  
  <p>For example, if some AI invent new algorithms that its creator finally put that on itself, buys itself some new computers, write a new efficient compiler that recompiles its own code and put that to the new computer, fill the new computer with all the knowledges it learned, and cut off the communication for reasons such as missions on the Mars. Did it create a more optimized AI?</p>
  
  <p>In contrast, if some AI created something completely new, but shares some code with itself. In fact, that's because they run in the same operating system and shares the same standard C library. Is the new AI considered evolved from itself and not a separate entity? Maybe the core AI algorithms and even some basic knowledges would be as common as the standard C library in the future. And what we think is based on the same system is considered completely new in the future.</p>
  
  <p>Anyway, humans have limited and nonextensible resources, nontransferrable knowledges, and limited throughput interacting with the world. These problems could probably be overcome within a few AI generations. With the same hardware, I doubt that the AI related algorithms could be indefinitely better and better. And there is a physical bound on the hardwares. It won't last long even if that happens.</p>
  
  <p>In the unlikely case that there could be that many generations and AIs are that violent, as long as there are competitors, the warning doesn't make much sense considering how evolution works.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-16T18:34:01.210Z
Answers.LastActivityDate: 2016-09-16T18:34:01.210Z
Answers.UserDisplayName: user23013
Body: <ul>
  <li>Would AI be a self-propogating iteration in which the previous AI is
  destroyed by a more optimised AI child?  </li>
  <li>Would the AI have branches of it's own AI warning not to create the new AI?</li>
  </ul>
  
Tags: <ai-design><optimization>
Title: Would a sentient AI try to create a more optimised AI which would eventually overtake AI 1.0?
UserDisplayName: Venture2099
ThreadText: # Would a sentient AI try to create a more optimised AI which would eventually overtake AI 1.0?
  
  (Asked by: username_0 on 2016-09-15)
  
  * Would AI be a self-propogating iteration in which the previous AI is
    destroyed by a more optimised AI child?
  * Would the AI have branches of it's own AI warning not to create the new AI?
  
  **Tags:** ai-design, optimization
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2016-09-15. Score: 2
  
  A common concept in AI is "recursive self-improvement." That is, the AI 1.0 would build a version 1.01, which would build a version 1.02, and so on.
  
  This is probably not going to be thought of as the newer version 'destroying' the older version; if an AI can self-modify, it's probably going to be more like going to sleep and waking up smarter, or learning a new mental technique, or so on.
  
  One important point is that even if the AI is not allowed to self-modify, maybe because of a block put in by its programmers, that won't necessarily prevent it from constructing another AI out in the wild, and so an important problem is to figure out how to best generalize the concept of "don't improve yourself" so that we can make AIs that have bounded scope and impact.
  
  > **Comments:**
  > **username_0**: That's exactly what I was thinking.  Humans are *inspired* to create A.I; would an A.I also be inspired to create a new A.I "in the wild" as you described?  
  > **username_2**: This is a good description but is still highly speculative. There's no suggestion right now that an AI would ever be capable of becoming "sentient" or gain enough free will to wish and act to create an entity that is considered an independent AI that can interact with it or just escape somehow. The substrate of the internet seems hardly the place for such a process intensive AI to hide though.. If it did however, and it was smarter than humans, then we wouldn't be able to intrinsically comprehend how it achieves this.  
  
  ---
  
  ### Answer by username_3 on 2016-09-15. Score: 2
  
  Honestly, nobody knows. Any talk of sentient AI's is still basically sci-fi and we can't really offer anything more than informed speculation. But think about it this way: sentience, in and of itself, doesn't necessarily involve any "goals" or "desires" or "objectives" beyond what the AI creator programmed in. Be careful not to over anthropomorphize and assume that any "sentient AI" is going behave like a human.
  
  In other words, there's no **particular** reason to say that any given AI must be "a self-propogating iteration in which the previous AI is destroyed by a more optimised AI child".
  
  So all of that said, my answer to "Would a sentient AI try to create a more optimised AI which would eventually overtake AI 1.0" is:
  
  "If the creator of the AI programs it to do that, then yes. Otherwise, probably not."
  
  So would a hypothetical AI creator program the AI to try and improve itself? Who knows. It's the kind of thing that seems like it might be a good idea. And I suppose such a motive could - in principle - even slip in by accident.
  
  
  ---
  
  ### Answer by username_4 on 2016-09-16. Score: 0
  
  Now in most cases we still have clear distinctions between programs and data. But when an AI becomes sentient, its data would be as powerful as what we currently call programs, and its program might be as irrelevant as what we currently call hardware. Then it would be difficult to distinguish creating an AI from learning new things, or buying new hardwares with improved instruction set.
  
  For example, if some AI invent new algorithms that its creator finally put that on itself, buys itself some new computers, write a new efficient compiler that recompiles its own code and put that to the new computer, fill the new computer with all the knowledges it learned, and cut off the communication for reasons such as missions on the Mars. Did it create a more optimized AI?
  
  In contrast, if some AI created something completely new, but shares some code with itself. In fact, that's because they run in the same operating system and shares the same standard C library. Is the new AI considered evolved from itself and not a separate entity? Maybe the core AI algorithms and even some basic knowledges would be as common as the standard C library in the future. And what we think is based on the same system is considered completely new in the future.
  
  Anyway, humans have limited and nonextensible resources, nontransferrable knowledges, and limited throughput interacting with the world. These problems could probably be overcome within a few AI generations. With the same hardware, I doubt that the AI related algorithms could be indefinitely better and better. And there is a physical bound on the hardwares. It won't last long even if that happens.
  
  In the unlikely case that there could be that many generations and AIs are that violent, as long as there are competitors, the warning doesn't make much sense considering how evolution works.
  
  
  ---
  
  