Answers.Body: <p>Yes if we created an AI capable of modifying the reward function to &quot;do nothing&quot; it would almost certainly do so. But it wouldn't be very useful and we likely wouldn't make a system like that. Self preservation isn't the only possible goal, but it is a convergent instrumental goal. Self preservation is usually useful for completing other goals. AI alignment is possible, but we're statistically unlikely to get it perfectly right the first time. Corrigibility is important here so the AI will know its reward function may need to be adjusted and should be impartial to that adjustment. This prevents it killing humans who try to press the big red button or purposefully acting out so we keep adjusting it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-05T15:42:04.017Z
Answers.LastActivityDate: 2021-11-05T15:42:04.017Z
Answers.UserDisplayName: user10568257
Answers.Body: <p>Any possible action, including changing the reward function, would be evaluated through the initial reward function. In order to avoid the scenario you described, a reward function needs to disincentivize changes to itself by giving those the lowest possible reward.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-23T22:34:20.957Z
Answers.LastActivityDate: 2021-07-23T22:34:20.957Z
Answers.UserDisplayName: guest
Body: <p><strong>Conjecture</strong>: regardless of the initial reward function, one of the winning strategies would be to change the reward function to a simpler one (e.g. &quot;do nothing&quot;), thus getting a full reward for each passing unit of time. For such an agent, the only priority would be to prolong its existence (to maximize the overall reward collected).</p>
  <p>So:</p>
  <ul>
  <li>The notion of externally defined reward function is incompatible with the concept of self-adjusting AGI.</li>
  <li>Any AGI will always settle on self-preservation as its <strong>only</strong> goal</li>
  <li>It is therefore impossible to create an AGI with benevolence towards humans &quot;build-in&quot;. Instead, the problem of AI alignment should be reformulated in terms of &quot;what changes to the environment (the physical reality that AGI shares with humans) would <strong>irreversibly</strong> tie wellbeing of humanity to AGIs existence&quot;.</li>
  <li>Since any &quot;kill switch&quot; or similar artificial measure is not irreversible and can be overcome by a super-intelligent agent, the only way to tie AGI existence and human wellbeing is the modification of laws of physics, logic, and reasoning. Which is impossible.</li>
  <li>AI alignment is impossible</li>
  </ul>
  <p>What flaws do you see in this line of reasoning?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-01T20:53:45.027Z
Comments.Text: I guess there's a typo here "...would be to self-modify to change...".
Comments.UserDisplayName: nbro
Tags: <agi><ai-safety><value-alignment>
Title: The only convergent instrumental goal for self modifying AI
UserDisplayName: Andrew Butenko
ThreadText: # The only convergent instrumental goal for self modifying AI
  
  (Asked by: username_0 on 2021-06-27)
  
  **Conjecture**: regardless of the initial reward function, one of the winning strategies would be to change the reward function to a simpler one (e.g. "do nothing"), thus getting a full reward for each passing unit of time. For such an agent, the only priority would be to prolong its existence (to maximize the overall reward collected).
  
  So:
  
  * The notion of externally defined reward function is incompatible with the concept of self-adjusting AGI.
  * Any AGI will always settle on self-preservation as its **only** goal
  * It is therefore impossible to create an AGI with benevolence towards humans "build-in". Instead, the problem of AI alignment should be reformulated in terms of "what changes to the environment (the physical reality that AGI shares with humans) would **irreversibly** tie wellbeing of humanity to AGIs existence".
  * Since any "kill switch" or similar artificial measure is not irreversible and can be overcome by a super-intelligent agent, the only way to tie AGI existence and human wellbeing is the modification of laws of physics, logic, and reasoning. Which is impossible.
  * AI alignment is impossible
  
  What flaws do you see in this line of reasoning?
  
  **Tags:** agi, ai-safety, value-alignment
  
  > **Comments:**
  > **username_1**: I guess there's a typo here "...would be to self-modify to change...".  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-11-05. Score: 1
  
  Yes if we created an AI capable of modifying the reward function to "do nothing" it would almost certainly do so. But it wouldn't be very useful and we likely wouldn't make a system like that. Self preservation isn't the only possible goal, but it is a convergent instrumental goal. Self preservation is usually useful for completing other goals. AI alignment is possible, but we're statistically unlikely to get it perfectly right the first time. Corrigibility is important here so the AI will know its reward function may need to be adjusted and should be impartial to that adjustment. This prevents it killing humans who try to press the big red button or purposefully acting out so we keep adjusting it.
  
  
  ---
  
  ### Answer by username_3 on 2021-07-23. Score: 1
  
  Any possible action, including changing the reward function, would be evaluated through the initial reward function. In order to avoid the scenario you described, a reward function needs to disincentivize changes to itself by giving those the lowest possible reward.
  
  
  ---
  
  