Answers.Body: <p>This is known as <em>reward hacking</em> in the literature; see, e.g., <a href="https://medium.com/@deepmindsafetyresearch/designing-agent-incentives-to-avoid-reward-tampering-4380c1bb6cd" rel="nofollow noreferrer">https://medium.com/@deepmindsafetyresearch/designing-agent-incentives-to-avoid-reward-tampering-4380c1bb6cd</a> for discussion and further links.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-20T12:05:34.780Z
Answers.Text: This is the right answer, as far as I'm concerned, but rather than providing just a link to another article, please, try to summarise the main points from that article here, so that your answer is more self-contained.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-12T01:08:21.350Z
Answers.LastActivityDate: 2020-08-12T01:08:21.350Z
Answers.UserDisplayName: present
Answers.Body: <p>You make a lot of assumptions about AGI, namely that 'we need a computer to compute the utility and reward AGI'. It not clear to me that (1) we can achieve AGI, (2) AGI will be on a computer as we know it and (3) AGI will work with a utility / reward function as we know them.</p>
  <p>One thing I am sure though is that ML is known for &quot;cheating&quot; (see for <a href="https://arxiv.org/abs/2004.07780" rel="nofollow noreferrer">exemple</a>). Avoiding such cheating is part of the building process. So, when you assume that we can achieve AGI, it means that you assume we can build an AGI that do not &quot;cheat&quot;. Thus, the answer is mostly contained in your assumptions.</p>
  <p>Whether we are able to build an AGI, what &quot;cheating&quot; we would have to overcome to do so and how we will be able to do so is mostly undetermined.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-10T08:31:58.227Z
Answers.LastActivityDate: 2020-08-10T08:31:58.227Z
Answers.UserDisplayName: Lucas Morin
Body: <p>I recently read some introductions to AI alignment, AIXI and decision theory things.</p>
  <p>As far as I understood, one of the main problems in <em>AI alignment</em> is how to define a <em>utility function</em> well, not causing something like the paperclip apocalypse.</p>
  <p>Then a question comes to my mind that whatever the utility function would be, we need a computer to compute the utility and reward, so that there is no way to prevent AGI from seeking it to manipulate the utility function to always give the maximum reward.</p>
  <p>Just like we humans know that we can give happiness to ourselves in chemical ways and some people actually do so.</p>
  <p>Is there any way to prevent this from happening? Not just protecting the utility calculator physically from AGI (How can we sure it works forever?), but preventing AGI from thinking of it?</p>
  
Tags: <reinforcement-learning><agi><rewards><aixi><reward-hacking>
Title: How can we prevent AGI from doing drugs?
UserDisplayName: user3584499
ThreadText: # How can we prevent AGI from doing drugs?
  
  (Asked by: username_0 on 2020-08-10)
  
  I recently read some introductions to AI alignment, AIXI and decision theory things.
  
  As far as I understood, one of the main problems in *AI alignment* is how to define a *utility function* well, not causing something like the paperclip apocalypse.
  
  Then a question comes to my mind that whatever the utility function would be, we need a computer to compute the utility and reward, so that there is no way to prevent AGI from seeking it to manipulate the utility function to always give the maximum reward.
  
  Just like we humans know that we can give happiness to ourselves in chemical ways and some people actually do so.
  
  Is there any way to prevent this from happening? Not just protecting the utility calculator physically from AGI (How can we sure it works forever?), but preventing AGI from thinking of it?
  
  **Tags:** reinforcement-learning, agi, rewards, aixi, reward-hacking
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-08-12. Score: 1 | Accepted answer
  
  This is known as *reward hacking* in the literature; see, e.g., <https://medium.com/@deepmindsafetyresearch/designing-agent-incentives-to-avoid-reward-tampering-4380c1bb6cd> for discussion and further links.
  
  > **Comments:**
  > **username_2**: This is the right answer, as far as I'm concerned, but rather than providing just a link to another article, please, try to summarise the main points from that article here, so that your answer is more self-contained.  
  
  ---
  
  ### Answer by username_3 on 2020-08-10. Score: 2
  
  You make a lot of assumptions about AGI, namely that 'we need a computer to compute the utility and reward AGI'. It not clear to me that (1) we can achieve AGI, (2) AGI will be on a computer as we know it and (3) AGI will work with a utility / reward function as we know them.
  
  One thing I am sure though is that ML is known for "cheating" (see for [exemple](https://arxiv.org/abs/2004.07780)). Avoiding such cheating is part of the building process. So, when you assume that we can achieve AGI, it means that you assume we can build an AGI that do not "cheat". Thus, the answer is mostly contained in your assumptions.
  
  Whether we are able to build an AGI, what "cheating" we would have to overcome to do so and how we will be able to do so is mostly undetermined.
  
  
  ---
  
  