Answers.Body: <p>In the paper that you cite, <a href="https://papers.nips.cc/paper/7253-inverse-reward-design" rel="nofollow noreferrer">Inverse Reward Design</a> (2017), the authors actually define what they mean by &quot;proxy reward function&quot;.</p>
  <blockquote>
  <p>We formalize this in a probabilistic model that relates the <strong>proxy (designed) reward</strong> to the true reward</p>
  </blockquote>
  <p>So, the <strong>proxy</strong> reward function is the reward function designed by the human, which may not necessarily be the reward function that he/she intended (i.e. it may be a misspecified reward function), given that the human may have forgotten to model/incorporate certain (unpredicted by the human) scenarios or situations that the agent may face. This usage of the word &quot;proxy&quot; is thus consistent with the general usage of the word in computer science, i.e. a &quot;proxy reward function&quot; is a reward function that is used <strong>instead of</strong> the intended (optimal) reward function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-14T16:18:37.303Z
Answers.LastActivityDate: 2020-10-14T16:18:37.303Z
Answers.UserDisplayName: nbro
Body: <p>The understanding I have is that they somehow adjust the objective to make it easier to meet, without changing the reward function.</p>
  <blockquote>
  <p>... the observed proxy reward function is the approximate solution to a reward design problem</p>
  </blockquote>
  <p>(source: <a href="https://arxiv.org/pdf/1711.02827.pdf" rel="nofollow noreferrer">Inverse Reward Design</a>)</p>
  <p>But I have trouble getting how they fit the overall reward objective and got confused by some examples of them.
  I had the idea of them being small reward functions (as in the case of solving for sparse rewards) eventually leading to the main goal. But the statement below, from <a href="https://ai-alignment.com/thoughts-on-reward-engineering-82b193ec03f6" rel="nofollow noreferrer">this post</a>, made me question that.</p>
  <blockquote>
  <p>Typical examples of proxy reward functions include “partial credit” for behaviors that look promising; artificially high discount rates and careful reward shaping;...</p>
  </blockquote>
  <ol>
  <li><p>What are they, and how would one go about identifying and integrating proxy rewards in an RL problem?</p>
  </li>
  <li><p>In the examples above, how would high discount rates form a proxy reward?</p>
  </li>
  </ol>
  <p>I'm also curious about how they are used as a source of multiple rewards</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-22T11:24:18.027Z
Comments.Text: Maybe have a look at https://ai.stackexchange.com/q/20040/2444, although I think that you know what "proxy" usually means.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-23T05:05:15.407Z
Comments.Text: So "proxies" are intermediaries. This helped add clarity into the "what are they" part of the question
Comments.UserDisplayName: mugoh
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-16T18:17:30.617Z
Comments.Text: Please, read my answer below https://ai.stackexchange.com/a/24074/2444, if you haven't yet. You should accept it, if it answers your question. I read that IRD paper and other related papers, so I'm pretty confident that the answer is correct. If something is still unclear, let me know. Please, take a look at https://ai.stackexchange.com/help/someone-answers for more details.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-17T04:13:19.063Z
Comments.Text: Thanks for pointing this out  @nbro. I was also seeking clarity on how high discount rates form proxy rewards and hiw proxies can be used as a source of multiple rewards
Comments.UserDisplayName: mugoh
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-17T18:34:16.423Z
Comments.Text: It's not indeed clear what the article's author meant by "artificially high discount rates". It's not even clear if they are referring to the discount factor of the RL algorithm. Even if that was the case, it's not very clear why that would be a "proxy reward function". The discount factor is the way to weight short-term and long-term rewards, but it's not very clear why one would consider that a "proxy reward function". Maybe this has to do with the "long time horizon" part of that article. Maybe the best thing to do is e-mail the article's author and ask him to answer your other questions.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-17T18:39:34.437Z
Comments.Text: Regarding how multiple "proxies" can be used as a source for multiple rewards, I think that the idea is that you can specify partial reward functions, i.e. reward functions that specify some type of partial behaviour that you want to agent to have. So, for instance, if you want to train an agent to play football/soccer, you may have a proxy reward function that specifies how the agent is supposed to kick the ball and another that specifies how the agent needs to move in the field. This is at least the idea that I have after having read that article.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><definitions><papers><reward-functions>
Title: What are proxy reward functions?
UserDisplayName: mugoh
ThreadText: # What are proxy reward functions?
  
  (Asked by: username_0 on 2020-09-20)
  
  The understanding I have is that they somehow adjust the objective to make it easier to meet, without changing the reward function.
  
  > ... the observed proxy reward function is the approximate solution to a reward design problem
  
  (source: [Inverse Reward Design](https://arxiv.org/pdf/1711.02827.pdf))
  
  But I have trouble getting how they fit the overall reward objective and got confused by some examples of them.
  I had the idea of them being small reward functions (as in the case of solving for sparse rewards) eventually leading to the main goal. But the statement below, from [this post](https://ai-alignment.com/thoughts-on-reward-engineering-82b193ec03f6), made me question that.
  
  > Typical examples of proxy reward functions include “partial credit” for behaviors that look promising; artificially high discount rates and careful reward shaping;...
  
  1. What are they, and how would one go about identifying and integrating proxy rewards in an RL problem?
  2. In the examples above, how would high discount rates form a proxy reward?
  
  I'm also curious about how they are used as a source of multiple rewards
  
  **Tags:** reinforcement-learning, definitions, papers, reward-functions
  
  > **Comments:**
  > **username_1**: Maybe have a look at https://ai.stackexchange.com/q/20040/2444, although I think that you know what "proxy" usually means.  
  > **username_0**: So "proxies" are intermediaries. This helped add clarity into the "what are they" part of the question  
  > **username_1**: Please, read my answer below https://ai.stackexchange.com/a/24074/2444, if you haven't yet. You should accept it, if it answers your question. I read that IRD paper and other related papers, so I'm pretty confident that the answer is correct. If something is still unclear, let me know. Please, take a look at https://ai.stackexchange.com/help/someone-answers for more details.  
  > **username_0**: Thanks for pointing this out  @username_1. I was also seeking clarity on how high discount rates form proxy rewards and hiw proxies can be used as a source of multiple rewards  
  > **username_1**: It's not indeed clear what the article's author meant by "artificially high discount rates". It's not even clear if they are referring to the discount factor of the RL algorithm. Even if that was the case, it's not very clear why that would be a "proxy reward function". The discount factor is the way to weight short-term and long-term rewards, but it's not very clear why one would consider that a "proxy reward function". Maybe this has to do with the "long time horizon" part of that article. Maybe the best thing to do is e-mail the article's author and ask him to answer your other questions.  
  > **username_1**: Regarding how multiple "proxies" can be used as a source for multiple rewards, I think that the idea is that you can specify partial reward functions, i.e. reward functions that specify some type of partial behaviour that you want to agent to have. So, for instance, if you want to train an agent to play football/soccer, you may have a proxy reward function that specifies how the agent is supposed to kick the ball and another that specifies how the agent needs to move in the field. This is at least the idea that I have after having read that article.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-10-14. Score: 1 | Accepted answer
  
  In the paper that you cite, [Inverse Reward Design](https://papers.nips.cc/paper/7253-inverse-reward-design) (2017), the authors actually define what they mean by "proxy reward function".
  
  > We formalize this in a probabilistic model that relates the **proxy (designed) reward** to the true reward
  
  So, the **proxy** reward function is the reward function designed by the human, which may not necessarily be the reward function that he/she intended (i.e. it may be a misspecified reward function), given that the human may have forgotten to model/incorporate certain (unpredicted by the human) scenarios or situations that the agent may face. This usage of the word "proxy" is thus consistent with the general usage of the word in computer science, i.e. a "proxy reward function" is a reward function that is used **instead of** the intended (optimal) reward function.
  
  
  ---
  
  