Answers.Body: <p>The reward likely should be based on the dynamics of the environment not the intention of the agent.</p>
  <p>Imagine if <span class="math-container">$r_2$</span> was a large negative reward and <span class="math-container">$p_2 &gt; p_1$</span>.  This should be captured in the value of the state.  This way as the values back up the agent can learn a path around a risky set of states.</p>
  <p>The most prominent example of this I can think of is epsilon greedy on the cliff grid word.  The agent under high settings of epsilon learns a further path around the cliff to avoid epsilon attempting to explore off of it.  It may have the best intention of walk close to the cliff but under the same epsilon, the agent which is aware of the risk and takes the safer path has a higher average reward.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-08T19:38:57.357Z
Answers.LastActivityDate: 2024-04-08T19:38:57.357Z
Answers.UserDisplayName: foreverska
Answers.Body: <p>Depends on the setting, many times you don't know the underlying transition matrix, and thus we resort to a monte carlo estimate of the
  <span class="math-container">$$
  \mathbb{E}[\sum \gamma^i r_i]
  $$</span></p>
  <p>By running multiple simulations</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-08T18:07:47.763Z
Answers.LastActivityDate: 2024-04-08T18:55:43.683Z
Answers.LastEditDate: 2024-04-08T18:55:43.683Z
Answers.UserDisplayName: Alberto
Body: <p>Assume the agent is in a state <span class="math-container">$s$</span> and takes action <span class="math-container">$a$</span>, intending to move to <span class="math-container">$s_1$</span>, however the dynamics may drop the agent in <span class="math-container">$s_1$</span> or <span class="math-container">$s_2$</span>: may move to <span class="math-container">$s_1$</span> and receive reward <span class="math-container">$r_1$</span> with a probability <span class="math-container">$p_1$</span>, or move to state <span class="math-container">$s_2$</span> and receive reward <span class="math-container">$r_2$</span> with probability <span class="math-container">$p_2$</span>.</p>
  <p>When calculating <span class="math-container">$v_{k+1}$</span> (Sutton eq. 4.4), is it:</p>
  <p>(1)  <span class="math-container">$\pi(a|s) * [p_1 * (r_1 * \gamma * s_1)+ p_2*(r_2 * \gamma * s_2)]$</span></p>
  <p>(2)  <span class="math-container">$\pi(a|s) * [p_1 * (r_1 * \gamma * s_1)+ p_2*(r_1 * \gamma * s_2)]$</span></p>
  <p>Ie is the reward based on the intention of the agent or the state the agent ends up in?</p>
  
Tags: <reinforcement-learning><value-functions><policy-iteration>
Title: In reinforcement learning policy evaluation, which reward is received?
UserDisplayName: Nat
ThreadText: # In reinforcement learning policy evaluation, which reward is received?
  
  (Asked by: username_0 on 2024-04-08)
  
  Assume the agent is in a state $s$ and takes action $a$, intending to move to $s\_1$, however the dynamics may drop the agent in $s\_1$ or $s\_2$: may move to $s\_1$ and receive reward $r\_1$ with a probability $p\_1$, or move to state $s\_2$ and receive reward $r\_2$ with probability $p\_2$.
  
  When calculating $v\_{k+1}$ (Sutton eq. 4.4), is it:
  
  (1) $\pi(a|s) \* [p\_1 \* (r\_1 \* \gamma \* s\_1)+ p\_2\*(r\_2 \* \gamma \* s\_2)]$
  
  (2) $\pi(a|s) \* [p\_1 \* (r\_1 \* \gamma \* s\_1)+ p\_2\*(r\_1 \* \gamma \* s\_2)]$
  
  Ie is the reward based on the intention of the agent or the state the agent ends up in?
  
  **Tags:** reinforcement-learning, value-functions, policy-iteration
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-04-08. Score: 1 | Accepted answer
  
  The reward likely should be based on the dynamics of the environment not the intention of the agent.
  
  Imagine if $r\_2$ was a large negative reward and $p\_2 > p\_1$. This should be captured in the value of the state. This way as the values back up the agent can learn a path around a risky set of states.
  
  The most prominent example of this I can think of is epsilon greedy on the cliff grid word. The agent under high settings of epsilon learns a further path around the cliff to avoid epsilon attempting to explore off of it. It may have the best intention of walk close to the cliff but under the same epsilon, the agent which is aware of the risk and takes the safer path has a higher average reward.
  
  
  ---
  
  ### Answer by username_2 on 2024-04-08. Score: 0
  
  Depends on the setting, many times you don't know the underlying transition matrix, and thus we resort to a monte carlo estimate of the
  $$
  \mathbb{E}[\sum \gamma^i r\_i]
  $$
  
  By running multiple simulations
  
  
  ---
  
  