Answers.Body: <p>The renowned book <a href="http://incompleteideas.net/book/RLbook2020.pdf" rel="nofollow noreferrer">Reinforcement Learning: An Introduction</a> (2nd edition), by Sutton and Barto, provides a different update rule than your first update rule for policy evaluation. Their update rule is more similar to your second update rule. See <a href="http://incompleteideas.net/book/RLbook2020.pdf#page=96" rel="nofollow noreferrer">section 4.1</a>. They also provide the pseudocode for policy evaluation on page 75 of the book. You can also find the pseudocode <a href="https://github.com/udacity/rl-cheatsheet/blob/master/cheatsheet.pdf" rel="nofollow noreferrer">here</a>. </p>
  
  <p>Moreover, note that the update rule doesn't need to change only because the rewards are tied to the outcome of an action. This information is encoded in the functions <span class="math-container">$p$</span> (the transition function) and <span class="math-container">$r$</span> (the reward function) of the Markov decision process, which is incorporated in the update rule. If you want to understand the update rule, you should read the relevant pages (especially, chapter 4) of the cited book.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-13T18:21:43.693Z
Answers.LastActivityDate: 2020-04-13T18:28:56.597Z
Answers.LastEditDate: 2020-04-13T18:28:56.597Z
Answers.UserDisplayName: nbro
Body: <p>I'm following Stanford reinforcement learning videos on youtube. One of the assignments asks to write code for policy evaluation for <a href="https://gym.openai.com/envs/FrozenLake-v0/" rel="nofollow noreferrer">Gym's FrozenLake-v0 environment</a>.</p>
  
  <p>In the course (and books I have seen), they define policy evaluation as</p>
  
  <p><span class="math-container">$$V^\pi_k(s)=r(s,\pi(s))+\gamma\sum_{s'}p(s'|s,\pi(s))V^\pi_{k-1}(s')$$</span></p>
  
  <p>My confusion is that in the frozen lake example, the reward is tied to the result of the action. So, for each pair state-action, I have a list that contains a possible next-state, the probability to get to that next-state and the reward. For example, being in the target state and performing any action brings a reward of <span class="math-container">$0$</span>, but being in any state that brings me to the target state gives me a reward of <span class="math-container">$1$</span>.</p>
  
  <p>Does this mean that, for this example, I need to rewrite <span class="math-container">$V^\pi_k(s)$</span> as something like this:</p>
  
  <p><span class="math-container">$$V^\pi_k(s)= \sum_{s'} p(s'|s,\pi(s)) [r(s,\pi(s), s')+ \gamma V^\pi_{k-1}(s')]$$</span></p>
  
Tags: <reinforcement-learning><math><implementation><gym><policy-evaluation>
Title: How can I implement policy evaluation when reward is tied to an action outcome?
UserDisplayName: Argod
ThreadText: # How can I implement policy evaluation when reward is tied to an action outcome?
  
  (Asked by: username_0 on 2020-04-13)
  
  I'm following Stanford reinforcement learning videos on youtube. One of the assignments asks to write code for policy evaluation for [Gym's FrozenLake-v0 environment](https://gym.openai.com/envs/FrozenLake-v0/).
  
  In the course (and books I have seen), they define policy evaluation as
  
  $$V^\pi\_k(s)=r(s,\pi(s))+\gamma\sum\_{s'}p(s'|s,\pi(s))V^\pi\_{k-1}(s')$$
  
  My confusion is that in the frozen lake example, the reward is tied to the result of the action. So, for each pair state-action, I have a list that contains a possible next-state, the probability to get to that next-state and the reward. For example, being in the target state and performing any action brings a reward of $0$, but being in any state that brings me to the target state gives me a reward of $1$.
  
  Does this mean that, for this example, I need to rewrite $V^\pi\_k(s)$ as something like this:
  
  $$V^\pi\_k(s)= \sum\_{s'} p(s'|s,\pi(s)) [r(s,\pi(s), s')+ \gamma V^\pi\_{k-1}(s')]$$
  
  **Tags:** reinforcement-learning, math, implementation, gym, policy-evaluation
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-13. Score: 1 | Accepted answer
  
  The renowned book [Reinforcement Learning: An Introduction](http://incompleteideas.net/book/RLbook2020.pdf) (2nd edition), by Sutton and Barto, provides a different update rule than your first update rule for policy evaluation. Their update rule is more similar to your second update rule. See [section 4.1](http://incompleteideas.net/book/RLbook2020.pdf#page=96). They also provide the pseudocode for policy evaluation on page 75 of the book. You can also find the pseudocode [here](https://github.com/udacity/rl-cheatsheet/blob/master/cheatsheet.pdf).
  
  Moreover, note that the update rule doesn't need to change only because the rewards are tied to the outcome of an action. This information is encoded in the functions $p$ (the transition function) and $r$ (the reward function) of the Markov decision process, which is incorporated in the update rule. If you want to understand the update rule, you should read the relevant pages (especially, chapter 4) of the cited book.
  
  
  ---
  
  