Answers.Body: <p>Yes, the two update equations are equivalent. As an aside, technically the equation you give is not the Bellman equation, but the update step re-written as an equation - in the Bellman equation instead of <span class="math-container">$v_{k+1}(s)$</span> or <span class="math-container">$v_{k}(s)$</span> (showing iterations of approximate value functions), you would have <span class="math-container">$v_{\pi}(s)$</span> (representing the true value of a state under policy <span class="math-container">$\pi$</span>).</p>
  <p>The difference between the equations is that</p>
  <ul>
  <li><p>In the first case of Policy Evaluation, in order to be general, a stochastic policy <span class="math-container">$\pi(a|s): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R} = Pr\{A_t = a|S_t =s\}$</span> is used. That means to get the expected value, you must sum over all possible actions <span class="math-container">$a$</span> and weight them by the policy function output.</p>
  </li>
  <li><p>In the case of Policy Iteration, a deterministic policy <span class="math-container">$\pi(s): \mathcal{S} \rightarrow \mathcal{A}$</span> is used. For that, you don't need to know all possible values of <span class="math-container">$a$</span> for probabilities, but use the output of the policy function directly as the action that is taken by the agent. That action therefore has a probability of <span class="math-container">$1$</span> of being chosen by the policy in the given state.</p>
  </li>
  </ul>
  <p>The equation used in Policy Iteration is simplified for a deterministic policy. If you want you could represent the policy using <span class="math-container">$\pi(a|s)$</span> and use the same equation as for Policy Evaluation. If you do that, you would also need to alter the Policy Improvement policy update step to something like:</p>
  <blockquote>
  <p><span class="math-container">$a_{max} \leftarrow \text{argmax}_a\sum_{r,s'}p(r,s'|s,a)[r + \gamma V(s')]$</span></p>
  <p><span class="math-container">$\text{ for each } a \in \mathcal{A(s)}$</span>:</p>
  <p><span class="math-container">$\qquad \pi(a|s) \leftarrow 1 \text{ if } a = a_{max}, 0 \text{ otherwise }$</span></p>
  </blockquote>
  <p>Doing this will result in exactly the same value function and policy as before. The only reason to do this would be to show the equivalence between the two sets of update equations when dealing with a deterministic policy.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-19T07:38:56.250Z
Answers.LastActivityDate: 2020-05-19T13:00:18.587Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: Neil Slater
Body: <p>In the textbook "Reinforcement Learning: An Introduction", by Richard Sutton and Andrew Barto, the pseudo code for Policy Evaluation is given as follows:</p>
  
  <p><a href="https://i.sstatic.net/nGbDN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/nGbDN.png" alt="enter image description here"></a></p>
  
  <p>The update equation for <span class="math-container">$V(s)$</span> comes from the Bellman equation for <span class="math-container">$v_{\pi}(s)$</span> which is mentioned below (the update equation) for your convenience:
  <span class="math-container">$$v_{k+1}(s) = \sum_{a} \pi(a|s)\sum_{s',r}p(s',r|s,a)[r+\gamma v_{k}(s')]$$</span></p>
  
  <p>Now, in Policy Iteration, the Policy Evaluation comes in stage 2, as mentioned in the following pseudo code:</p>
  
  <p><a href="https://i.sstatic.net/HEbLW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/HEbLW.png" alt="enter image description here"></a></p>
  
  <p>Here, in the policy Evaluation stage, <span class="math-container">$V(s)$</span> is updated using a different equation:
  <span class="math-container">$$\begin{align} v_{k+1}(s) = \sum_{s',r}p(s',r|s,\pi (s))[r + \gamma v_{k}(s)] \end{align}$$</span> where <span class="math-container">$a = \pi(s)$</span> is used. </p>
  
  <p>Can someone please help me in understanding why this change is made in Policy Iteration? Are the two equations the same?</p>
  
Tags: <reinforcement-learning><comparison><policy-iteration><policy-evaluation><dynamic-programming>
Title: Why is update rule of the value function different in policy evaluation and policy iteration?
UserDisplayName: Jarvis1997
ThreadText: # Why is update rule of the value function different in policy evaluation and policy iteration?
  
  (Asked by: username_0 on 2020-05-19)
  
  In the textbook "Reinforcement Learning: An Introduction", by Richard Sutton and Andrew Barto, the pseudo code for Policy Evaluation is given as follows:
  
  [![enter image description here](https://i.sstatic.net/nGbDN.png)](https://i.sstatic.net/nGbDN.png)
  
  The update equation for $V(s)$ comes from the Bellman equation for $v\_{\pi}(s)$ which is mentioned below (the update equation) for your convenience:
  $$v\_{k+1}(s) = \sum\_{a} \pi(a|s)\sum\_{s',r}p(s',r|s,a)[r+\gamma v\_{k}(s')]$$
  
  Now, in Policy Iteration, the Policy Evaluation comes in stage 2, as mentioned in the following pseudo code:
  
  [![enter image description here](https://i.sstatic.net/HEbLW.png)](https://i.sstatic.net/HEbLW.png)
  
  Here, in the policy Evaluation stage, $V(s)$ is updated using a different equation:
  $$\begin{align} v\_{k+1}(s) = \sum\_{s',r}p(s',r|s,\pi (s))[r + \gamma v\_{k}(s)] \end{align}$$ where $a = \pi(s)$ is used.
  
  Can someone please help me in understanding why this change is made in Policy Iteration? Are the two equations the same?
  
  **Tags:** reinforcement-learning, comparison, policy-iteration, policy-evaluation, dynamic-programming
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-19. Score: 4 | Accepted answer
  
  Yes, the two update equations are equivalent. As an aside, technically the equation you give is not the Bellman equation, but the update step re-written as an equation - in the Bellman equation instead of $v\_{k+1}(s)$ or $v\_{k}(s)$ (showing iterations of approximate value functions), you would have $v\_{\pi}(s)$ (representing the true value of a state under policy $\pi$).
  
  The difference between the equations is that
  
  * In the first case of Policy Evaluation, in order to be general, a stochastic policy $\pi(a|s): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R} = Pr\{A\_t = a|S\_t =s\}$ is used. That means to get the expected value, you must sum over all possible actions $a$ and weight them by the policy function output.
  * In the case of Policy Iteration, a deterministic policy $\pi(s): \mathcal{S} \rightarrow \mathcal{A}$ is used. For that, you don't need to know all possible values of $a$ for probabilities, but use the output of the policy function directly as the action that is taken by the agent. That action therefore has a probability of $1$ of being chosen by the policy in the given state.
  
  The equation used in Policy Iteration is simplified for a deterministic policy. If you want you could represent the policy using $\pi(a|s)$ and use the same equation as for Policy Evaluation. If you do that, you would also need to alter the Policy Improvement policy update step to something like:
  
  > $a\_{max} \leftarrow \text{argmax}\_a\sum\_{r,s'}p(r,s'|s,a)[r + \gamma V(s')]$
  >
  > $\text{ for each } a \in \mathcal{A(s)}$:
  >
  > $\qquad \pi(a|s) \leftarrow 1 \text{ if } a = a\_{max}, 0 \text{ otherwise }$
  
  Doing this will result in exactly the same value function and policy as before. The only reason to do this would be to show the equivalence between the two sets of update equations when dealing with a deterministic policy.
  
  
  ---
  
  