Answers.Body: <blockquote>
    <p>Is it still a policy iteration algorithm if the policy is updated optimizing a function of the immediate reward instead of the value function?</p>
  </blockquote>
  
  <p>Technically yes.</p>
  
  <p>The value update step in Policy Iteration is:</p>
  
  <p><span class="math-container">$$v(s) \leftarrow \sum_{r,s'}p(r,s'|s,\pi(s))(r + \gamma v(s'))$$</span></p>
  
  <p>The discount factor <span class="math-container">$\gamma$</span> can be set to <span class="math-container">$0$</span>, making the update:</p>
  
  <p><span class="math-container">$$v(s) \leftarrow \sum_{r,s'}p(r,s'|s,\pi(s))r$$</span></p>
  
  <p>However, there are two key details that are important, and make this a <em>technically</em> yes, not some alternative way of solving problems:</p>
  
  <ul>
  <li><p>Changing discount factor <span class="math-container">$\gamma$</span> changes what it means for an agent to act optimally. Setting it to zero means that the agent will prioritise only its immediate reward signal, and make no long-term decisions at all. This would be useless for instance if the task was to escape a maze in minimum time.</p></li>
  <li><p>Technically there is still a value function being updated. The function <span class="math-container">$v(s)$</span> is still the expected future reward, just we have set it to only care a very short step into the future. So short that it doesn't care what the value of the next state is, so the next state does not appear in any updates.</p></li>
  <li><p>Due to the lack of bootstrapping between states, <em>all</em> the data for optimal behaviour is already available in the reward distribution. So the entire MDP can be solved with a single sweep through all the states. Or it could just be solved on-demand using <span class="math-container">$\pi(s) = \text{argmax}_a[\sum_{r,s'}p(r,s'|s,a)r]$</span> for any state, making a policy iteration process redundant.</p></li>
  </ul>
  
  <p>However, with these caveats in mind, yes this is still policy iteration. It is the <em>same</em> update process, just with a particular choice of one of the parameters.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-22T07:25:16.477Z
Answers.Text: Here is a [related question](https://ai.stackexchange.com/q/20527/35821), you might find interesting! Thank you for the detailed answer here!
Answers.UserDisplayName: Ray Walker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-11T18:51:05.097Z
Answers.LastActivityDate: 2019-09-11T18:57:37.010Z
Answers.LastEditDate: 2019-09-11T18:57:37.010Z
Answers.UserDisplayName: Neil Slater
Body: <p>Is it still a policy iteration algorithm if the policy is updated optimizing a function of the immediate reward instead of the value function?</p>
  
Tags: <reinforcement-learning><value-iteration><policy-iteration>
Title: Can policy iteration use only the immediate reward for updates?
UserDisplayName: qiang li
ThreadText: # Can policy iteration use only the immediate reward for updates?
  
  (Asked by: username_0 on 2019-09-11)
  
  Is it still a policy iteration algorithm if the policy is updated optimizing a function of the immediate reward instead of the value function?
  
  **Tags:** reinforcement-learning, value-iteration, policy-iteration
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-09-11. Score: 2 | Accepted answer
  
  > Is it still a policy iteration algorithm if the policy is updated optimizing a function of the immediate reward instead of the value function?
  
  Technically yes.
  
  The value update step in Policy Iteration is:
  
  $$v(s) \leftarrow \sum\_{r,s'}p(r,s'|s,\pi(s))(r + \gamma v(s'))$$
  
  The discount factor $\gamma$ can be set to $0$, making the update:
  
  $$v(s) \leftarrow \sum\_{r,s'}p(r,s'|s,\pi(s))r$$
  
  However, there are two key details that are important, and make this a *technically* yes, not some alternative way of solving problems:
  
  * Changing discount factor $\gamma$ changes what it means for an agent to act optimally. Setting it to zero means that the agent will prioritise only its immediate reward signal, and make no long-term decisions at all. This would be useless for instance if the task was to escape a maze in minimum time.
  * Technically there is still a value function being updated. The function $v(s)$ is still the expected future reward, just we have set it to only care a very short step into the future. So short that it doesn't care what the value of the next state is, so the next state does not appear in any updates.
  * Due to the lack of bootstrapping between states, *all* the data for optimal behaviour is already available in the reward distribution. So the entire MDP can be solved with a single sweep through all the states. Or it could just be solved on-demand using $\pi(s) = \text{argmax}\_a[\sum\_{r,s'}p(r,s'|s,a)r]$ for any state, making a policy iteration process redundant.
  
  However, with these caveats in mind, yes this is still policy iteration. It is the *same* update process, just with a particular choice of one of the parameters.
  
  > **Comments:**
  > **username_2**: Here is a [related question](https://ai.stackexchange.com/q/20527/35821), you might find interesting! Thank you for the detailed answer here!  
  
  ---
  
  