<blockquote>
  <p>Is it still a policy iteration algorithm if the policy is updated optimizing a function of the immediate reward instead of the value function?</p>
</blockquote>

<p>Technically yes.</p>

<p>The value update step in Policy Iteration is:</p>

<p><span class="math-container">$$v(s) \leftarrow \sum_{r,s'}p(r,s'|s,\pi(s))(r + \gamma v(s'))$$</span></p>

<p>The discount factor <span class="math-container">$\gamma$</span> can be set to <span class="math-container">$0$</span>, making the update:</p>

<p><span class="math-container">$$v(s) \leftarrow \sum_{r,s'}p(r,s'|s,\pi(s))r$$</span></p>

<p>However, there are two key details that are important, and make this a <em>technically</em> yes, not some alternative way of solving problems:</p>

<ul>
<li><p>Changing discount factor <span class="math-container">$\gamma$</span> changes what it means for an agent to act optimally. Setting it to zero means that the agent will prioritise only its immediate reward signal, and make no long-term decisions at all. This would be useless for instance if the task was to escape a maze in minimum time.</p></li>
<li><p>Technically there is still a value function being updated. The function <span class="math-container">$v(s)$</span> is still the expected future reward, just we have set it to only care a very short step into the future. So short that it doesn't care what the value of the next state is, so the next state does not appear in any updates.</p></li>
<li><p>Due to the lack of bootstrapping between states, <em>all</em> the data for optimal behaviour is already available in the reward distribution. So the entire MDP can be solved with a single sweep through all the states. Or it could just be solved on-demand using <span class="math-container">$\pi(s) = \text{argmax}_a[\sum_{r,s'}p(r,s'|s,a)r]$</span> for any state, making a policy iteration process redundant.</p></li>
</ul>

<p>However, with these caveats in mind, yes this is still policy iteration. It is the <em>same</em> update process, just with a particular choice of one of the parameters.</p>

CC BY-SA 4.0
2020-04-22T07:25:16.477Z
Here is a [related question](https://ai.stackexchange.com/q/20527/35821), you might find interesting! Thank you for the detailed answer here!
Ray Walker
CC BY-SA 4.0
2019-09-11T18:51:05.097Z
2019-09-11T18:57:37.010Z
2019-09-11T18:57:37.010Z
Neil Slater
<p>Is it still a policy iteration algorithm if the policy is updated optimizing a function of the immediate reward instead of the value function?</p>

<reinforcement-learning><value-iteration><policy-iteration>
Can policy iteration use only the immediate reward for updates?
qiang li
# Can policy iteration use only the immediate reward for updates?

(Asked by: username_0 on 2019-09-11)

Is it still a policy iteration algorithm if the policy is updated optimizing a function of the immediate reward instead of the value function?

**Tags:** reinforcement-learning, value-iteration, policy-iteration


---

## 1 answers

### Answer by username_1 on 2019-09-11. Score: 2 | Accepted answer

> Is it still a policy iteration algorithm if the policy is updated optimizing a function of the immediate reward instead of the value function?

Technically yes.

The value update step in Policy Iteration is:

$$v(s) \leftarrow \sum\_{r,s'}p(r,s'|s,\pi(s))(r + \gamma v(s'))$$

The discount factor $\gamma$ can be set to $0$, making the update:

$$v(s) \leftarrow \sum\_{r,s'}p(r,s'|s,\pi(s))r$$

However, there are two key details that are important, and make this a *technically* yes, not some alternative way of solving problems:

* Changing discount factor $\gamma$ changes what it means for an agent to act optimally. Setting it to zero means that the agent will prioritise only its immediate reward signal, and make no long-term decisions at all. This would be useless for instance if the task was to escape a maze in minimum time.
* Technically there is still a value function being updated. The function $v(s)$ is still the expected future reward, just we have set it to only care a very short step into the future. So short that it doesn't care what the value of the next state is, so the next state does not appear in any updates.
* Due to the lack of bootstrapping between states, *all* the data for optimal behaviour is already available in the reward distribution. So the entire MDP can be solved with a single sweep through all the states. Or it could just be solved on-demand using $\pi(s) = \text{argmax}\_a[\sum\_{r,s'}p(r,s'|s,a)r]$ for any state, making a policy iteration process redundant.

However, with these caveats in mind, yes this is still policy iteration. It is the *same* update process, just with a particular choice of one of the parameters.

> **Comments:**
> **username_2**: Here is a [related question](https://ai.stackexchange.com/q/20527/35821), you might find interesting! Thank you for the detailed answer here!  

---

