<p>The same <span class="math-container">$\gamma = 0.9$</span> that you use in the definition <span class="math-container">$F \doteq \gamma \Phi(s') - \Phi(s)$</span> should also be used as the discount factor in computing returns for multi-step trajectories. So, rather than simply adding up all the rewards for your different time-steps for the different trajectories, you should discount them by <span class="math-container">$\gamma$</span> for every time step that expires.</p>

<p>Therefore, the returns of the blue route are:</p>

<p><span class="math-container">$$0 + (0.9 \times -1) + (0.9^2 \times 0) + (0.9^3 \times 1) = -0.9 + 0.729 = -0.171,$$</span></p>

<p>and the returns of the red route are:</p>

<p><span class="math-container">$$(0 + 0.9 \times 100 - 0) + 0.9 \times (1 + 0.9 \times 0 - 100) = 90 - 89.1 = 0.9.$$</span></p>

CC BY-SA 4.0
2021-01-20T17:36:13.153Z
Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/118703/discussion-on-answer-by-dennis-soemers-why-does-potential-based-reward-shaping-s).
nbro
CC BY-SA 4.0
2019-01-26T14:30:38.727Z
2019-01-26T14:30:38.727Z
Dennis Soemers
<p>It is known that <em>every potential function</em> won't alter the optimal policy [<a href="https://people.eecs.berkeley.edu/%7Epabbeel/cs287-fa09/readings/NgHaradaRussell-shaping-ICML1999.pdf" rel="nofollow noreferrer">1</a>]. I lack of understanding why is that.</p>
<p>The definition:</p>
<p><span class="math-container">$$R' = R + F,$$</span> with <span class="math-container">$$F = \gamma\Phi(s') - \Phi(s),$$</span></p>
<p>where, let's suppose, <span class="math-container">$\gamma = 0.9$</span>.</p>
<p>If I have the following setup:</p>
<ul>
<li>on the left is my <span class="math-container">$R$</span>.</li>
<li>on the right my potential function <span class="math-container">$\Phi(s)$</span></li>
<li>the top left is the start state, the top right is the goal state</li>
</ul>
<p><a href="https://i.sstatic.net/WFhWc.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/WFhWc.png" alt="enter image description here" /></a></p>
<p>The reward for the red route is: <span class="math-container">$(0 + (0.9 * 100 - 0)) + (1 + (0.9 * 0 - 100)) = -9$</span>.</p>
<p>And the reward for the blue route is: <span class="math-container">$(-1 + 0) + (1 + 0) = 0$</span>.</p>
<p>So, for me, it seems like the blue route is better than the optimal red route and thus the optimal policy changed. Do I have erroneous thoughts here?</p>

<reinforcement-learning><reward-functions><reward-shaping><potential-reward-shaping>
Why does potential-based reward shaping seem to alter the optimal policy in this case?
ScientiaEtVeritas
# Why does potential-based reward shaping seem to alter the optimal policy in this case?

(Asked by: username_0 on 2019-01-26)

It is known that *every potential function* won't alter the optimal policy [[1](https://people.eecs.berkeley.edu/%7Epabbeel/cs287-fa09/readings/NgHaradaRussell-shaping-ICML1999.pdf)]. I lack of understanding why is that.

The definition:

$$R' = R + F,$$ with $$F = \gamma\Phi(s') - \Phi(s),$$

where, let's suppose, $\gamma = 0.9$.

If I have the following setup:

* on the left is my $R$.
* on the right my potential function $\Phi(s)$
* the top left is the start state, the top right is the goal state

[![enter image description here](https://i.sstatic.net/WFhWc.png)](https://i.sstatic.net/WFhWc.png)

The reward for the red route is: $(0 + (0.9 \* 100 - 0)) + (1 + (0.9 \* 0 - 100)) = -9$.

And the reward for the blue route is: $(-1 + 0) + (1 + 0) = 0$.

So, for me, it seems like the blue route is better than the optimal red route and thus the optimal policy changed. Do I have erroneous thoughts here?

**Tags:** reinforcement-learning, reward-functions, reward-shaping, potential-reward-shaping


---

## 1 answers

### Answer by username_1 on 2019-01-26. Score: 2 | Accepted answer

The same $\gamma = 0.9$ that you use in the definition $F \doteq \gamma \Phi(s') - \Phi(s)$ should also be used as the discount factor in computing returns for multi-step trajectories. So, rather than simply adding up all the rewards for your different time-steps for the different trajectories, you should discount them by $\gamma$ for every time step that expires.

Therefore, the returns of the blue route are:

$$0 + (0.9 \times -1) + (0.9^2 \times 0) + (0.9^3 \times 1) = -0.9 + 0.729 = -0.171,$$

and the returns of the red route are:

$$(0 + 0.9 \times 100 - 0) + 0.9 \times (1 + 0.9 \times 0 - 100) = 90 - 89.1 = 0.9.$$

> **Comments:**
> **username_2**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/118703/discussion-on-answer-by-dennis-soemers-why-does-potential-based-reward-shaping-s).  

---

