Answers.Body: <p>The same <span class="math-container">$\gamma = 0.9$</span> that you use in the definition <span class="math-container">$F \doteq \gamma \Phi(s') - \Phi(s)$</span> should also be used as the discount factor in computing returns for multi-step trajectories. So, rather than simply adding up all the rewards for your different time-steps for the different trajectories, you should discount them by <span class="math-container">$\gamma$</span> for every time step that expires.</p>
  
  <p>Therefore, the returns of the blue route are:</p>
  
  <p><span class="math-container">$$0 + (0.9 \times -1) + (0.9^2 \times 0) + (0.9^3 \times 1) = -0.9 + 0.729 = -0.171,$$</span></p>
  
  <p>and the returns of the red route are:</p>
  
  <p><span class="math-container">$$(0 + 0.9 \times 100 - 0) + 0.9 \times (1 + 0.9 \times 0 - 100) = 90 - 89.1 = 0.9.$$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-20T17:36:13.153Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/118703/discussion-on-answer-by-dennis-soemers-why-does-potential-based-reward-shaping-s).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-26T14:30:38.727Z
Answers.LastActivityDate: 2019-01-26T14:30:38.727Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>It is known that <em>every potential function</em> won't alter the optimal policy [<a href="https://people.eecs.berkeley.edu/%7Epabbeel/cs287-fa09/readings/NgHaradaRussell-shaping-ICML1999.pdf" rel="nofollow noreferrer">1</a>]. I lack of understanding why is that.</p>
  <p>The definition:</p>
  <p><span class="math-container">$$R' = R + F,$$</span> with <span class="math-container">$$F = \gamma\Phi(s') - \Phi(s),$$</span></p>
  <p>where, let's suppose, <span class="math-container">$\gamma = 0.9$</span>.</p>
  <p>If I have the following setup:</p>
  <ul>
  <li>on the left is my <span class="math-container">$R$</span>.</li>
  <li>on the right my potential function <span class="math-container">$\Phi(s)$</span></li>
  <li>the top left is the start state, the top right is the goal state</li>
  </ul>
  <p><a href="https://i.sstatic.net/WFhWc.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/WFhWc.png" alt="enter image description here" /></a></p>
  <p>The reward for the red route is: <span class="math-container">$(0 + (0.9 * 100 - 0)) + (1 + (0.9 * 0 - 100)) = -9$</span>.</p>
  <p>And the reward for the blue route is: <span class="math-container">$(-1 + 0) + (1 + 0) = 0$</span>.</p>
  <p>So, for me, it seems like the blue route is better than the optimal red route and thus the optimal policy changed. Do I have erroneous thoughts here?</p>
  
Tags: <reinforcement-learning><reward-functions><reward-shaping><potential-reward-shaping>
Title: Why does potential-based reward shaping seem to alter the optimal policy in this case?
UserDisplayName: ScientiaEtVeritas
ThreadText: # Why does potential-based reward shaping seem to alter the optimal policy in this case?
  
  (Asked by: username_0 on 2019-01-26)
  
  It is known that *every potential function* won't alter the optimal policy [[1](https://people.eecs.berkeley.edu/%7Epabbeel/cs287-fa09/readings/NgHaradaRussell-shaping-ICML1999.pdf)]. I lack of understanding why is that.
  
  The definition:
  
  $$R' = R + F,$$ with $$F = \gamma\Phi(s') - \Phi(s),$$
  
  where, let's suppose, $\gamma = 0.9$.
  
  If I have the following setup:
  
  * on the left is my $R$.
  * on the right my potential function $\Phi(s)$
  * the top left is the start state, the top right is the goal state
  
  [![enter image description here](https://i.sstatic.net/WFhWc.png)](https://i.sstatic.net/WFhWc.png)
  
  The reward for the red route is: $(0 + (0.9 \* 100 - 0)) + (1 + (0.9 \* 0 - 100)) = -9$.
  
  And the reward for the blue route is: $(-1 + 0) + (1 + 0) = 0$.
  
  So, for me, it seems like the blue route is better than the optimal red route and thus the optimal policy changed. Do I have erroneous thoughts here?
  
  **Tags:** reinforcement-learning, reward-functions, reward-shaping, potential-reward-shaping
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-01-26. Score: 2 | Accepted answer
  
  The same $\gamma = 0.9$ that you use in the definition $F \doteq \gamma \Phi(s') - \Phi(s)$ should also be used as the discount factor in computing returns for multi-step trajectories. So, rather than simply adding up all the rewards for your different time-steps for the different trajectories, you should discount them by $\gamma$ for every time step that expires.
  
  Therefore, the returns of the blue route are:
  
  $$0 + (0.9 \times -1) + (0.9^2 \times 0) + (0.9^3 \times 1) = -0.9 + 0.729 = -0.171,$$
  
  and the returns of the red route are:
  
  $$(0 + 0.9 \times 100 - 0) + 0.9 \times (1 + 0.9 \times 0 - 100) = 90 - 89.1 = 0.9.$$
  
  > **Comments:**
  > **username_2**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/118703/discussion-on-answer-by-dennis-soemers-why-does-potential-based-reward-shaping-s).  
  
  ---
  
  