Answers.Body: <p>Generally we can write for <span class="math-container">$R_c$</span> the total reward with added constant <span class="math-container">$c$</span> of a policy as
  <span class="math-container">$$
  R_c = \sum_{i=0}^K (r_i + c) \gamma^i = \sum_{i=0}^K r_i \gamma^i + \sum_{i=0}^K c \gamma^i
  $$</span>
  So if we have two policies with the same total reward (without added constant)
  <span class="math-container">$$
  \sum_{i=0}^{K_1} r_i^1 \gamma^i = \sum_{i=0}^{K_2} r_i^2 \gamma^i
  $$</span>
  but with different lengths <span class="math-container">$K_1 \neq K_2$</span> the total reward with added constant will be different, because the second term in <span class="math-container">$R_c$</span> ( <span class="math-container">$\sum_{i=0}^K c \gamma^i$</span> ) will be different.</p>
  
  <p>As an example: Consider two optimal policies, both generating the same cumulative reward of 10, but the first policy visits 4 states, before it reaches a terminal state, while the second visits only two states. The rewards can be written as:
  <span class="math-container">$$
  10 + 0 + 0 + 0 = 10
  $$</span>
  and 
  <span class="math-container">$$
  0 + 10 = 10
  $$</span>
  But when we add 100 to every reward:
  <span class="math-container">$$
  110 + 100 + 100 + 100 = 410
  $$</span>
  and 
  <span class="math-container">$$
  100 + 110 = 210
  $$</span>
  Thus, now the first one is better.</p>
  
  <p>In the continious case, the episodes always have length <span class="math-container">$K = \infty$</span>. Therefore, they always have the same length, and adding a constant doesnt change anything, because the second term in <span class="math-container">$R_c$</span> stays the same.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-14T22:13:40.463Z
Answers.Text: Excellent answer!
Answers.UserDisplayName: InvestingScientist
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-14T20:10:15.897Z
Answers.Text: Another important effect that distorts the problem and changes its solution is rewards sign changing. Explanation [here](https://www.coursera.org/learn/practical-rl/lecture/e5sNr/reward-design) at time 00:12
Answers.UserDisplayName: Hermes Morales
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-03T13:41:59.080Z
Answers.LastActivityDate: 2019-12-03T13:41:59.080Z
Answers.UserDisplayName: Luca Thiede
Body: <p>I'm taking a <a href="https://www.coursera.org/learn/fundamentals-of-reinforcement-learning" rel="noreferrer">Coursera course on Reinforcement learning</a>. There was a question there that wasn't addressed in the learning material: Does adding a constant to all rewards change the set of optimal policies in episodic tasks?</p>
  
  <p>The answer is Yes - Adding a constant to the reward signal can make longer episodes more or less advantageous (depending on whether the constant is positive or negative).</p>
  
  <p>Can anyone explain why is this so? And why it doesn't change in the case of continuous (non episodic) tasks? I don't see why adding a constant matters - as an optimal policy would still want to get the maximum reward...</p>
  
  <p>Can anyone give an example of this?</p>
  
Tags: <reinforcement-learning>
Title: Does adding a constant to all rewards change the set of optimal policies in episodic tasks?
UserDisplayName: Maverick Meerkat
ThreadText: # Does adding a constant to all rewards change the set of optimal policies in episodic tasks?
  
  (Asked by: username_0 on 2019-12-03)
  
  I'm taking a [Coursera course on Reinforcement learning](https://www.coursera.org/learn/fundamentals-of-reinforcement-learning). There was a question there that wasn't addressed in the learning material: Does adding a constant to all rewards change the set of optimal policies in episodic tasks?
  
  The answer is Yes - Adding a constant to the reward signal can make longer episodes more or less advantageous (depending on whether the constant is positive or negative).
  
  Can anyone explain why is this so? And why it doesn't change in the case of continuous (non episodic) tasks? I don't see why adding a constant matters - as an optimal policy would still want to get the maximum reward...
  
  Can anyone give an example of this?
  
  **Tags:** reinforcement-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-12-03. Score: 14 | Accepted answer
  
  Generally we can write for $R\_c$ the total reward with added constant $c$ of a policy as
  $$
  R\_c = \sum\_{i=0}^K (r\_i + c) \gamma^i = \sum\_{i=0}^K r\_i \gamma^i + \sum\_{i=0}^K c \gamma^i
  $$
  So if we have two policies with the same total reward (without added constant)
  $$
  \sum\_{i=0}^{K\_1} r\_i^1 \gamma^i = \sum\_{i=0}^{K\_2} r\_i^2 \gamma^i
  $$
  but with different lengths $K\_1 \neq K\_2$ the total reward with added constant will be different, because the second term in $R\_c$ ( $\sum\_{i=0}^K c \gamma^i$ ) will be different.
  
  As an example: Consider two optimal policies, both generating the same cumulative reward of 10, but the first policy visits 4 states, before it reaches a terminal state, while the second visits only two states. The rewards can be written as:
  $$
  10 + 0 + 0 + 0 = 10
  $$
  and
  $$
  0 + 10 = 10
  $$
  But when we add 100 to every reward:
  $$
  110 + 100 + 100 + 100 = 410
  $$
  and
  $$
  100 + 110 = 210
  $$
  Thus, now the first one is better.
  
  In the continious case, the episodes always have length $K = \infty$. Therefore, they always have the same length, and adding a constant doesnt change anything, because the second term in $R\_c$ stays the same.
  
  > **Comments:**
  > **username_2**: Excellent answer!  
  > **username_3**: Another important effect that distorts the problem and changes its solution is rewards sign changing. Explanation [here](https://www.coursera.org/learn/practical-rl/lecture/e5sNr/reward-design) at time 00:12  
  
  ---
  
  