Answers.Body: <p>You have some freedom to re-define reward schemes, whilst still describing the same goals for an agent. How this works depends to some degree on whether you are dealing with an episodic or continuing problem.</p>
  <h2>Episodic problems</h2>
  <p>An episodic problem ends, and once an agent reaches the terminal state, it is guaranteed zero rewards from that point on. The optimal behaviour can therefore depend quite critically on balance between positive and negative rewards.</p>
  <ul>
  <li><p>If an environment contains many unavoidable negative rewards, and these outweigh total positive rewards, then the agent will be motivated to complete an episode sooner.</p>
  </li>
  <li><p>If an environment contains repeatable positive rewards, and these outweigh total negative rewards, then the agent will be motivated to loop through the postive rewards and not end the episode.</p>
  </li>
  </ul>
  <p>Scaling all rewards by the same positive factor makes no difference to the goals of an agent in an episodic problem. Adding a positive or negative offset to all rewards can make a difference though. It is likely to be most notable when such a change moves rewards from positive to negative or vice versa. In the MountainCar example, adding +2 to all rewards would mean the agent would gain +1 for each time step. As it would stop gaining any reward for reaching the goal, even though reaching that goal would score the highest possible +2.5 reward, the fact that this ends the episode means that it now becomes a poor choice. The best action for the car in this modified MountainCar is to stay at the bottom of the valley collecting the +1 reward per time step forever.</p>
  <h2>Continuing problems</h2>
  <p>In a continuing problem, there is no way for the agent to avoid the stream of new reward data. That means any positive scaling of all reward values or positive or negative offset, by the same amount, has no impact on what counts as the optimal policy. The calculated value of any state under the same policy, but with rewards all transformed with the same multiplier and offset, will be different, but the optimal policy in that environment will be the same.</p>
  <p>If you scale or offset rewards differently to each other, then that can change the goals of the agent and what the optimal policy is. The balance does not really depend on whether rewards are positive or negative in a continuing environment.</p>
  <p>There may be some exceptions to this for continuing problems when using a discount factor, and setting it relatively low (compared to the typical state &quot;cycling&quot; length in the problem). That can cause changes in behaviour due to offsets, similar to those seen in episodic problems. If you use an average reward setting this tends to be less relevant. Often in DQN, you will choose a high discount factor e.g. 0.99 or 0.999, and this will tend to behave close to an average reward setting provided rewards are not very sparse.</p>
  <h2>In general</h2>
  <p>In either case, if you change a reward system, and that results in an agent that consistently learns a different policy, that will usually mean one of two things:</p>
  <ul>
  <li><p>The original reward system was incorrect. It described a goal that you did not intend, or had &quot;loopholes&quot; that the agent could exploit to gain more reward in a way that you did not intend.</p>
  </li>
  <li><p>The implementation of the agent was sensitive in some way to absolute values of total reward. That could be due to a hyperparameter choice in something like a neural network for example, or maybe a bug.</p>
  </li>
  </ul>
  <p>Another possibility, that you may see if you only run a few experiments, is that the agent is not learning 100% consistently, and you are accidentally correlating your changes to reward scheme with the noise/randomness in the results. A DQN-based agent will usually have some variability in how well it solves a problem. After training, DQN is usually only approximately optimal, and by chance some approximations are closer than others.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-02T01:38:45.117Z
Answers.Text: u mention that "If an environment contains many unavoidable negative rewards, and these outweigh total positive rewards, then the agent will be motivated to complete an episode sooner." Why and how does this happen in a DQN? Where does the motivation come from?
Answers.UserDisplayName: Fishfish
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-02T07:33:44.580Z
Answers.Text: @Fishfish: I use "will be motivated to" as loose short-hand for "will converge to this option because it is numerically optimal". It happens because the solution - of completing an episode as soon as possible - is optimal, nothing to do specifcally with a DQN, it applies equally to any solver that works in an environment where the average time step reward is *unavoidably* negative.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-01T08:13:14.927Z
Answers.LastActivityDate: 2020-07-01T11:36:24.233Z
Answers.LastEditDate: 2020-07-01T11:36:24.233Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>Our paper 'Exploit Reward Shifting in Value-Based DRL' answered this question.</p>
  <p>In the case mentioned, using a negative shift will lead to explorative behaviors, therefore the DQN agent has better convergence behavior.</p>
  <p>In general, we show that a negative (constant) shift is equivalent to optimistic value initialization, while a positive shift is equivalent to pessimistic (conservative) value initialization --- hence the former helps exploration, and the latter helps exploitation.</p>
  <p>Reference:
  <a href="https://sites.google.com/view/rewardshaping" rel="nofollow noreferrer">1</a> <a href="https://sites.google.com/view/rewardshaping" rel="nofollow noreferrer">[Google Site]</a>
  <a href="https://neurips.cc/virtual/2022/poster/55336" rel="nofollow noreferrer">2</a> <a href="https://neurips.cc/virtual/2022/poster/55336" rel="nofollow noreferrer">NeurIPS'2022 Paper</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-21T12:01:41.847Z
Answers.LastActivityDate: 2023-02-21T12:01:41.847Z
Answers.UserDisplayName: user68435
Body: <p>I am new to reinforcement learning. For my application, I have found out that if my reward function contains some negative and positive values, my model does not give the optimal solution, but the solution is not bad as it still gives positive reward at the end.</p>
  <p>However, if I just shift all readings by subtracting a constant until my reward function is all negative, my model can reach the optimal solution easily.</p>
  <p>Why is this happening?</p>
  <p>I am using DQN for my application.</p>
  <p>I feel that this is also the reason why the gym environment mountaincar-v0 uses <span class="math-container">$-1$</span> for each time step and <span class="math-container">$0.5$</span> at the goal, but correct me if I am wrong.</p>
  
Tags: <reinforcement-learning><dqn><rewards><reward-shaping><reward-functions>
Title: Why does shifting all the rewards have a different impact on the performance of the agent?
UserDisplayName: Fishfish
ThreadText: # Why does shifting all the rewards have a different impact on the performance of the agent?
  
  (Asked by: username_0 on 2020-07-01)
  
  I am new to reinforcement learning. For my application, I have found out that if my reward function contains some negative and positive values, my model does not give the optimal solution, but the solution is not bad as it still gives positive reward at the end.
  
  However, if I just shift all readings by subtracting a constant until my reward function is all negative, my model can reach the optimal solution easily.
  
  Why is this happening?
  
  I am using DQN for my application.
  
  I feel that this is also the reason why the gym environment mountaincar-v0 uses $-1$ for each time step and $0.5$ at the goal, but correct me if I am wrong.
  
  **Tags:** reinforcement-learning, dqn, rewards, reward-shaping, reward-functions
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-07-01. Score: 2
  
  You have some freedom to re-define reward schemes, whilst still describing the same goals for an agent. How this works depends to some degree on whether you are dealing with an episodic or continuing problem.
  
  ## Episodic problems
  
  An episodic problem ends, and once an agent reaches the terminal state, it is guaranteed zero rewards from that point on. The optimal behaviour can therefore depend quite critically on balance between positive and negative rewards.
  
  * If an environment contains many unavoidable negative rewards, and these outweigh total positive rewards, then the agent will be motivated to complete an episode sooner.
  * If an environment contains repeatable positive rewards, and these outweigh total negative rewards, then the agent will be motivated to loop through the postive rewards and not end the episode.
  
  Scaling all rewards by the same positive factor makes no difference to the goals of an agent in an episodic problem. Adding a positive or negative offset to all rewards can make a difference though. It is likely to be most notable when such a change moves rewards from positive to negative or vice versa. In the MountainCar example, adding +2 to all rewards would mean the agent would gain +1 for each time step. As it would stop gaining any reward for reaching the goal, even though reaching that goal would score the highest possible +2.5 reward, the fact that this ends the episode means that it now becomes a poor choice. The best action for the car in this modified MountainCar is to stay at the bottom of the valley collecting the +1 reward per time step forever.
  
  ## Continuing problems
  
  In a continuing problem, there is no way for the agent to avoid the stream of new reward data. That means any positive scaling of all reward values or positive or negative offset, by the same amount, has no impact on what counts as the optimal policy. The calculated value of any state under the same policy, but with rewards all transformed with the same multiplier and offset, will be different, but the optimal policy in that environment will be the same.
  
  If you scale or offset rewards differently to each other, then that can change the goals of the agent and what the optimal policy is. The balance does not really depend on whether rewards are positive or negative in a continuing environment.
  
  There may be some exceptions to this for continuing problems when using a discount factor, and setting it relatively low (compared to the typical state "cycling" length in the problem). That can cause changes in behaviour due to offsets, similar to those seen in episodic problems. If you use an average reward setting this tends to be less relevant. Often in DQN, you will choose a high discount factor e.g. 0.99 or 0.999, and this will tend to behave close to an average reward setting provided rewards are not very sparse.
  
  ## In general
  
  In either case, if you change a reward system, and that results in an agent that consistently learns a different policy, that will usually mean one of two things:
  
  * The original reward system was incorrect. It described a goal that you did not intend, or had "loopholes" that the agent could exploit to gain more reward in a way that you did not intend.
  * The implementation of the agent was sensitive in some way to absolute values of total reward. That could be due to a hyperparameter choice in something like a neural network for example, or maybe a bug.
  
  Another possibility, that you may see if you only run a few experiments, is that the agent is not learning 100% consistently, and you are accidentally correlating your changes to reward scheme with the noise/randomness in the results. A DQN-based agent will usually have some variability in how well it solves a problem. After training, DQN is usually only approximately optimal, and by chance some approximations are closer than others.
  
  > **Comments:**
  > **username_0**: u mention that "If an environment contains many unavoidable negative rewards, and these outweigh total positive rewards, then the agent will be motivated to complete an episode sooner." Why and how does this happen in a DQN? Where does the motivation come from?  
  > **username_1**: @Fishfish: I use "will be motivated to" as loose short-hand for "will converge to this option because it is numerically optimal". It happens because the solution - of completing an episode as soon as possible - is optimal, nothing to do specifcally with a DQN, it applies equally to any solver that works in an environment where the average time step reward is *unavoidably* negative.  
  
  ---
  
  ### Answer by username_2 on 2023-02-21. Score: 1
  
  Our paper 'Exploit Reward Shifting in Value-Based DRL' answered this question.
  
  In the case mentioned, using a negative shift will lead to explorative behaviors, therefore the DQN agent has better convergence behavior.
  
  In general, we show that a negative (constant) shift is equivalent to optimistic value initialization, while a positive shift is equivalent to pessimistic (conservative) value initialization --- hence the former helps exploration, and the latter helps exploitation.
  
  Reference:
  [1](https://sites.google.com/view/rewardshaping) [[Google Site]](https://sites.google.com/view/rewardshaping)
  [2](https://neurips.cc/virtual/2022/poster/55336) [NeurIPS'2022 Paper](https://neurips.cc/virtual/2022/poster/55336)
  
  
  ---
  
  