Answers.Body: <p>Reward normalisation is constrained by the need to not change the problem definition. The optimal policy should be the same with or without reward normalisation. The intended improvement is to make it easier to converge on the right policy, and possibly - as in DQN Atari paper - to allow the use of the same hyperparameters elsewhere.</p>
  <p>A change to reward ratios could change the problem definition. Consider an environment where the agent could choose between receiving a reward of +1 three times or a reward of +2 once, due to the rewards being associated with exclusive pathways. Offset these rewards towards zero then the optimal choice will change.</p>
  <p>A change of sign to rewards, when they become centered around zero, can also change what is optimal. This can change things so much that an agent will prefer to end an episode instead of completing a task as originally described in the problem definition.</p>
  <p>These problems don't always occur, but due to the risk, general purpose reward normalisation will not offset, just scale.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-08T11:07:17.043Z
Answers.Text: Regarding your mentioned example, that means that subtracting the mean, depending on the other rewards, could result in the agent choosing the '+2 once' reward over the '+1 3 times' reward, leading to an unintended behavior?
Answers.UserDisplayName: kklaw
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-08T11:45:27.103Z
Answers.Text: @kklaw Exactly, yes
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-08T10:52:37.230Z
Answers.LastActivityDate: 2023-07-08T10:52:37.230Z
Answers.UserDisplayName: Neil Slater
Body: <p>I want to introduce Reward Normalization for my current RL project. For this, I looked at how other libraries implemented this functionality. I found the following implementation:</p>
  <p><a href="https://github.com/openai/gym/blob/master/gym/wrappers/normalize.py" rel="nofollow noreferrer">https://github.com/openai/gym/blob/master/gym/wrappers/normalize.py</a></p>
  <p>Looking at the reward normalization itself :</p>
  <pre><code>def normalize(self, rews):
      &quot;&quot;&quot;Normalizes the rewards with the running mean rewards and their variance.&quot;&quot;&quot;
      self.return_rms.update(self.returns)
      return rews / np.sqrt(self.return_rms.var + self.epsilon)
  </code></pre>
  <p>I do not understand why we don't substract the mean from the rewards? Especially, since in the Observation Normalization, this substraction is present:</p>
  <pre><code>def normalize(self, obs):
      &quot;&quot;&quot;Normalises the observation using the running mean and variance of the observations.&quot;&quot;&quot;
      self.obs_rms.update(obs)
      return (obs - self.obs_rms.mean) / np.sqrt(self.obs_rms.var + self.epsilon)
  </code></pre>
  <p>I assume there must be a good reason for this, but I could not figure it out yet.</p>
  
Tags: <reinforcement-learning><reward-design>
Title: Why doesn't Reward Normalization subtract the mean?
UserDisplayName: kklaw
ThreadText: # Why doesn't Reward Normalization subtract the mean?
  
  (Asked by: username_0 on 2023-07-08)
  
  I want to introduce Reward Normalization for my current RL project. For this, I looked at how other libraries implemented this functionality. I found the following implementation:
  https://github.com/openai/gym/blob/master/gym/wrappers/normalize.py
  Looking at the reward normalization itself :
  def normalize(self, rews):
      """Normalizes the rewards with the running mean rewards and their variance."""
      self.return_rms.update(self.returns)
      return rews / np.sqrt(self.return_rms.var + self.epsilon)
  
  I do not understand why we don't substract the mean from the rewards? Especially, since in the Observation Normalization, this substraction is present:
  def normalize(self, obs):
      """Normalises the observation using the running mean and variance of the observations."""
      self.obs_rms.update(obs)
      return (obs - self.obs_rms.mean) / np.sqrt(self.obs_rms.var + self.epsilon)
  
  I assume there must be a good reason for this, but I could not figure it out yet.
  
  **Tags:** reinforcement-learning, reward-design
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-08. Score: 4 | Accepted answer
  
  Reward normalisation is constrained by the need to not change the problem definition. The optimal policy should be the same with or without reward normalisation. The intended improvement is to make it easier to converge on the right policy, and possibly - as in DQN Atari paper - to allow the use of the same hyperparameters elsewhere.
  
  A change to reward ratios could change the problem definition. Consider an environment where the agent could choose between receiving a reward of +1 three times or a reward of +2 once, due to the rewards being associated with exclusive pathways. Offset these rewards towards zero then the optimal choice will change.
  
  A change of sign to rewards, when they become centered around zero, can also change what is optimal. This can change things so much that an agent will prefer to end an episode instead of completing a task as originally described in the problem definition.
  
  These problems don't always occur, but due to the risk, general purpose reward normalisation will not offset, just scale.
  
  > **Comments:**
  > **username_0**: Regarding your mentioned example, that means that subtracting the mean, depending on the other rewards, could result in the agent choosing the '+2 once' reward over the '+1 3 times' reward, leading to an unintended behavior?  
  > **username_1**: @kklaw Exactly, yes  
  
  ---
  
  