Answers.Body: <p>It's just a hack to normalize the rewards to make optimization less problematic in cases where the reward might have very large/small values. It's fine because scaling the rewards by a constant factor won't change what the optimal policy is. You can think of the mean reward as the baseline. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-18T08:17:50.070Z
Answers.Text: The mean reward is the baseline, i get it, but why divide by the standard deviation?
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-24T14:37:08.050Z
Answers.Text: Also, that constant does change between episodes. Could you please elaborate on how that actually works?
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-18T02:12:46.213Z
Answers.LastActivityDate: 2018-12-18T02:12:46.213Z
Answers.UserDisplayName: shimao
Body: <p><a href="https://github.com/pytorch/examples/blob/master/reinforcement_learning/reinforce.py" rel="nofollow noreferrer">Pytorch's example for the REINFORCE algorithm</a> for reinforcement learning has the following code:</p>
  
  <pre><code>import argparse
  import gym
  import numpy as np
  from itertools import count
  
  import torch
  import torch.nn as nn
  import torch.nn.functional as F
  import torch.optim as optim
  from torch.distributions import Categorical
  
  
  parser = argparse.ArgumentParser(description='PyTorch REINFORCE example')
  parser.add_argument('--gamma', type=float, default=0.99, metavar='G',
                      help='discount factor (default: 0.99)')
  parser.add_argument('--seed', type=int, default=543, metavar='N',
                      help='random seed (default: 543)')
  parser.add_argument('--render', action='store_true',
                      help='render the environment')
  parser.add_argument('--log-interval', type=int, default=10, metavar='N',
                      help='interval between training status logs (default: 10)')
  args = parser.parse_args()
  
  
  env = gym.make('CartPole-v0')
  env.seed(args.seed)
  torch.manual_seed(args.seed)
  
  
  class Policy(nn.Module):
      def __init__(self):
          super(Policy, self).__init__()
          self.affine1 = nn.Linear(4, 128)
          self.affine2 = nn.Linear(128, 2)
  
          self.saved_log_probs = []
          self.rewards = []
  
      def forward(self, x):
          x = F.relu(self.affine1(x))
          action_scores = self.affine2(x)
          return F.softmax(action_scores, dim=1)
  
  
  policy = Policy()
  optimizer = optim.Adam(policy.parameters(), lr=1e-2)
  eps = np.finfo(np.float32).eps.item()
  
  
  def select_action(state):
      state = torch.from_numpy(state).float().unsqueeze(0)
      probs = policy(state)
      m = Categorical(probs)
      action = m.sample()
      policy.saved_log_probs.append(m.log_prob(action))
      return action.item()
  
  
  def finish_episode():
      R = 0
      policy_loss = []
      rewards = []
      for r in policy.rewards[::-1]:
          R = r + args.gamma * R
          rewards.insert(0, R)
      rewards = torch.tensor(rewards)
      rewards = (rewards - rewards.mean()) / (rewards.std() + eps)
      for log_prob, reward in zip(policy.saved_log_probs, rewards):
          policy_loss.append(-log_prob * reward)
      optimizer.zero_grad()
      policy_loss = torch.cat(policy_loss).sum()
      policy_loss.backward()
      optimizer.step()
      del policy.rewards[:]
      del policy.saved_log_probs[:]
  
  
  def main():
      running_reward = 10
      for i_episode in count(1):
          state = env.reset()
          for t in range(10000):  # Don't infinite loop while learning
              action = select_action(state)
              state, reward, done, _ = env.step(action)
              if args.render:
                  env.render()
              policy.rewards.append(reward)
              if done:
                  break
  
          running_reward = running_reward * 0.99 + t * 0.01
          finish_episode()
          if i_episode % args.log_interval == 0:
              print('Episode {}\tLast length: {:5d}\tAverage length: {:.2f}'.format(
                  i_episode, t, running_reward))
          if running_reward &gt; env.spec.reward_threshold:
              print("Solved! Running reward is now {} and "
                    "the last episode runs to {} time steps!".format(running_reward, t))
              break
  
  
  if __name__ == '__main__':
  main()
  </code></pre>
  
  <p>I am interested in the function <code>finish_episode()</code>:</p>
  
  <p>the line</p>
  
  <pre><code> rewards = (rewards - rewards.mean()) / (rewards.std() + eps)
  </code></pre>
  
  <p>makes no sense to me.</p>
  
  <p>I thought this might be <a href="https://www.quora.com/Why-can-reinforcement-of-the-baseline-reduce-variance" rel="nofollow noreferrer">baseline reduction</a>, but I can't see why divide by the standard deviation.</p>
  
  <p>If it isn't baseline reduction, then why normalize the rewards, and where should the baseline reduction go?</p>
  
  <p>Please explain that line</p>
  
Tags: <machine-learning><reinforcement-learning><reward-normalization>
Title: Why is the reward signal normalized in openAI's REINFORCE?
UserDisplayName: Gulzar
ThreadText: # Why is the reward signal normalized in openAI's REINFORCE?
  
  (Asked by: username_0 on 2018-12-15)
  
  [Pytorch's example for the REINFORCE algorithm](https://github.com/pytorch/examples/blob/master/reinforcement_learning/reinforce.py) for reinforcement learning has the following code:
  
  ```
  import argparse
  import gym
  import numpy as np
  from itertools import count
  
  import torch
  import torch.nn as nn
  import torch.nn.functional as F
  import torch.optim as optim
  from torch.distributions import Categorical
  
  parser = argparse.ArgumentParser(description='PyTorch REINFORCE example')
  parser.add_argument('--gamma', type=float, default=0.99, metavar='G',
                      help='discount factor (default: 0.99)')
  parser.add_argument('--seed', type=int, default=543, metavar='N',
                      help='random seed (default: 543)')
  parser.add_argument('--render', action='store_true',
                      help='render the environment')
  parser.add_argument('--log-interval', type=int, default=10, metavar='N',
                      help='interval between training status logs (default: 10)')
  args = parser.parse_args()
  
  env = gym.make('CartPole-v0')
  env.seed(args.seed)
  torch.manual_seed(args.seed)
  
  class Policy(nn.Module):
      def __init__(self):
          super(Policy, self).__init__()
          self.affine1 = nn.Linear(4, 128)
          self.affine2 = nn.Linear(128, 2)
  
          self.saved_log_probs = []
          self.rewards = []
  
      def forward(self, x):
          x = F.relu(self.affine1(x))
          action_scores = self.affine2(x)
          return F.softmax(action_scores, dim=1)
  
  policy = Policy()
  optimizer = optim.Adam(policy.parameters(), lr=1e-2)
  eps = np.finfo(np.float32).eps.item()
  
  def select_action(state):
      state = torch.from_numpy(state).float().unsqueeze(0)
      probs = policy(state)
      m = Categorical(probs)
      action = m.sample()
      policy.saved_log_probs.append(m.log_prob(action))
      return action.item()
  
  def finish_episode():
      R = 0
      policy_loss = []
      rewards = []
      for r in policy.rewards[::-1]:
          R = r + args.gamma * R
          rewards.insert(0, R)
      rewards = torch.tensor(rewards)
      rewards = (rewards - rewards.mean()) / (rewards.std() + eps)
      for log_prob, reward in zip(policy.saved_log_probs, rewards):
          policy_loss.append(-log_prob * reward)
      optimizer.zero_grad()
      policy_loss = torch.cat(policy_loss).sum()
      policy_loss.backward()
      optimizer.step()
      del policy.rewards[:]
      del policy.saved_log_probs[:]
  
  def main():
      running_reward = 10
      for i_episode in count(1):
          state = env.reset()
          for t in range(10000):  # Don't infinite loop while learning
              action = select_action(state)
              state, reward, done, _ = env.step(action)
              if args.render:
                  env.render()
              policy.rewards.append(reward)
              if done:
                  break
  
          running_reward = running_reward * 0.99 + t * 0.01
          finish_episode()
          if i_episode % args.log_interval == 0:
              print('Episode {}\tLast length: {:5d}\tAverage length: {:.2f}'.format(
                  i_episode, t, running_reward))
          if running_reward > env.spec.reward_threshold:
              print("Solved! Running reward is now {} and "
                    "the last episode runs to {} time steps!".format(running_reward, t))
              break
  
  if __name__ == '__main__':
  main()
  
  ```
  
  I am interested in the function `finish_episode()`:
  
  the line
  
  ```
   rewards = (rewards - rewards.mean()) / (rewards.std() + eps)
  
  ```
  
  makes no sense to me.
  
  I thought this might be [baseline reduction](https://www.quora.com/Why-can-reinforcement-of-the-baseline-reduce-variance), but I can't see why divide by the standard deviation.
  
  If it isn't baseline reduction, then why normalize the rewards, and where should the baseline reduction go?
  
  Please explain that line
  
  **Tags:** machine-learning, reinforcement-learning, reward-normalization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-12-18. Score: 3
  
  It's just a hack to normalize the rewards to make optimization less problematic in cases where the reward might have very large/small values. It's fine because scaling the rewards by a constant factor won't change what the optimal policy is. You can think of the mean reward as the baseline.
  
  > **Comments:**
  > **username_0**: The mean reward is the baseline, i get it, but why divide by the standard deviation?  
  > **username_0**: Also, that constant does change between episodes. Could you please elaborate on how that actually works?  
  
  ---
  
  