Answers.Body: <p>As you have mentioned using log is nicer because it makes multiplications to additions etc etc(it helps in numerical stability issues). But I think over here, the reason they are doing it like that is because of enforcing a simple constraint in a much more simpler way. In the <code>__init__</code> we are noticing that the <code>log_std</code> is being formulated instead of the <code>std</code> itself. Would it be wrong if you formulated the <code>std</code> itself? No. But it would be a bit messy to have some constrained imposed on it. For example, if we modeled <code>std</code> it could have been possible for the <code>std</code> to become <code>0</code> or <code>negative</code>. Well that's not a correct values of a std. But over here that is being enforced automatically, this kind of little things makes the learning for the model easier. Even if the <code>log_std</code> becomes negative or zero it does not matter because the <code>exp</code> will take care of it by exponentiating it to a positive number.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-11T09:20:51.703Z
Answers.Text: thank you for the answer and also mentioning that using std is also not wrong!
Answers.UserDisplayName: R.Sar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-11T08:09:22.153Z
Answers.LastActivityDate: 2021-04-11T08:09:22.153Z
Answers.UserDisplayName: user2736738
Body: <p>I am currently writing my bachelor thesis, which is an implementation of proximal policy optimization. Sometimes, I hit a wall because of the gaps in my mathematical knowledge. However, implementing the algorithm helped me to understand the math behind the algorithm.</p>
  <p>Unfortunately, I still have a question.</p>
  <p>When the action space is continuous, I am using the normal distribution (same as in the PPO implementation by Spinning up). In the mentioned implementation, the logarithm of the standard deviation is used initially to give the same probability to all of the possible action, then they use the standard deviation when choosing an action. Why do we use the logarithm? Why not directly use simply the standard deviation?</p>
  <p>I know that the logarithm is easier when it comes to the computations, but I can not see the benefits of the logarithm in the Spinning up implementation.</p>
  <pre><code>class MLPGaussianActor(Actor):
  
      def __init__(self, obs_dim, act_dim, hidden_sizes, activation):
          super().__init__()
          log_std = -0.5 * np.ones(act_dim, dtype=np.float32)
          self.log_std = torch.nn.Parameter(torch.as_tensor(log_std))
          self.mu_net = mlp([obs_dim] + list(hidden_sizes) + [act_dim], activation)
  
      def _distribution(self, obs):
          mu = self.mu_net(obs)
          std = torch.exp(self.log_std)
          return Normal(mu, std)
  
      def _log_prob_from_distribution(self, pi, act):
          return pi.log_prob(act).sum(axis=-1)    # Last axis sum needed for Torch Normal distribution
  </code></pre>
  
Tags: <reinforcement-learning><implementation><proximal-policy-optimization><normal-distribution>
Title: Why is the logarithm of the standard deviation used in this implementation of proximal policy optimization?
UserDisplayName: R.Sar
ThreadText: # Why is the logarithm of the standard deviation used in this implementation of proximal policy optimization?
  
  (Asked by: username_0 on 2021-04-11)
  
  I am currently writing my bachelor thesis, which is an implementation of proximal policy optimization. Sometimes, I hit a wall because of the gaps in my mathematical knowledge. However, implementing the algorithm helped me to understand the math behind the algorithm.
  Unfortunately, I still have a question.
  When the action space is continuous, I am using the normal distribution (same as in the PPO implementation by Spinning up). In the mentioned implementation, the logarithm of the standard deviation is used initially to give the same probability to all of the possible action, then they use the standard deviation when choosing an action. Why do we use the logarithm? Why not directly use simply the standard deviation?
  I know that the logarithm is easier when it comes to the computations, but I can not see the benefits of the logarithm in the Spinning up implementation.
  class MLPGaussianActor(Actor):
  
      def __init__(self, obs_dim, act_dim, hidden_sizes, activation):
          super().__init__()
          log_std = -0.5 * np.ones(act_dim, dtype=np.float32)
          self.log_std = torch.nn.Parameter(torch.as_tensor(log_std))
          self.mu_net = mlp([obs_dim] + list(hidden_sizes) + [act_dim], activation)
  
      def _distribution(self, obs):
          mu = self.mu_net(obs)
          std = torch.exp(self.log_std)
          return Normal(mu, std)
  
      def _log_prob_from_distribution(self, pi, act):
          return pi.log_prob(act).sum(axis=-1)    # Last axis sum needed for Torch Normal distribution
  
  **Tags:** reinforcement-learning, implementation, proximal-policy-optimization, normal-distribution
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-11. Score: 0 | Accepted answer
  
  As you have mentioned using log is nicer because it makes multiplications to additions etc etc(it helps in numerical stability issues). But I think over here, the reason they are doing it like that is because of enforcing a simple constraint in a much more simpler way. In the `__init__` we are noticing that the `log_std` is being formulated instead of the `std` itself. Would it be wrong if you formulated the `std` itself? No. But it would be a bit messy to have some constrained imposed on it. For example, if we modeled `std` it could have been possible for the `std` to become `0` or `negative`. Well that's not a correct values of a std. But over here that is being enforced automatically, this kind of little things makes the learning for the model easier. Even if the `log_std` becomes negative or zero it does not matter because the `exp` will take care of it by exponentiating it to a positive number.
  
  > **Comments:**
  > **username_0**: thank you for the answer and also mentioning that using std is also not wrong!  
  
  ---
  
  