Answers.Body: <ol>
  <li>Bounding the log-std is a common practice to avoid/reduce instabilities during training and also avoid numerical issues and NaNs since the log-std is unbounded, being in <span class="math-container">$(-\infty,\infty)$</span>, when taking the exponential of such quantity (to actually obtain the std) such value can easily blow up causing a NaN, which is propagated until the learning objective making also debug difficult. Therefore you want to do this in practice, but keep in mind that the interval you choose to limit the log-std also limits the range of the actions that the policy can sample. So don't be too restrictive.</li>
  <li>I'm not sure but I think the solution with the <code>tanh</code> bounding may yields smoother gradients compared to the <code>clamp</code>, which should yield a zero-gradient for that layer if the value is actually clamped. Also, the resulting gradient of the first solution should have a term that accounts for the derivative of <code>tanh</code>, so I think the overall gradient to be different. Actually without experiments and a fair comparison is difficult to tell which solution is better (more stable, faster to learn, etc.)</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-28T23:54:55.850Z
Answers.Text: btw, nobody is forcing us to use the exponential to get the std.. you can definitely use any strictly positive function, such as a $ELU+1$, and you can solve the risk of getting exponentially big gradients
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-28T23:58:32.690Z
Answers.Text: a personal note on the second one, IMHO the tanh might be better, as it has been shown that smooth landscape of the loss function helps the RL agent to learn better policy (to avoid stuff like cliffs in the loss), as in the case of the "what really matters in Deep RL" with the activation function of the FF layers (buu relu, upp tanh in that case)
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-29T04:55:30.687Z
Answers.Text: @AlbertoSinigaglia I understand you're suggesting using a positive function like `ELU+1` to achieve a positive **std**, rather than employing the `exp` of an unbounded **log_std**. Then we apply the aforementioned `tanh and mapping`, or `clamp` to restrict the output. Right? This approach seems viable, but it's not commonly used. Could you provide relevant open-source code for further study?
Answers.UserDisplayName: XiaoBanni
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-29T16:08:39.903Z
Answers.Text: @AlbertoSinigaglia Sure, also a `softplus` can be used. The point is that all of these are unbounded. So, by clipping the log-std you're also implicitly defining a maximum value for the std regardless the function you use to exponentiate, which prevents to get infinite values.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-29T18:10:31.903Z
Answers.Text: yes you are definitely right on that, I was just pointing out that there are ways to avoid the others (also valid) problems you brought up... about it being unbounded, nothing to say against it ahah
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-31T12:25:05.013Z
Answers.Text: @XiaoBanni they recommend it here also https://arxiv.org/pdf/2006.05990.pdf
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-28T18:47:03.960Z
Answers.LastActivityDate: 2023-07-28T18:47:03.960Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I'm in the process of implementing Actor-Critic structures for Reinforcement Learning (RL) and I've noticed that it's a common practice to limit the standard deviation (std). I've seen this in different implementations, specifically in two types of Actor classes, which I'll refer to as ActorOne (from <a href="https://github.com/facebookresearch/deep_bisim4control/blob/5967b6d0ccfc1032837cbe542f7bc5a96dc02cbb/sac_ae.py#L84" rel="nofollow noreferrer">link</a>) and ActorTwo (from <a href="https://github.com/watchernyu/REDQ/blob/45a28d3e0960e222d1864c2cc8dabc1af287839f/redq/algos/core.py#L173" rel="nofollow noreferrer">link</a>).</p>
  <p>In both implementations, I've observed the restriction of the standard deviation. In ActorOne, the output of a Linear layer is passed through a tanh function to constrain it to the range [-1,1], which is then further transformed to map to a specific range (LOG_SIG_MIN and LOG_SIG_MAX) to get the final log_std. In ActorTwo, the output of a Linear layer is directly clamped to a specific range (LOG_SIG_MIN, LOG_SIG_MAX) to obtain the final log_std.</p>
  <p>Here's a snippet from the ActorOne class:</p>
  <pre><code>log_std = linear_layer(h)
  log_std = torch.tanh(log_std)
  log_std = LOG_SIG_MIN + 0.5 * (LOG_SIG_MAX - LOG_SIG_MIN) * (log_std + 1)
  </code></pre>
  <p>And here's a snippet from the ActorTwo class:</p>
  <pre><code>log_std = linear_layer(h)
  log_std = torch.clamp(log_std, LOG_SIG_MIN, LOG_SIG_MAX)
  </code></pre>
  <p>I have two main questions about this:</p>
  <ol>
  <li><p>What is the purpose of limiting the standard deviation in these implementations? Could there be any practical implications or potential issues if the standard deviation is not limited?</p>
  </li>
  <li><p>What are the differences between these two methods? From what I understand, in the first approach, the gradient is always preserved, while in the second method, the data that gets processed by Clamp will have its gradient cut off. What are the strengths and weaknesses of each method, and how should I choose between them?</p>
  </li>
  </ol>
  <p>Any insights or recommendations on this topic would be greatly appreciated.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-28T18:47:20.357Z
Comments.Text: Related: https://ai.stackexchange.com/a/40292/69468
Comments.UserDisplayName: Luca Anzalone
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-29T04:24:17.230Z
Comments.Text: @LucaAnzalone The link contains a clear explanation with an illustrative example, which answered my first question. Thanks.
Comments.UserDisplayName: XiaoBanni
Tags: <reinforcement-learning><deep-rl><actor-critic-methods><proximal-policy-optimization><soft-actor-critic>
Title: Why do we limit the standard deviation in Actor architectures in Reinforcement Learning?
UserDisplayName: XiaoBanni
ThreadText: # Why do we limit the standard deviation in Actor architectures in Reinforcement Learning?
  
  (Asked by: username_0 on 2023-07-27)
  
  I'm in the process of implementing Actor-Critic structures for Reinforcement Learning (RL) and I've noticed that it's a common practice to limit the standard deviation (std). I've seen this in different implementations, specifically in two types of Actor classes, which I'll refer to as ActorOne (from link) and ActorTwo (from link).
  In both implementations, I've observed the restriction of the standard deviation. In ActorOne, the output of a Linear layer is passed through a tanh function to constrain it to the range [-1,1], which is then further transformed to map to a specific range (LOG_SIG_MIN and LOG_SIG_MAX) to get the final log_std. In ActorTwo, the output of a Linear layer is directly clamped to a specific range (LOG_SIG_MIN, LOG_SIG_MAX) to obtain the final log_std.
  Here's a snippet from the ActorOne class:
  log_std = linear_layer(h)
  log_std = torch.tanh(log_std)
  log_std = LOG_SIG_MIN + 0.5 * (LOG_SIG_MAX - LOG_SIG_MIN) * (log_std + 1)
  
  And here's a snippet from the ActorTwo class:
  log_std = linear_layer(h)
  log_std = torch.clamp(log_std, LOG_SIG_MIN, LOG_SIG_MAX)
  
  I have two main questions about this:
  
  What is the purpose of limiting the standard deviation in these implementations? Could there be any practical implications or potential issues if the standard deviation is not limited?
  
  What are the differences between these two methods? From what I understand, in the first approach, the gradient is always preserved, while in the second method, the data that gets processed by Clamp will have its gradient cut off. What are the strengths and weaknesses of each method, and how should I choose between them?
  
  
  Any insights or recommendations on this topic would be greatly appreciated.
  
  **Tags:** reinforcement-learning, deep-rl, actor-critic-methods, proximal-policy-optimization, soft-actor-critic
  > **Comments:**
  > **username_1**: Related: https://ai.stackexchange.com/a/40292/69468  
  > **username_0**: @LucaAnzalone The link contains a clear explanation with an illustrative example, which answered my first question. Thanks.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-28. Score: 1
  
  1. Bounding the log-std is a common practice to avoid/reduce instabilities during training and also avoid numerical issues and NaNs since the log-std is unbounded, being in $(-\infty,\infty)$, when taking the exponential of such quantity (to actually obtain the std) such value can easily blow up causing a NaN, which is propagated until the learning objective making also debug difficult. Therefore you want to do this in practice, but keep in mind that the interval you choose to limit the log-std also limits the range of the actions that the policy can sample. So don't be too restrictive.
  2. I'm not sure but I think the solution with the `tanh` bounding may yields smoother gradients compared to the `clamp`, which should yield a zero-gradient for that layer if the value is actually clamped. Also, the resulting gradient of the first solution should have a term that accounts for the derivative of `tanh`, so I think the overall gradient to be different. Actually without experiments and a fair comparison is difficult to tell which solution is better (more stable, faster to learn, etc.)
  
  > **Comments:**
  > **username_2**: btw, nobody is forcing us to use the exponential to get the std.. you can definitely use any strictly positive function, such as a $ELU+1$, and you can solve the risk of getting exponentially big gradients  
  > **username_2**: a personal note on the second one, IMHO the tanh might be better, as it has been shown that smooth landscape of the loss function helps the RL agent to learn better policy (to avoid stuff like cliffs in the loss), as in the case of the "what really matters in Deep RL" with the activation function of the FF layers (buu relu, upp tanh in that case)  
  > **username_0**: @AlbertoSinigaglia I understand you're suggesting using a positive function like `ELU+1` to achieve a positive **std**, rather than employing the `exp` of an unbounded **log_std**. Then we apply the aforementioned `tanh and mapping`, or `clamp` to restrict the output. Right? This approach seems viable, but it's not commonly used. Could you provide relevant open-source code for further study?  
  > **username_1**: @AlbertoSinigaglia Sure, also a `softplus` can be used. The point is that all of these are unbounded. So, by clipping the log-std you're also implicitly defining a maximum value for the std regardless the function you use to exponentiate, which prevents to get infinite values.  
  > **username_2**: yes you are definitely right on that, I was just pointing out that there are ways to avoid the others (also valid) problems you brought up... about it being unbounded, nothing to say against it ahah  
  > **username_2**: @XiaoBanni they recommend it here also https://arxiv.org/pdf/2006.05990.pdf  
  
  ---
  
  