Answers.Body: <p>This is a common trick done in practice: this helps to stabilize training, and prevent large values that can blow up in NaNs.</p>
  <p>The reason is that the std of the Gaussian is learned in <em>log-space</em> (because it's easier to learn with neural-nets) that is unbounded (<span class="math-container">$-\infty, +\infty$</span>), and so you want to bound to something like <span class="math-container">$[-20, 2]$</span> because such log-std will be exponentiated later: you have <span class="math-container">$\exp(-20)\approx 0$</span>, and <span class="math-container">$\exp(2) = 7.389$</span> that is reasonably small; a similar trick is also employed for log-probabilities.</p>
  <p>Practical tricks like this one are often not documented in the paper (unfortunately), but can be found in code implementations.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-04T01:10:37.730Z
Answers.Text: Thank you, I tried to look at other methods papers to see if it was documented and could not find any. I really appreciate your answer. One final question if you have a second, so the bound are truly arbitrary and I could make them anything as long as it it reasonably small?
Answers.UserDisplayName: chadmc
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-04T07:04:42.940Z
Answers.Text: Yes, the bounds are arbitrary but just consider that you're are in log-space and that the $\exp$ can easily underflow or overflow going to infinite. If you want to change the bounds, think about the range of values you want and take the log of that, finally clamp: say you want the std to be in $[1,3]$, you take the log to determine the bounds i.e. $[\log(1), \log(3)] = [0, 1.0986]$ for example.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-08T21:48:43.790Z
Answers.Text: I finally found a reference to your answer, the stable baselines, https://github.com/DLR-RM/stable-baselines3/blob/master/stable_baselines3/common/distributions.py, say that they clamp to avoid NaN values in the comments of the code. You can find it in the TanhBisector function.
Answers.UserDisplayName: chadmc
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-03T20:16:41.690Z
Answers.LastActivityDate: 2023-05-03T20:16:41.690Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>In the Soft Actor Critic Paper (found here <a href="https://arxiv.org/pdf/1801.01290.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1801.01290.pdf</a>), they use a neural network to approximate a diagonal gaussian distribution. In the sample function you can see that it has a function called reparameterize. As you can see in the reparameterize function, we use a tanh function to squash the action bound between -1 and 1. So why is it that we clamp the standard deviation between -20 and 2. I have read both Soft Actor Critic papers and can't find why we do this. Is there something about the normal distribution that would make that range of clamping of the std more desirable? Does bounding the range of the std help with convergence. Is there a paper you my recommend that would help with this answer? This is my first post here so if you need more information please let me know.</p>
  <p><a href="https://i.sstatic.net/8ysdq.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/8ysdq.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/VJqQV.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/VJqQV.png" alt="enter image description here" /></a></p>
  <p><a href="https://github.com/toshikwa/slac.pytorch" rel="nofollow noreferrer">Link to code Repo</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-03T23:43:36.703Z
Comments.Text: Can you please provide the code instead of a screenshot? Although note that programming issues are off-topic here.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><soft-actor-critic><reparameterization-trick>
Title: Why clamp std for reparameterization trick between -20 and 2?
UserDisplayName: chadmc
ThreadText: # Why clamp std for reparameterization trick between -20 and 2?
  
  (Asked by: username_0 on 2023-05-03)
  
  In the Soft Actor Critic Paper (found here <https://arxiv.org/pdf/1801.01290.pdf>), they use a neural network to approximate a diagonal gaussian distribution. In the sample function you can see that it has a function called reparameterize. As you can see in the reparameterize function, we use a tanh function to squash the action bound between -1 and 1. So why is it that we clamp the standard deviation between -20 and 2. I have read both Soft Actor Critic papers and can't find why we do this. Is there something about the normal distribution that would make that range of clamping of the std more desirable? Does bounding the range of the std help with convergence. Is there a paper you my recommend that would help with this answer? This is my first post here so if you need more information please let me know.
  
  [![enter image description here](https://i.sstatic.net/8ysdq.png)](https://i.sstatic.net/8ysdq.png)
  
  [![enter image description here](https://i.sstatic.net/VJqQV.png)](https://i.sstatic.net/VJqQV.png)
  
  [Link to code Repo](https://github.com/toshikwa/slac.pytorch)
  
  **Tags:** reinforcement-learning, soft-actor-critic, reparameterization-trick
  
  > **Comments:**
  > **username_1**: Can you please provide the code instead of a screenshot? Although note that programming issues are off-topic here.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-05-03. Score: 1 | Accepted answer
  
  This is a common trick done in practice: this helps to stabilize training, and prevent large values that can blow up in NaNs.
  
  The reason is that the std of the Gaussian is learned in *log-space* (because it's easier to learn with neural-nets) that is unbounded ($-\infty, +\infty$), and so you want to bound to something like $[-20, 2]$ because such log-std will be exponentiated later: you have $\exp(-20)\approx 0$, and $\exp(2) = 7.389$ that is reasonably small; a similar trick is also employed for log-probabilities.
  
  Practical tricks like this one are often not documented in the paper (unfortunately), but can be found in code implementations.
  
  > **Comments:**
  > **username_0**: Thank you, I tried to look at other methods papers to see if it was documented and could not find any. I really appreciate your answer. One final question if you have a second, so the bound are truly arbitrary and I could make them anything as long as it it reasonably small?  
  > **username_2**: Yes, the bounds are arbitrary but just consider that you're are in log-space and that the $\exp$ can easily underflow or overflow going to infinite. If you want to change the bounds, think about the range of values you want and take the log of that, finally clamp: say you want the std to be in $[1,3]$, you take the log to determine the bounds i.e. $[\log(1), \log(3)] = [0, 1.0986]$ for example.  
  > **username_0**: I finally found a reference to your answer, the stable baselines, https://github.com/DLR-RM/stable-baselines3/blob/master/stable_baselines3/common/distributions.py, say that they clamp to avoid NaN values in the comments of the code. You can find it in the TanhBisector function.  
  
  ---
  
  