Answers.Body: <p>The SAC algorithm was designed for control and robotics tasks in mind, i.e. environments with continuous states and actions spaces.</p>
  <p>In practice, SAC implements a policy that is called a <strong>Squashed Gaussian</strong> (read <a href="https://spinningup.openai.com/en/latest/algorithms/sac.html" rel="nofollow noreferrer">here</a>): is a Gaussian distribution that is bounded (or squashed) to output actions within the <span class="math-container">$[-1, 1]$</span> interval by a <span class="math-container">$\tanh$</span> operation. Intuitively, you can think of it as <span class="math-container">$a \sim \tanh(\mathcal{N}(\mu, \Sigma))$</span>. This is because the Gaussian is unbounded having support in <span class="math-container">$[-\infty, \infty]$</span>, and so limiting the actions was done by truncating the distribution which is not much clever, and in fact usually results in lower performance: I guess because you have no gradients when actions fall outside  the desired range, instead the tanh provides a <em>continuous yet bounded</em> interval. (An alternative to the squashed gaussian could be a Beta distribution that is in <span class="math-container">$[0, 1]$</span>, instead, as motivated <a href="https://proceedings.mlr.press/v70/chou17a.html" rel="nofollow noreferrer">here</a> - but I think almost nobody uses that.)</p>
  <p>Now, the term <span class="math-container">$\mu(u\mid s)$</span> corresponds to the underlying Gaussian distribution that outputs/samples actions <span class="math-container">$u$</span>, providing they probability to occur. But since the <span class="math-container">$\tanh$</span> is applied you have to <em>correct</em> such probability, since you want to evaluate <span class="math-container">$\log \pi(a\mid a)$</span> instead of <span class="math-container">$\log \mu(u\mid s)$</span>. In few words, the correction factor is equal to the inverse of the <em>Jacobian</em> determinant of the tanh transformation. So you compute the probability of the Gaussian policy <span class="math-container">$\mu(u\mid s)$</span> then multiply by the Jacobian, and finally take the log to obtain the log-prob of the squashed actions <span class="math-container">$a$</span>.</p>
  <p>If you're interested in how this is done in practice check the <a href="https://github.com/haarnoja/sac/blob/master/sac/policies/gaussian_policy.py#L131" rel="nofollow noreferrer">official</a> and these (<a href="https://coax.readthedocs.io/en/latest/_modules/coax/proba_dists/_squashed_normal.html#SquashedNormalDist" rel="nofollow noreferrer">1</a> and <a href="https://github.com/DLR-RM/stable-baselines3/blob/master/stable_baselines3/common/distributions.py#L227-L240" rel="nofollow noreferrer">2</a>) other implementations. As a side note, if you use tools like tensorflow-probability you can implement this quite easily by applying a Tanh bijector to a Normal distribution.</p>
  <hr />
  <p><strong>Update:</strong> log of the Normal pdf.
  The Normal <span class="math-container">$\mathcal{N}(\mu,\sigma^2)$</span> pdf of a sample <span class="math-container">$x$</span> (e.g. actions) is defined as follows:</p>
  <p><span class="math-container">$$\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac12 \big(\frac{x-\mu}{\sigma}\big)}$$</span></p>
  <p>Now, in RL you care about the <em>logarithm</em> of such probability density. So:</p>
  <p><span class="math-container">$$\log\Big(\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac12 \big(\frac{x-\mu}{\sigma}\big)}\Big) \\ = \log\Big(\frac{1}{\sigma\sqrt{2\pi}}\Big) + \log\Big(e^{-\frac12 \big(\frac{x-\mu}{\sigma}\big)}\Big) \\ = -\log\sigma -\log\sqrt{2\pi} - \frac12\big(\frac{x-\mu}{\sigma}\big)^2$$</span></p>
  <p>Which is the same formula SB and pytorch implements.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-09T20:07:33.240Z
Answers.Text: I would say that I think the `tanh` transformation makes sense for an action space that is symmetric, like the MuJoCo tasks which were the benchmark environments in the original paper.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-09T20:11:10.680Z
Answers.Text: Sure, I guess that one (good at math) can extend this idea to a `sigmoid`, or in general to even `affine` transformations (scaling and biasing) to handle whatever action space.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-09T20:13:42.503Z
Answers.Text: Yes that's right, you can use the pdf transformation method for arbitrary transformations (assuming the function is one-to-one). I think the best is some combination of `tanh` or `sigmoid` followed by an affine transformation and you can satisfy most action space bounds.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-09T21:38:54.653Z
Answers.Text: Hey I really appreciate the answer and I'll definitely accept it.
Answers.UserDisplayName: chadmc
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-09T21:55:13.777Z
Answers.Text: I still am unsure about the implementation, which is why I posted the code. I looked at the implementation you posted. You linked to the the squashed gaussian. This function calls the log probability of the normal distribution and uses that as the value of log p. The log prob function on this page has yet another different implementation, https://coax.readthedocs.io/en/latest/_modules/coax/proba_dists/_normal.html. I have been trying to get an answer to that implementation or the code I posted. The information you have been responding to is in the appendix c of both SAC papers.
Answers.UserDisplayName: chadmc
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-09T21:59:07.263Z
Answers.Text: I am really looking for where the implementations come from, where are the equations coming from for the log probability of the normal distribution before it is transformed via change of variable. How do you compute the Gaussian log probability. The implementation is so different for Stable baselines, your code, the code I posted. Again you have been awesome, thank you for your patients.
Answers.UserDisplayName: chadmc
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-10T13:52:07.973Z
Answers.Text: @chadmc I found the official implementation (I've added the link in the answer) and they follow exactly the equation 26 in the appendix C of the paper. For the coax implementation that is different, I'm not completely sure, but it can to improve the numerical stability of the log of the arctanh ($1-\tanh^2$)
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-10T15:55:34.340Z
Answers.Text: Let's not close the post, I want time to really dig deep on the implementations, I'll try to update by tomorrow. @LucaAnzalone thanks again I've enjoyed working with you.
Answers.UserDisplayName: chadmc
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-10T16:56:40.830Z
Answers.Text: @chadmc the ‘log probability’ of a normal distribution is literally just the log of the pdf evaluated at a given point in the support (in this case in your sampled action, but presumably the action was sampled from a squashed normal distribution so you want to evaluate the log pdf of _that_ distribution)
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-10T17:01:51.840Z
Answers.Text: I really appreciate insight @DavidIreland, the implementations in the code bases doesn't quite seem to match the pdf or at least they don't seem to I could be missing something, I'll update my post to show you what I'm talking about after work hopefully tonight, but maybe tomorrow.
Answers.UserDisplayName: chadmc
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-10T21:00:39.727Z
Answers.Text: If the code does `.log_prob(action)` of the distribution object, then it is evaluating the log pdf (this is how it works in PyTorch at least).
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-19T18:21:51.107Z
Answers.Text: @DavidIreland I've been very busy, but I added more context, does that clear up the question?
Answers.UserDisplayName: chadmc
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-20T10:14:42.107Z
Answers.Text: @chadmc I've updated my answer to include a derivation of the log-pdf of the Gaussian, have a look
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-20T14:06:20.550Z
Answers.Text: It’s hard to read your edits because you’ve not used mathematical notation (eg written the word ‘sigma’ instead of $\sigma$) but what you have for the stable baseline example looks precisely like the log-pdf of the normal distribution so I’m not sure why you’re confused.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-20T16:10:22.447Z
Answers.Text: @DavidIreland strange.. I used MathJax and I can see it rendered correctly
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-20T17:33:32.720Z
Answers.Text: Sorry @LucaAnzalone, I was talking to OP. Your update is great and should clear the confusion for OP!
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-21T14:20:46.743Z
Answers.Text: @LucaAnzalone Thank you very much, you really have been great. Wish I could upvote 2x.
Answers.UserDisplayName: chadmc
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-09T19:55:40.957Z
Answers.LastActivityDate: 2023-05-20T10:13:37.440Z
Answers.LastEditDate: 2023-05-20T10:13:37.440Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>This question comes from trying to build a SAC model. The action space is derived from a log normal distribution.</p>
  <p>If in the appendix c of the original paper the equation for the log policy is:</p>
  <p><span class="math-container">$\log \pi(a|s) = \log \mu(u|s) - \sum^D_{i = 1} \log (1 - \tanh^2(u_i))$</span></p>
  <p>where <span class="math-container">$u \in R^D$</span> is a random variable, <span class="math-container">$\mu(u|s)$</span> is the corresponding density wit infinite support (in this case being a Gaussian distribution.)</p>
  <p>So, my question is, according to the code, the log probabilities are calculated using the following:</p>
  <p><span class="math-container">$\log \mu(u|s) = -0.5 \times \log(2 \times \pi) \times n + \sum (-0.5 \times \epsilon^2 - \log \sigma^2) $</span></p>
  <p>where <span class="math-container">$\epsilon \sim \mathcal{N}(0,1)$</span></p>
  <p>n = size of log_std vector</p>
  <p>The implementation on stable baseline is more complicated.</p>
  <p>I want to know where this formula come from because doesn't look normal pdf or log normal pdf.</p>
  <p><span class="math-container">\begin{equation}
  \frac{1}{\sigma \sqrt{2 \pi}} e^{\frac{1}{2}(\frac{x-\mu}{\sigma})^2} \\
  \frac{1}{x \sigma \sqrt{2 \pi}} e^{ -( \frac{\log(x-\mu)^2}{2\sigma^2})}
  \end{equation}</span></p>
  <p>Further elaborations on implementations that i'm trying to understand:</p>
  <p><a href="https://coax.readthedocs.io/en/latest/_modules/coax/proba_dists/_normal.html#log_proba" rel="nofollow noreferrer">Example 1: Look up log_proba here</a></p>
  <p>The log probability is defined as:</p>
  <p><span class="math-container">$\log \pi_\theta = -0.5 \times (n \times \log(2 \pi) + logdetvar + quadratic)$</span></p>
  <p>where logdetvar is the log determinant of the variance or <span class="math-container">$\log(det(M)) = trace(\log(M))$</span></p>
  <p>where quadratic seems to be the row sum of (<span class="math-container">$x - \mu$</span>) and the negative variance.</p>
  <p>and finally <span class="math-container">$n$</span> is the number of rows in the log variance.</p>
  <p>So example exmaple 1 doesn't look like the either pdf.</p>
  <p><a href="https://github.com/pytorch/pytorch/blob/main/torch/distributions/normal.py" rel="nofollow noreferrer">Example 2: Stable baselines 3 normal distribution</a></p>
  <p>Stable baselines defines the log probability of the normal distribution as:</p>
  <p><span class="math-container">$-((X - \mu)^2) / (2 \times \sigma^2) - \log(\sigma) - \log((2\pi)^2) $</span></p>
  <p>Again nothing like the pdf of the normal. Can someone explain how they are deriving these different implementations, and where do they come from?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-08T21:52:18.283Z
Comments.Text: CDF is the cumulative distribution function. If you’re using RL I’d be amazed if what you really want is the PDF (probability density function)
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-08T22:11:11.643Z
Comments.Text: I am unsure what the statement is saying, I never mentioned the PDF.
Comments.UserDisplayName: chadmc
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-08T23:58:15.603Z
Comments.Text: I am saying, if you are doing RL you will almost certainly not be wanting the CDF, you are wanting the PDF. In SAC, the log of the policy (note, this is _not_ a log-normal distribution) is simply the log of the PDF (in a continuous case) evaluated at the given action.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-09T08:58:55.570Z
Comments.Text: Please, don't provide screenshots. Provide the code, if that's really necessary. We can't directly copy from screenshots and that could be a problem.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-09T11:45:55.043Z
Comments.Text: The screenshot gives the context I need, I also provided the code repo, the screen shot is what I am asking about. The code is running fine, how can I ask the question without the screenshot? What do you mean by provide the code, I linked the repo.
Comments.UserDisplayName: chadmc
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-09T15:33:49.550Z
Comments.Text: @DavidIreland is completely right. In SAC you only need the log-probability (not CDF) of actions; the policy is still Gaussian (actually is a SquashedGaussian by tanh to bound actions in $[-1, 1]$), you have the policy net to predict its `mean` and `log-variance` (which is later properly exponentiated.) In papers, if you see $\log \pi(a\mid s)$ it's a way to say "the log probability of predicted action $a$ from the policy $\pi$ in state $s$".
Comments.UserDisplayName: Luca Anzalone
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-09T17:35:39.940Z
Comments.Text: @LucaAnzalone Thank you, that was very helpful. I updated the question. What is up with all the downvotes, geeze. I updated when people asked me to change.
Comments.UserDisplayName: chadmc
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-09T17:52:08.910Z
Comments.Text: I'm new to this community, I read the guidelines, I didn't see any violations that I made, if I need to make an edit or clarify I'm happy to. Please be nice, tell me what you aren't liking and I'll do better on future posts.
Comments.UserDisplayName: chadmc
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-09T20:05:49.120Z
Comments.Text: @chadmc you should really remove the images of the equations and write them yourself.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-09T21:07:04.697Z
Comments.Text: I'll do that, just to understand the community better, why would having images matter. Does it just look cleaner to hand type them?
Comments.UserDisplayName: chadmc
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-09T21:15:33.257Z
Comments.Text: @chadmc yes, it looks much much much cleaner to type the equations
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-09T21:24:55.137Z
Comments.Text: It's done thank you for the clarification.
Comments.UserDisplayName: chadmc
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-09T22:59:04.180Z
Comments.Text: @chadmc Note that you can provide a link to the original code or even copy and paste the code. The screenshots of the code are not good because it's difficult then for people to refer to their content.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-10T16:57:02.347Z
Comments.Text: @nbro thank you I didn't see how to do that, is there a place on here I can find the guide for that?
Comments.UserDisplayName: chadmc
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-11T09:03:59.843Z
Comments.Text: @chadmc Including code here is the same as including code on Stack Overflow, which you already did. Anyway, I recommend that you read our on-topic page: https://ai.stackexchange.com/help/on-topic, if you haven't yet.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-19T18:18:09.387Z
Comments.Text: @nbro Implementation questions in the context of understanding the theoretical topics are on-topic. For example, if a theoretical topic is described by a certain mathematical formula and you want to understand how a certain implementation is related to the formula, then your question is on-topic.     So, I'm not sure what you are getting at?
Comments.UserDisplayName: chadmc
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-19T18:23:03.017Z
Comments.Text: @nbro Not trying to be rude, but you are definitely being more of a gatekeeper than helpful. Feel like I'm wasting my time pandering to someone who is not even trying to answer my question.
Comments.UserDisplayName: chadmc
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-20T01:54:01.833Z
Comments.Text: @chadmc You're free to answer or ignore my comments. Mine was a recommendation for you. I was not referring to any specific thing in the on-topic. It was a general suggestion for you to get familiar with the site. Having said that, I think in the past I've answered a similar question (although I didn't even fully read your post), but now I don't want to waste my time searching for it. You can search for it. I wasn't planning to answer your question. I'm here to make this community a better place not just by answering questions, which I already did a lot.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><deep-rl><soft-actor-critic>
Title: Where does the term $\log \mu(u \mid s)$ come from?
UserDisplayName: chadmc
ThreadText: # Where does the term $\log \mu(u \mid s)$ come from?
  
  (Asked by: username_0 on 2023-05-08)
  
  This question comes from trying to build a SAC model. The action space is derived from a log normal distribution.
  
  If in the appendix c of the original paper the equation for the log policy is:
  
  $\log \pi(a|s) = \log \mu(u|s) - \sum^D\_{i = 1} \log (1 - \tanh^2(u\_i))$
  
  where $u \in R^D$ is a random variable, $\mu(u|s)$ is the corresponding density wit infinite support (in this case being a Gaussian distribution.)
  
  So, my question is, according to the code, the log probabilities are calculated using the following:
  
  $\log \mu(u|s) = -0.5 \times \log(2 \times \pi) \times n + \sum (-0.5 \times \epsilon^2 - \log \sigma^2) $
  
  where $\epsilon \sim \mathcal{N}(0,1)$
  
  n = size of log\_std vector
  
  The implementation on stable baseline is more complicated.
  
  I want to know where this formula come from because doesn't look normal pdf or log normal pdf.
  
  \begin{equation}
  \frac{1}{\sigma \sqrt{2 \pi}} e^{\frac{1}{2}(\frac{x-\mu}{\sigma})^2} \\
  \frac{1}{x \sigma \sqrt{2 \pi}} e^{ -( \frac{\log(x-\mu)^2}{2\sigma^2})}
  \end{equation}
  
  Further elaborations on implementations that i'm trying to understand:
  
  [Example 1: Look up log\_proba here](https://coax.readthedocs.io/en/latest/_modules/coax/proba_dists/_normal.html#log_proba)
  
  The log probability is defined as:
  
  $\log \pi\_\theta = -0.5 \times (n \times \log(2 \pi) + logdetvar + quadratic)$
  
  where logdetvar is the log determinant of the variance or $\log(det(M)) = trace(\log(M))$
  
  where quadratic seems to be the row sum of ($x - \mu$) and the negative variance.
  
  and finally $n$ is the number of rows in the log variance.
  
  So example exmaple 1 doesn't look like the either pdf.
  
  [Example 2: Stable baselines 3 normal distribution](https://github.com/pytorch/pytorch/blob/main/torch/distributions/normal.py)
  
  Stable baselines defines the log probability of the normal distribution as:
  
  $-((X - \mu)^2) / (2 \times \sigma^2) - \log(\sigma) - \log((2\pi)^2) $
  
  Again nothing like the pdf of the normal. Can someone explain how they are deriving these different implementations, and where do they come from?
  
  **Tags:** reinforcement-learning, deep-rl, soft-actor-critic
  
  > **Comments:**
  > **username_1**: CDF is the cumulative distribution function. If you’re using RL I’d be amazed if what you really want is the PDF (probability density function)  
  > **username_0**: I am unsure what the statement is saying, I never mentioned the PDF.  
  > **username_1**: I am saying, if you are doing RL you will almost certainly not be wanting the CDF, you are wanting the PDF. In SAC, the log of the policy (note, this is _not_ a log-normal distribution) is simply the log of the PDF (in a continuous case) evaluated at the given action.  
  > **username_2**: Please, don't provide screenshots. Provide the code, if that's really necessary. We can't directly copy from screenshots and that could be a problem.  
  > **username_0**: The screenshot gives the context I need, I also provided the code repo, the screen shot is what I am asking about. The code is running fine, how can I ask the question without the screenshot? What do you mean by provide the code, I linked the repo.  
  > **username_3**: @DavidIreland is completely right. In SAC you only need the log-probability (not CDF) of actions; the policy is still Gaussian (actually is a SquashedGaussian by tanh to bound actions in $[-1, 1]$), you have the policy net to predict its `mean` and `log-variance` (which is later properly exponentiated.) In papers, if you see $\log \pi(a\mid s)$ it's a way to say "the log probability of predicted action $a$ from the policy $\pi$ in state $s$".  
  > **username_0**: @username_3 Thank you, that was very helpful. I updated the question. What is up with all the downvotes, geeze. I updated when people asked me to change.  
  > **username_0**: I'm new to this community, I read the guidelines, I didn't see any violations that I made, if I need to make an edit or clarify I'm happy to. Please be nice, tell me what you aren't liking and I'll do better on future posts.  
  > **username_1**: @username_0 you should really remove the images of the equations and write them yourself.  
  > **username_0**: I'll do that, just to understand the community better, why would having images matter. Does it just look cleaner to hand type them?  
  > **username_1**: @username_0 yes, it looks much much much cleaner to type the equations  
  > **username_0**: It's done thank you for the clarification.  
  > **username_2**: @username_0 Note that you can provide a link to the original code or even copy and paste the code. The screenshots of the code are not good because it's difficult then for people to refer to their content.  
  > **username_0**: @username_2 thank you I didn't see how to do that, is there a place on here I can find the guide for that?  
  > **username_2**: @username_0 Including code here is the same as including code on Stack Overflow, which you already did. Anyway, I recommend that you read our on-topic page: https://ai.stackexchange.com/help/on-topic, if you haven't yet.  
  > **username_0**: @username_2 Implementation questions in the context of understanding the theoretical topics are on-topic. For example, if a theoretical topic is described by a certain mathematical formula and you want to understand how a certain implementation is related to the formula, then your question is on-topic.     So, I'm not sure what you are getting at?  
  > **username_0**: @username_2 Not trying to be rude, but you are definitely being more of a gatekeeper than helpful. Feel like I'm wasting my time pandering to someone who is not even trying to answer my question.  
  > **username_2**: @username_0 You're free to answer or ignore my comments. Mine was a recommendation for you. I was not referring to any specific thing in the on-topic. It was a general suggestion for you to get familiar with the site. Having said that, I think in the past I've answered a similar question (although I didn't even fully read your post), but now I don't want to waste my time searching for it. You can search for it. I wasn't planning to answer your question. I'm here to make this community a better place not just by answering questions, which I already did a lot.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2023-05-09. Score: 1 | Accepted answer
  
  The SAC algorithm was designed for control and robotics tasks in mind, i.e. environments with continuous states and actions spaces.
  
  In practice, SAC implements a policy that is called a **Squashed Gaussian** (read [here](https://spinningup.openai.com/en/latest/algorithms/sac.html)): is a Gaussian distribution that is bounded (or squashed) to output actions within the $[-1, 1]$ interval by a $\tanh$ operation. Intuitively, you can think of it as $a \sim \tanh(\mathcal{N}(\mu, \Sigma))$. This is because the Gaussian is unbounded having support in $[-\infty, \infty]$, and so limiting the actions was done by truncating the distribution which is not much clever, and in fact usually results in lower performance: I guess because you have no gradients when actions fall outside the desired range, instead the tanh provides a *continuous yet bounded* interval. (An alternative to the squashed gaussian could be a Beta distribution that is in $[0, 1]$, instead, as motivated [here](https://proceedings.mlr.press/v70/chou17a.html) - but I think almost nobody uses that.)
  
  Now, the term $\mu(u\mid s)$ corresponds to the underlying Gaussian distribution that outputs/samples actions $u$, providing they probability to occur. But since the $\tanh$ is applied you have to *correct* such probability, since you want to evaluate $\log \pi(a\mid a)$ instead of $\log \mu(u\mid s)$. In few words, the correction factor is equal to the inverse of the *Jacobian* determinant of the tanh transformation. So you compute the probability of the Gaussian policy $\mu(u\mid s)$ then multiply by the Jacobian, and finally take the log to obtain the log-prob of the squashed actions $a$.
  
  If you're interested in how this is done in practice check the [official](https://github.com/haarnoja/sac/blob/master/sac/policies/gaussian_policy.py#L131) and these ([1](https://coax.readthedocs.io/en/latest/_modules/coax/proba_dists/_squashed_normal.html#SquashedNormalDist) and [2](https://github.com/DLR-RM/stable-baselines3/blob/master/stable_baselines3/common/distributions.py#L227-L240)) other implementations. As a side note, if you use tools like tensorflow-probability you can implement this quite easily by applying a Tanh bijector to a Normal distribution.
  
  ---
  
  **Update:** log of the Normal pdf.
  The Normal $\mathcal{N}(\mu,\sigma^2)$ pdf of a sample $x$ (e.g. actions) is defined as follows:
  
  $$\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac12 \big(\frac{x-\mu}{\sigma}\big)}$$
  
  Now, in RL you care about the *logarithm* of such probability density. So:
  
  $$\log\Big(\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac12 \big(\frac{x-\mu}{\sigma}\big)}\Big) \\ = \log\Big(\frac{1}{\sigma\sqrt{2\pi}}\Big) + \log\Big(e^{-\frac12 \big(\frac{x-\mu}{\sigma}\big)}\Big) \\ = -\log\sigma -\log\sqrt{2\pi} - \frac12\big(\frac{x-\mu}{\sigma}\big)^2$$
  
  Which is the same formula SB and pytorch implements.
  
  > **Comments:**
  > **username_1**: I would say that I think the `tanh` transformation makes sense for an action space that is symmetric, like the MuJoCo tasks which were the benchmark environments in the original paper.  
  > **username_3**: Sure, I guess that one (good at math) can extend this idea to a `sigmoid`, or in general to even `affine` transformations (scaling and biasing) to handle whatever action space.  
  > **username_1**: Yes that's right, you can use the pdf transformation method for arbitrary transformations (assuming the function is one-to-one). I think the best is some combination of `tanh` or `sigmoid` followed by an affine transformation and you can satisfy most action space bounds.  
  > **username_0**: Hey I really appreciate the answer and I'll definitely accept it.  
  > **username_0**: I still am unsure about the implementation, which is why I posted the code. I looked at the implementation you posted. You linked to the the squashed gaussian. This function calls the log probability of the normal distribution and uses that as the value of log p. The log prob function on this page has yet another different implementation, https://coax.readthedocs.io/en/latest/_modules/coax/proba_dists/_normal.html. I have been trying to get an answer to that implementation or the code I posted. The information you have been responding to is in the appendix c of both SAC papers.  
  > **username_0**: I am really looking for where the implementations come from, where are the equations coming from for the log probability of the normal distribution before it is transformed via change of variable. How do you compute the Gaussian log probability. The implementation is so different for Stable baselines, your code, the code I posted. Again you have been awesome, thank you for your patients.  
  > **username_3**: @username_0 I found the official implementation (I've added the link in the answer) and they follow exactly the equation 26 in the appendix C of the paper. For the coax implementation that is different, I'm not completely sure, but it can to improve the numerical stability of the log of the arctanh ($1-\tanh^2$)  
  > **username_0**: Let's not close the post, I want time to really dig deep on the implementations, I'll try to update by tomorrow. @username_3 thanks again I've enjoyed working with you.  
  > **username_1**: @username_0 the ‘log probability’ of a normal distribution is literally just the log of the pdf evaluated at a given point in the support (in this case in your sampled action, but presumably the action was sampled from a squashed normal distribution so you want to evaluate the log pdf of _that_ distribution)  
  > **username_0**: I really appreciate insight @DavidIreland, the implementations in the code bases doesn't quite seem to match the pdf or at least they don't seem to I could be missing something, I'll update my post to show you what I'm talking about after work hopefully tonight, but maybe tomorrow.  
  > **username_1**: If the code does `.log_prob(action)` of the distribution object, then it is evaluating the log pdf (this is how it works in PyTorch at least).  
  > **username_0**: @DavidIreland I've been very busy, but I added more context, does that clear up the question?  
  > **username_3**: @username_0 I've updated my answer to include a derivation of the log-pdf of the Gaussian, have a look  
  > **username_1**: It’s hard to read your edits because you’ve not used mathematical notation (eg written the word ‘sigma’ instead of $\sigma$) but what you have for the stable baseline example looks precisely like the log-pdf of the normal distribution so I’m not sure why you’re confused.  
  > **username_3**: @DavidIreland strange.. I used MathJax and I can see it rendered correctly  
  > **username_1**: Sorry @username_3, I was talking to OP. Your update is great and should clear the confusion for OP!  
  > **username_0**: @username_3 Thank you very much, you really have been great. Wish I could upvote 2x.  
  
  ---
  
  