Answers.Body: <p>The policy gradient tells us that
  <span class="math-container">$$\nabla_\theta v_\pi(s) = \mathbb{E}_\pi\left[ G_t \nabla_\theta \log \pi_\theta(a | s) \right] \; ;$$</span>
  where as usual <span class="math-container">$G_t$</span> are the returns, <span class="math-container">$\pi$</span> is policy parameterised by <span class="math-container">$\theta$</span> and the expectation is taken with respect to the policy <span class="math-container">$\pi$</span> and the state distribution induced by <span class="math-container">$\pi$</span>.</p>
  <p>Hopefully you can see that here we do not need to take the log of the actions, but instead the log of the probability of taking an action (or, in a continuous action case, the log of the pdf of taking an action), i.e. the log-likelihood of an action.</p>
  <p>In your case, as with most MuJoCo environments, it is common to train a neural network to output the mean and standard deviation of a Gaussian distribution -- assuming the action has <span class="math-container">$d$</span> dimensions, this is done by having a neural network with two heads, i.e. a neural network that takes the input and outputs two <span class="math-container">$d$</span> length vectors. Note that in practice, it is common for the network to output the <em>log</em>-standard deviation for stability reasons. Further, it is typically assumed that the log-sd is clamped to some pre-specified range too, again to help with numerical stability. For <span class="math-container">$d &gt; 1$</span> we assume that the Multi-Variate Gaussian has diagonal covariance matrix, so the neural network still need only predict a <span class="math-container">$d$</span> length vector for the log-sd, as opposed to a full covariance matrix.</p>
  <p>Now, as the action range for the MuJoCo environments is typically symmetric, it is also common to see that a tanh activation is applied <em>to the action</em> to get the action to be in the range (-1, 1). This makes things a bit more nuanced in terms of how to apply the policy gradient, because your random variable (the action) is no longer gaussian. Luckily, as we have assumed independent gaussians, the resulting distribution of the random variable is not that tricky to calculate analytically.</p>
  <p>If we assume that <span class="math-container">$A$</span> is the random variable representing the action, then <span class="math-container">$A$</span> is defined as <span class="math-container">$A = \tanh(X)$</span>, where <span class="math-container">$X \sim \mbox{MVN}(\boldsymbol{\mu}_\theta, \boldsymbol{\Sigma}_\theta)$</span>, where <span class="math-container">$\boldsymbol{\mu}, \boldsymbol{\Sigma}$</span> are the predicted mean and (diagonal) co-variance parameters of the gaussian, with the <span class="math-container">$\theta$</span> subscript added to remind us of the fact that they are predicted by the NN parameterised by <span class="math-container">$\theta$</span>. Now, we can use the fact that the pdf of an arbitrary (one-to-one) function of a random variable defined by <span class="math-container">$Y = g(X)$</span> is given (for the univariate case) by
  <span class="math-container">$$f_Y(y) = f_X\left(g^{-1}(y)\right) \left|\frac{dx}{dy}\right| \; ;$$</span>
  note that the multivariate case is a simple extension of this and details can be found online.</p>
  <p>In our example, <span class="math-container">$g = \tanh$</span>, and so the pdf of the policy is given by
  <span class="math-container">$$\pi_\theta(\textbf{a}|s) = \phi_\theta(\textbf{x}|s)\left|\mbox{det}\left(\frac{d\textbf{x}}{d\textbf{a}}\right)\right| = \phi_\theta(\textbf{x}|s)\left|\mbox{det}\left(\frac{d\textbf{a}}{d\textbf{x}}\right)\right|^{-1} \; ;$$</span>
  where <span class="math-container">$\textbf{x} \sim X$</span> and <span class="math-container">$\phi$</span> is the pdf of the MVN with <span class="math-container">$\theta$</span> notation to imply that the mean and covariance come from the NN prediction. Note that we invert the determinant of the transformation because it is often easier to calculate the derivative this way; in the case of <span class="math-container">$tanh$</span> being the transformation function along with the fact that we assume the MVN has diagonal covariance matrix we get that <span class="math-container">$\frac{d\textbf{a}}{d\textbf{x}} = \mbox{diag}(1 - \tanh^2(\textbf{x}))$</span>. Now, using this, you can evaluate <span class="math-container">$\log \pi_\theta(a|s)$</span> in the policy gradient term correctly by using the correct pdf.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-10T05:56:48.603Z
Answers.Text: I understood up to the 5th para. I couldn't understand what is fy = fx(g(x)|dx/dy|. Can you please elaborate more or suggest me some content to understand that?
Answers.UserDisplayName: Himanshu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-10T10:01:06.963Z
Answers.Text: If you go to the [wiki page for a random variable](https://en.m.wikipedia.org/wiki/Random_variable) and click the section ‘function of a random variable’ it should explain what this paragraph means, and also has some examples you can work through.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-11T07:13:57.283Z
Answers.Text: I have updated my question and included the code. Please help me find the reason why my agent is note converging.
Answers.UserDisplayName: Himanshu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-08T10:36:50.053Z
Answers.LastActivityDate: 2022-03-08T10:36:50.053Z
Answers.UserDisplayName: David
Body: <p>I am currently trying to train BipedalWalker of OpenAI gym by using policy gradient approach. My action space contains 4 continuous actions, all ranging [-1.0, 1.0]. In this case, how can we calculate gradient of log policy when <code>log(negative_action)</code> gives <code>nan</code> value.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-05T09:53:39.927Z
Comments.Text: You don’t want to take log of the action. You want the log likelihood of taking an action. If your network outputs parameters of a Gaussian distribution then you would want to evaluate $\log f(a_t| \mu, \sigma)$, where $f$ is the pdf of the standard normal. Note that a tanh activation is usually wrapped around the action when the action space is symmetric in these MuJoCo environments and so the gradient will change slightly as your distribution is now $Y = \text{tanh}(X)$.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-06T11:25:21.243Z
Comments.Text: Correct me if I am wrong. Mean, SD = Policy_Model(current_state). Then, action = random.gauss(mean, SD). Then probability_density_function = exp(-0.5((x-mu)/SD)^2)/(SD√2pi). Then, log of this function then gradient. Also, I didn't get why we use tanh in the policy_models last layer.
Comments.UserDisplayName: Himanshu
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-06T11:26:14.880Z
Comments.Text: Why are we using tanh for getting mu and sd.
Comments.UserDisplayName: Himanshu
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-06T13:33:53.663Z
Comments.Text: You are correct that action comes from sampling the Gaussian. However, the support of the Gaussian random variable is $\mathbb{R}$, so your action can be any real number. As you want it to be constrained to $[-1, 1]$ you apply the tanh function to enforce this.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-06T13:37:02.253Z
Comments.Text: Am I correct about the probability density function. Also, should I use separate gaussian parameters for all four i.e.,  [mu1, mu2, mu3, mu4, sig1, sig2, sig3, sig4] = policy_model(current_state) Or should I use multivariate guassian distribution which has a covariance matrix.
Comments.UserDisplayName: Himanshu
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-06T20:10:07.827Z
Comments.Text: Yes, you had the details of the Gaussian pdf correct. 
  
  Typically what is done is use a multi variate Gaussian with diagonal covariants matrix — this is equivalent to using independent gaussians.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-07T11:18:54.957Z
Comments.Text: Correct me I am wrong. I need to use tanh at the end of policy network to limit the mu, SD. Or do I need to use tanh on the sampled actions. Can we use any other non linearity in place of tanh, like clamped.
Comments.UserDisplayName: Himanshu
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-07T16:31:09.377Z
Comments.Text: You have to use it on the sampled actions. You _could_ use a clamped activation, but this would make the resulting pdf more complex than using tanh.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-07T16:55:42.683Z
Comments.Text: If i use tanh on limiting the parameters, the training goes normal but I've not got any useful results so far. But if I use tanh on actions the gradients become unstable, they become either 0 of Infinity. What should I do then?
Comments.UserDisplayName: Himanshu
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-07T17:26:04.293Z
Comments.Text: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/134623/discussion-between-himanshu-and-david-ireland).
Comments.UserDisplayName: Himanshu
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-07T19:32:30.690Z
Comments.Text: I will add a more detailed answer if/when I get the opportunity including some tips for implementation details, hopefully within 1-2 days.
Comments.UserDisplayName: David
Tags: <reinforcement-learning><machine-learning><deep-rl><open-ai>
Title: How to take gradient of log policy when actions are negative?
UserDisplayName: Himanshu
ThreadText: # How to take gradient of log policy when actions are negative?
  
  (Asked by: username_0 on 2022-03-05)
  
  I am currently trying to train BipedalWalker of OpenAI gym by using policy gradient approach. My action space contains 4 continuous actions, all ranging [-1.0, 1.0]. In this case, how can we calculate gradient of log policy when `log(negative_action)` gives `nan` value.
  
  **Tags:** reinforcement-learning, machine-learning, deep-rl, open-ai
  
  > **Comments:**
  > **username_1**: You don’t want to take log of the action. You want the log likelihood of taking an action. If your network outputs parameters of a Gaussian distribution then you would want to evaluate $\log f(a_t| \mu, \sigma)$, where $f$ is the pdf of the standard normal. Note that a tanh activation is usually wrapped around the action when the action space is symmetric in these MuJoCo environments and so the gradient will change slightly as your distribution is now $Y = \text{tanh}(X)$.  
  > **username_0**: Correct me if I am wrong. Mean, SD = Policy_Model(current_state). Then, action = random.gauss(mean, SD). Then probability_density_function = exp(-0.5((x-mu)/SD)^2)/(SD√2pi). Then, log of this function then gradient. Also, I didn't get why we use tanh in the policy_models last layer.  
  > **username_0**: Why are we using tanh for getting mu and sd.  
  > **username_1**: You are correct that action comes from sampling the Gaussian. However, the support of the Gaussian random variable is $\mathbb{R}$, so your action can be any real number. As you want it to be constrained to $[-1, 1]$ you apply the tanh function to enforce this.  
  > **username_0**: Am I correct about the probability density function. Also, should I use separate gaussian parameters for all four i.e.,  [mu1, mu2, mu3, mu4, sig1, sig2, sig3, sig4] = policy_model(current_state) Or should I use multivariate guassian distribution which has a covariance matrix.  
  > **username_1**: Yes, you had the details of the Gaussian pdf correct.   Typically what is done is use a multi variate Gaussian with diagonal covariants matrix — this is equivalent to using independent gaussians.  
  > **username_0**: Correct me I am wrong. I need to use tanh at the end of policy network to limit the mu, SD. Or do I need to use tanh on the sampled actions. Can we use any other non linearity in place of tanh, like clamped.  
  > **username_1**: You have to use it on the sampled actions. You _could_ use a clamped activation, but this would make the resulting pdf more complex than using tanh.  
  > **username_0**: If i use tanh on limiting the parameters, the training goes normal but I've not got any useful results so far. But if I use tanh on actions the gradients become unstable, they become either 0 of Infinity. What should I do then?  
  > **username_0**: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/134623/discussion-between-username_0-and-username_1-ireland).  
  > **username_1**: I will add a more detailed answer if/when I get the opportunity including some tips for implementation details, hopefully within 1-2 days.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-03-08. Score: 1
  
  The policy gradient tells us that
  $$\nabla\_\theta v\_\pi(s) = \mathbb{E}\_\pi\left[ G\_t \nabla\_\theta \log \pi\_\theta(a | s) \right] \; ;$$
  where as usual $G\_t$ are the returns, $\pi$ is policy parameterised by $\theta$ and the expectation is taken with respect to the policy $\pi$ and the state distribution induced by $\pi$.
  
  Hopefully you can see that here we do not need to take the log of the actions, but instead the log of the probability of taking an action (or, in a continuous action case, the log of the pdf of taking an action), i.e. the log-likelihood of an action.
  
  In your case, as with most MuJoCo environments, it is common to train a neural network to output the mean and standard deviation of a Gaussian distribution -- assuming the action has $d$ dimensions, this is done by having a neural network with two heads, i.e. a neural network that takes the input and outputs two $d$ length vectors. Note that in practice, it is common for the network to output the *log*-standard deviation for stability reasons. Further, it is typically assumed that the log-sd is clamped to some pre-specified range too, again to help with numerical stability. For $d > 1$ we assume that the Multi-Variate Gaussian has diagonal covariance matrix, so the neural network still need only predict a $d$ length vector for the log-sd, as opposed to a full covariance matrix.
  
  Now, as the action range for the MuJoCo environments is typically symmetric, it is also common to see that a tanh activation is applied *to the action* to get the action to be in the range (-1, 1). This makes things a bit more nuanced in terms of how to apply the policy gradient, because your random variable (the action) is no longer gaussian. Luckily, as we have assumed independent gaussians, the resulting distribution of the random variable is not that tricky to calculate analytically.
  
  If we assume that $A$ is the random variable representing the action, then $A$ is defined as $A = \tanh(X)$, where $X \sim \mbox{MVN}(\boldsymbol{\mu}\_\theta, \boldsymbol{\Sigma}\_\theta)$, where $\boldsymbol{\mu}, \boldsymbol{\Sigma}$ are the predicted mean and (diagonal) co-variance parameters of the gaussian, with the $\theta$ subscript added to remind us of the fact that they are predicted by the NN parameterised by $\theta$. Now, we can use the fact that the pdf of an arbitrary (one-to-one) function of a random variable defined by $Y = g(X)$ is given (for the univariate case) by
  $$f\_Y(y) = f\_X\left(g^{-1}(y)\right) \left|\frac{dx}{dy}\right| \; ;$$
  note that the multivariate case is a simple extension of this and details can be found online.
  
  In our example, $g = \tanh$, and so the pdf of the policy is given by
  $$\pi\_\theta(\textbf{a}|s) = \phi\_\theta(\textbf{x}|s)\left|\mbox{det}\left(\frac{d\textbf{x}}{d\textbf{a}}\right)\right| = \phi\_\theta(\textbf{x}|s)\left|\mbox{det}\left(\frac{d\textbf{a}}{d\textbf{x}}\right)\right|^{-1} \; ;$$
  where $\textbf{x} \sim X$ and $\phi$ is the pdf of the MVN with $\theta$ notation to imply that the mean and covariance come from the NN prediction. Note that we invert the determinant of the transformation because it is often easier to calculate the derivative this way; in the case of $tanh$ being the transformation function along with the fact that we assume the MVN has diagonal covariance matrix we get that $\frac{d\textbf{a}}{d\textbf{x}} = \mbox{diag}(1 - \tanh^2(\textbf{x}))$. Now, using this, you can evaluate $\log \pi\_\theta(a|s)$ in the policy gradient term correctly by using the correct pdf.
  
  > **Comments:**
  > **username_0**: I understood up to the 5th para. I couldn't understand what is fy = fx(g(x)|dx/dy|. Can you please elaborate more or suggest me some content to understand that?  
  > **username_1**: If you go to the [wiki page for a random variable](https://en.m.wikipedia.org/wiki/Random_variable) and click the section ‘function of a random variable’ it should explain what this paragraph means, and also has some examples you can work through.  
  > **username_0**: I have updated my question and included the code. Please help me find the reason why my agent is note converging.  
  
  ---
  
  