Answers.Body: <p>It's true that computing the log prob of a sample from a gaussian requires inverting a matrix and dealing with the determinant in the general case of a full covariance matrix. If you wanted to backprop through the log prob, you'd need to backprop through these operations, which by the way are available as differentiable operations in pytorch. See the gaussian distribution class which deals with these operations: <a href="https://github.com/pytorch/pytorch/blob/master/torch/distributions/multivariate_normal.py#L177" rel="nofollow noreferrer">https://github.com/pytorch/pytorch/blob/master/torch/distributions/multivariate_normal.py#L177</a> 
  You can see there's a <code>half_log_det</code> and although you won't find an explicit matrix inversion, you'll see that it uses a differentiable triangular linear solve (<code>torch.trtrs</code>) at some point, which is essentially doing a matrix inversion in a more efficient way.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-15T09:47:37.460Z
Answers.LastActivityDate: 2019-02-15T09:47:37.460Z
Answers.UserDisplayName: yewang
Body: <p>I've noticed that when modelling a continuous action space, the default thing to do is to estimate a mean and a variance where each is parameterized by a neural network or some other model. </p>
  
  <p>I also often see that it is one network <span class="math-container">$\theta$</span> models both. The REINFORCE objective can be written as </p>
  
  <p><span class="math-container">$$\nabla \mathcal{J}(\theta) = \mathbb{E}_{\pi} [\nabla_\theta \log \pi(a_t|s_t) * R_t] $$</span></p>
  
  <p>For discrete action space this makes sense since the output of the network is determined by a softmax. However, if we explicitly model the output of the network as a Gaussian, then the gradient of the log likelihood is of a different form,</p>
  
  <p><span class="math-container">$$\pi_\theta(a_t|s_t) = Normal(\mu_\theta(s_t), \Sigma_\theta(s_t))$$</span></p>
  
  <p>and the log is:</p>
  
  <p><span class="math-container">$$\log \pi_\theta(a_t | s_t) =  -\frac{1}{2} (a_t-\mu_\theta)^\top \Sigma^{-1}_\theta(a_t-\mu_\theta) + \log 2 \pi \det({\Sigma_\theta})$$</span></p>
  
  <p>In the slides provided here (slide 18):
  <a href="http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/pg.pdf" rel="nofollow noreferrer">http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/pg.pdf</a></p>
  
  <p>IF the variance is held constant, then we can solve this analytically:</p>
  
  <p><span class="math-container">$$\nabla_\theta \log \pi_\theta(a_t|s_t) = (a_t - \mu_\theta) \Sigma^{-1} \phi(s)$$</span></p>
  
  <p>But, are things always modelled assuming a constant variance? If it's not constant then we have to account for the inverse of the covariance matrix 
  as well as the determinant? </p>
  
  <p>I've taken a look at code online and from what I've seen, most of them assume the variance is constant. </p>
  
  <hr>
  
  <p>@NielSlater</p>
  
  <p>Using the reparameterization trick we would use a normal distribution with fixed parameters 0 and 1. 
  <span class="math-container">$$ a_t \sim \mu_\theta(s_t) + \Sigma_\theta(s_t) * Normal(0, 1) $$</span> </p>
  
  <p>Which is the same as if we <em>had</em> actually sampled directly from a distribution, <span class="math-container">$ \pi_\theta(a_t | s_t) = Normal(\mu_\theta(s_t), \Sigma_\theta(s_t))$</span> and let's us calculate the corresponding <span class="math-container">$\log \pi_\theta(a_t|s_t)$</span> and <span class="math-container">$\nabla_\theta \log \pi_\theta(a_t | s_t)$</span> without having to differentiate through the actual density.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-13T13:10:41.563Z
Comments.Text: In your formulas, what is the variance? I understand that $\mu_\theta$ is the mean with respect to the current parameters $\theta$. $\Sigma$ is the covariance matrix, but of which random variables? I think it may be helpful to specify these details, so that readers can better understand your question. In general, I think you should try to explain the formulas that you provide, if possible.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-13T13:14:15.890Z
Comments.Text: Hi, the covariance is parameterized by $\theta$ in the second equation. In the fourth equation I leave out the $\theta$ because in the slide is says in the case where the covariance is constant.
Comments.UserDisplayName: tryingtolearn
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-13T13:23:33.637Z
Comments.Text: But is this the covariance of which variables? Maybe it is the covariance of the state and action (assuming they are considered random variables in this case), even though you denote it just as $\Sigma_\theta(s_t)$ instead of maybe $\Sigma_\theta(s_t, a_t)$. Or maybe $\Sigma_\theta(s_t)$ just represents the variance of $s_t$? Honestly, I'm not sure what that means, anyway.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-13T13:29:32.177Z
Comments.Text: @nbro I'm not sure I understand. It is the mean and covariance that models a distribution over actions which I have assumed to be gaussian. In reinforcement learning this is quite a common thing to do I believe. Each of those is estimated from the same neural network that is parameterized by $\theta$.
Comments.UserDisplayName: tryingtolearn
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-13T13:48:01.420Z
Comments.Text: This part of RL maths is something I am not yet familiar with. However, I believe the answer will be based around the ["the reparameterization trick"](https://medium.com/@llionj/the-reparameterization-trick-4ff30fe92954) . . . if no answer is given and you can figure out the link to the equations you have given, actually I'd be interested to see this Normal distribution with $\mu$ and $\sigma$ outputs from the NN worked through with an example . . .
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-13T14:53:32.010Z
Comments.Text: @NeilSlater I've included a short edit of how (I think) it would work with the reparameterization trick. Though I'd heard of it used in VAEs, I'd honestly never heard it mentioned in the context of RL and going through a lot of code I'm surprised to see that the majority of modelling is done with constant variance. Seemed strange!
Comments.UserDisplayName: tryingtolearn
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-13T17:21:33.123Z
Comments.Text: Both VAEs and RL policy gradients with stochastic policies use the trick. My understanding is that you cannot propagate gradient through a sampling node otherwise. And the initial error signal in RL relates to the actual action chosen, not the values of the NN output.
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><policy-gradients>
Title: Calculating gradient for log policy when variance is not constant
UserDisplayName: tryingtolearn
ThreadText: # Calculating gradient for log policy when variance is not constant
  
  (Asked by: username_0 on 2019-02-13)
  
  I've noticed that when modelling a continuous action space, the default thing to do is to estimate a mean and a variance where each is parameterized by a neural network or some other model.
  
  I also often see that it is one network $\theta$ models both. The REINFORCE objective can be written as
  
  $$\nabla \mathcal{J}(\theta) = \mathbb{E}\_{\pi} [\nabla\_\theta \log \pi(a\_t|s\_t) \* R\_t] $$
  
  For discrete action space this makes sense since the output of the network is determined by a softmax. However, if we explicitly model the output of the network as a Gaussian, then the gradient of the log likelihood is of a different form,
  
  $$\pi\_\theta(a\_t|s\_t) = Normal(\mu\_\theta(s\_t), \Sigma\_\theta(s\_t))$$
  
  and the log is:
  
  $$\log \pi\_\theta(a\_t | s\_t) = -\frac{1}{2} (a\_t-\mu\_\theta)^\top \Sigma^{-1}\_\theta(a\_t-\mu\_\theta) + \log 2 \pi \det({\Sigma\_\theta})$$
  
  In the slides provided here (slide 18):
  <http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/pg.pdf>
  
  IF the variance is held constant, then we can solve this analytically:
  
  $$\nabla\_\theta \log \pi\_\theta(a\_t|s\_t) = (a\_t - \mu\_\theta) \Sigma^{-1} \phi(s)$$
  
  But, are things always modelled assuming a constant variance? If it's not constant then we have to account for the inverse of the covariance matrix
  as well as the determinant?
  
  I've taken a look at code online and from what I've seen, most of them assume the variance is constant.
  
  ---
  
  @NielSlater
  
  Using the reparameterization trick we would use a normal distribution with fixed parameters 0 and 1.
  $$ a\_t \sim \mu\_\theta(s\_t) + \Sigma\_\theta(s\_t) \* Normal(0, 1) $$
  
  Which is the same as if we *had* actually sampled directly from a distribution, $ \pi\_\theta(a\_t | s\_t) = Normal(\mu\_\theta(s\_t), \Sigma\_\theta(s\_t))$ and let's us calculate the corresponding $\log \pi\_\theta(a\_t|s\_t)$ and $\nabla\_\theta \log \pi\_\theta(a\_t | s\_t)$ without having to differentiate through the actual density.
  
  **Tags:** reinforcement-learning, policy-gradients
  > **Comments:**
  > **username_1**: In your formulas, what is the variance? I understand that $\mu_\theta$ is the mean with respect to the current parameters $\theta$. $\Sigma$ is the covariance matrix, but of which random variables? I think it may be helpful to specify these details, so that readers can better understand your question. In general, I think you should try to explain the formulas that you provide, if possible.  
  > **username_0**: Hi, the covariance is parameterized by $\theta$ in the second equation. In the fourth equation I leave out the $\theta$ because in the slide is says in the case where the covariance is constant.  
  > **username_1**: But is this the covariance of which variables? Maybe it is the covariance of the state and action (assuming they are considered random variables in this case), even though you denote it just as $\Sigma_\theta(s_t)$ instead of maybe $\Sigma_\theta(s_t, a_t)$. Or maybe $\Sigma_\theta(s_t)$ just represents the variance of $s_t$? Honestly, I'm not sure what that means, anyway.  
  > **username_0**: @nbro I'm not sure I understand. It is the mean and covariance that models a distribution over actions which I have assumed to be gaussian. In reinforcement learning this is quite a common thing to do I believe. Each of those is estimated from the same neural network that is parameterized by $\theta$.  
  > **username_2**: This part of RL maths is something I am not yet familiar with. However, I believe the answer will be based around the ["the reparameterization trick"](https://medium.com/@llionj/the-reparameterization-trick-4ff30fe92954) . . . if no answer is given and you can figure out the link to the equations you have given, actually I'd be interested to see this Normal distribution with $\mu$ and $\sigma$ outputs from the NN worked through with an example . . .  
  > **username_0**: @NeilSlater I've included a short edit of how (I think) it would work with the reparameterization trick. Though I'd heard of it used in VAEs, I'd honestly never heard it mentioned in the context of RL and going through a lot of code I'm surprised to see that the majority of modelling is done with constant variance. Seemed strange!  
  > **username_2**: Both VAEs and RL policy gradients with stochastic policies use the trick. My understanding is that you cannot propagate gradient through a sampling node otherwise. And the initial error signal in RL relates to the actual action chosen, not the values of the NN output.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2019-02-15. Score: 2
  
  It's true that computing the log prob of a sample from a gaussian requires inverting a matrix and dealing with the determinant in the general case of a full covariance matrix. If you wanted to backprop through the log prob, you'd need to backprop through these operations, which by the way are available as differentiable operations in pytorch. See the gaussian distribution class which deals with these operations: <https://github.com/pytorch/pytorch/blob/master/torch/distributions/multivariate_normal.py#L177>
  You can see there's a `half_log_det` and although you won't find an explicit matrix inversion, you'll see that it uses a differentiable triangular linear solve (`torch.trtrs`) at some point, which is essentially doing a matrix inversion in a more efficient way.
  
  
  ---
  
  