Answers.Body: <p>@Brale_ 's answer is correct, it is common practice in a multitude of models to learn a representation of an independent multivariate normal, but don't let that stop you from pushing the envelope for your needs.   </p>
  
  <p>You can actually learn a dependent form as well. Normally the independent form is done by learning the means and standard deviations, because sampling a standard normal can achieve your draw by <span class="math-container">$z \sim N(\mu, Diag(\sigma^2))$</span> by <span class="math-container">$z = \mu + \sigma \epsilon$</span> where <span class="math-container">$\epsilon$</span> is drawn from a unit normal.   </p>
  
  <p>But you can actually achieve a similar trick for a generalized multivariate normal distribution: lets assume your trying to learn <span class="math-container">$N(\mu, \Sigma)$</span> where <span class="math-container">$\Sigma$</span> is the covariance matrix. So what you would do is learn the <a href="https://en.wikipedia.org/wiki/Cholesky_decomposition" rel="nofollow noreferrer">Cholesky decomposition</a> because where <span class="math-container">$\Sigma = AA^T$</span> you can now draw it through the parametrization trick: <span class="math-container">$z = \mu + A\epsilon$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-14T16:29:48.370Z
Answers.LastActivityDate: 2019-07-14T16:29:48.370Z
Answers.UserDisplayName: mshlis
Answers.Body: <p>@mshlis, if <span class="math-container">$\sigma$</span> is covariance matrix, then exist <span class="math-container">$A$</span> s.t. <span class="math-container">$\sigma=AA^T$</span>. </p>
  
  <p>But if we generate <span class="math-container">$A$</span> by model, and calculate <span class="math-container">$\sigma = AA^T$</span>, then Cholesky decomposition will not successful</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-11T08:13:00.690Z
Answers.LastActivityDate: 2019-12-11T08:13:00.690Z
Answers.UserDisplayName: C.M. Cai
Answers.Body: <p>Usually it is assumed that there is no correlation between different actions, so the covariance matrix will be zero everywhere except on the main diagonal. Diagonal will represent variances of actions. Diagonal covariance matrix will be positive semidefinite if all values on diagonal are <span class="math-container">$\geq$</span> 0 so you need to insure that output of final layer is <span class="math-container">$\geq$</span> 0, which can be done with ReLU activation for example.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-14T10:30:16.507Z
Answers.Text: Now I have the problem that the matrix becomes singular, i.e. the network learns to set the diagonal zero. Any idea how to fix this?
Answers.UserDisplayName: user9007131
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-14T11:10:40.237Z
Answers.Text: diagonal is zero probably because the output of network is < 0 so ReLU sets it to zero. Only one of the elements needs to be zero for diagonal matrix to be singular. You can try to evade this by adding a small constant to the diagonal like 1e-6
Answers.UserDisplayName: Brale
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-14T09:24:05.260Z
Answers.LastActivityDate: 2019-07-14T09:24:05.260Z
Answers.UserDisplayName: Brale
Body: <p>I have a continuous state space, and a continuous action space. The way I understand it, I can build a policy network which takes as input a continuous state vector and outputs both mean vector and covariance matrix of the action-distribution. To get a valid action I then sample from that distribution.</p>
  
  <p>However, when trying to implement such a network, I get the error message that the parts of my output layer which I want to be the covariance matrix are singular/not positive-semi-definite. How can I fix this? I tried different activation-functions and initializations for the last layer, but once in a while I run into the same problem again. </p>
  
  <p>How can I enforce that my network outputs a valid covariance matrix?</p>
  
Tags: <machine-learning><reinforcement-learning><policy-gradients>
Title: How to enforce covariance-matrix output as part of the last layer of a Policy Network?
UserDisplayName: user9007131
ThreadText: # How to enforce covariance-matrix output as part of the last layer of a Policy Network?
  
  (Asked by: username_0 on 2019-07-14)
  
  I have a continuous state space, and a continuous action space. The way I understand it, I can build a policy network which takes as input a continuous state vector and outputs both mean vector and covariance matrix of the action-distribution. To get a valid action I then sample from that distribution.
  
  However, when trying to implement such a network, I get the error message that the parts of my output layer which I want to be the covariance matrix are singular/not positive-semi-definite. How can I fix this? I tried different activation-functions and initializations for the last layer, but once in a while I run into the same problem again.
  
  How can I enforce that my network outputs a valid covariance matrix?
  
  **Tags:** machine-learning, reinforcement-learning, policy-gradients
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2019-07-14. Score: 3 | Accepted answer
  
  Usually it is assumed that there is no correlation between different actions, so the covariance matrix will be zero everywhere except on the main diagonal. Diagonal will represent variances of actions. Diagonal covariance matrix will be positive semidefinite if all values on diagonal are $\geq$ 0 so you need to insure that output of final layer is $\geq$ 0, which can be done with ReLU activation for example.
  
  > **Comments:**
  > **username_0**: Now I have the problem that the matrix becomes singular, i.e. the network learns to set the diagonal zero. Any idea how to fix this?  
  > **username_3**: diagonal is zero probably because the output of network is < 0 so ReLU sets it to zero. Only one of the elements needs to be zero for diagonal matrix to be singular. You can try to evade this by adding a small constant to the diagonal like 1e-6  
  
  ---
  
  ### Answer by username_1 on 2019-07-14. Score: 1
  
  @Brale\_ 's answer is correct, it is common practice in a multitude of models to learn a representation of an independent multivariate normal, but don't let that stop you from pushing the envelope for your needs.
  
  You can actually learn a dependent form as well. Normally the independent form is done by learning the means and standard deviations, because sampling a standard normal can achieve your draw by $z \sim N(\mu, Diag(\sigma^2))$ by $z = \mu + \sigma \epsilon$ where $\epsilon$ is drawn from a unit normal.
  
  But you can actually achieve a similar trick for a generalized multivariate normal distribution: lets assume your trying to learn $N(\mu, \Sigma)$ where $\Sigma$ is the covariance matrix. So what you would do is learn the [Cholesky decomposition](https://en.wikipedia.org/wiki/Cholesky_decomposition) because where $\Sigma = AA^T$ you can now draw it through the parametrization trick: $z = \mu + A\epsilon$.
  
  
  ---
  
  ### Answer by username_2 on 2019-12-11. Score: 0
  
  @mshlis, if $\sigma$ is covariance matrix, then exist $A$ s.t. $\sigma=AA^T$.
  
  But if we generate $A$ by model, and calculate $\sigma = AA^T$, then Cholesky decomposition will not successful
  
  
  ---
  
  