Hey Nikan!
 Yeah I realized that from your last name!
Sorry if I’m restating my question ( I modified it a couple of times at the top lol) and probably repeating your answer, I’m kinda new to the KL implementation in pytorch, so I really appreciate your help.
My question is :
I have two VAEs, each with a mu and standard deviation layers. Now I would like to calculate the KL between these two VAEs using either the mu and sds, or the sampling layers. This sounds to me like a multivariate gaussian KL divergence problem, so I looked at the formula and I noticed that I actually need the covariance matrix of q (if we assume that KL(p||q) ). And as far as I know there is no way to calculate the covariance in this case, am I correct?
@zhl515 suggested to directly calculate the KL between the two sampling layers as you said before using the bellow function, but as you said this requires probabilities which I do not have, so I think the following function cannot be used on the sampling layers, is that right? :
 torch.nn.functional.kl_div(p, q).

And based on your last comment, you are suggesting to register the distribution, and then use
torch.distributions.kl.kl_divergence(p, q)

The only problem is that in order to register the distribution I need to have the covariance matrix, and I can’t obtain that because I only have mu and std. It makes me think whether that is a possible thing to do at all in neural network. In the VAE paper they don’t have that problem because the are assuming that q has mean 0 and covariance I

  
      stats.stackexchange.com
  
  
      
    
  

  Deriving the KL divergence loss for VAEs 75



  kullback-leibler, autoencoders, variational-bayes



  asked by
  
  
    YellowPillow
  
  on 01:55AM - 14 Dec 17 UTC


  
  
    
    
  
  


Thank you!