Answers.Body: <p>The VAE architecture from the cs231n class is just a more general version of the code Keras provides, in which the covariance matrix is <span class="math-container">$\mathbf 0$</span>. You can see this from the reparametrization trick
  <span class="math-container">$$
  \begin{align}
  x&amp;=\mu+\Sigma\epsilon\\
  &amp;=\mu&amp;\mathrm{if}\ \Sigma=0
  \end{align}
  $$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-31T11:04:43.933Z
Answers.LastActivityDate: 2019-03-31T11:04:43.933Z
Answers.UserDisplayName: Maybe
Body: <p>I'm working on understanding VAEs, mostly through video lectures of Stanford cs231n, in particular <a href="https://youtu.be/5WoItGTWV54?t=2543" rel="nofollow noreferrer">lecture 13</a> tackles on this topic and I think I have a good theoretical grasp.</p>
  
  <p>However, when looking at actual code of implementations, such as <a href="https://github.com/keras-team/keras/blob/master/examples/variational_autoencoder.py" rel="nofollow noreferrer">this code</a>  from <a href="https://blog.keras.io/building-autoencoders-in-keras.html" rel="nofollow noreferrer">this blog</a> of VAEs I see some differences which I can't quite understand.</p>
  
  <p>Please take a look at this VAE <a href="https://i.sstatic.net/LUcel.jpg" rel="nofollow noreferrer">architecture visualization</a> from the class, specifically the decoder part. From the way it is presented here I understand that the decoder network outputs mean and covariance for the data <em>distribution</em>. To get an actual output (i.e. image) we need to sample from the distribution that is parametrized by mean and covariance - the outputs of the decoder.</p>
  
  <p>Now if you look at the code from the Keras blog VAE implementation, you will see that there is no such thing. A decoder takes in a sample from latent space and <strong>directly</strong> maps its input (sampled z) to an output (e.g. image), not to parameters of a distribution from which an output is to be sampled. </p>
  
  <p>Am I missing something or does this implementation not correspond to the one presented in the lecture? I've been trying to make sense of it for quite some time now but still can't seem to understand the discrepancy. </p>
  
Tags: <computer-vision><generative-model><autoencoders><latent-variable>
Title: Do we also need to model a probability distribution for the decoder of a VAE?
UserDisplayName: ytolochko
ThreadText: # Do we also need to model a probability distribution for the decoder of a VAE?
  
  (Asked by: username_0 on 2019-03-29)
  
  I'm working on understanding VAEs, mostly through video lectures of Stanford cs231n, in particular [lecture 13](https://youtu.be/5WoItGTWV54?t=2543) tackles on this topic and I think I have a good theoretical grasp.
  
  However, when looking at actual code of implementations, such as [this code](https://github.com/keras-team/keras/blob/master/examples/variational_autoencoder.py) from [this blog](https://blog.keras.io/building-autoencoders-in-keras.html) of VAEs I see some differences which I can't quite understand.
  
  Please take a look at this VAE [architecture visualization](https://i.sstatic.net/LUcel.jpg) from the class, specifically the decoder part. From the way it is presented here I understand that the decoder network outputs mean and covariance for the data *distribution*. To get an actual output (i.e. image) we need to sample from the distribution that is parametrized by mean and covariance - the outputs of the decoder.
  
  Now if you look at the code from the Keras blog VAE implementation, you will see that there is no such thing. A decoder takes in a sample from latent space and **directly** maps its input (sampled z) to an output (e.g. image), not to parameters of a distribution from which an output is to be sampled.
  
  Am I missing something or does this implementation not correspond to the one presented in the lecture? I've been trying to make sense of it for quite some time now but still can't seem to understand the discrepancy.
  
  **Tags:** computer-vision, generative-model, autoencoders, latent-variable
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-03-31. Score: 0
  
  The VAE architecture from the cs231n class is just a more general version of the code Keras provides, in which the covariance matrix is $\mathbf 0$. You can see this from the reparametrization trick
  $$
  \begin{align}
  x&=\mu+\Sigma\epsilon\\
  &=\mu&\mathrm{if}\ \Sigma=0
  \end{align}
  $$
  
  
  ---
  
  