Answers.Body: <p>The number of features <span class="math-container">$n$</span> of an input <span class="math-container">$\mathbf{x} \in \mathbb{R}^n$</span> to the VAE may be related to the dimension <span class="math-container">$m$</span> of the latent vector <span class="math-container">$\mathbf{z} \in \mathbb{R}^m \sim \mathcal{N}(\mu, \sigma^2)$</span>, but the formulation of the VAE does not go far enough to tell us how these parameters are related. The only assumption in the VAE is that <span class="math-container">$\mathbf{x}$</span> depends on <span class="math-container">$\mathbf{z}$</span>. Here's the diagram (from the <a href="https://arxiv.org/pdf/1312.6114.pdf" rel="nofollow noreferrer">VAE paper</a>) that illustrates the graphical model they consider</p>
  <p><a href="https://i.sstatic.net/Mw62M.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Mw62M.png" alt="enter image description here" /></a></p>
  <p>In practice, we may not use a univariate Gaussian, but most likely we use a multivariate Gaussian, i.e. <span class="math-container">$m &gt; 1$</span>, but <span class="math-container">$m$</span> is really a hyper-parameter, so it could also be <span class="math-container">$1$</span>. Moreover, note that we don't sample <span class="math-container">$\mathbf{z}$</span> directly from <span class="math-container">$\mathcal{N}$</span>, if we use the re-parametrization trick. See <a href="https://github.com/pytorch/examples/blob/main/vae/main.py" rel="nofollow noreferrer">this implementation</a>, where <span class="math-container">$\epsilon$</span> is actually sampled. <span class="math-container">$\epsilon$</span> is just a sample from a standard normal (usually), which we use (along with the learnable mean and variance) to construct <span class="math-container">$\mathbf{z}$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-26T20:31:22.627Z
Answers.LastActivityDate: 2022-06-26T20:38:38.697Z
Answers.LastEditDate: 2022-06-26T20:38:38.697Z
Answers.UserDisplayName: nbro
Body: <p>I am reading the book, Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems and came across the following paragraph -</p>
  <blockquote>
  <p>You can recognize the basic structure of all autoencoders, with an encoder followed by a decoder (in this example, they both have two hidden layers), but there is a twist: instead of directly producing a coding for a given input, the encoder produces a mean coding μ and a standard deviation σ. The actual coding is then sampled randomly from a Gaussian distribution with mean μ and standard deviation σ. After that the decoder decodes the sampled coding normally.</p>
  </blockquote>
  <p>I do not understand how sampling is conducted over here and have the following question -</p>
  <p>I understand it is trivial to sample from a univariate Gaussian distribution. However, for a dataset with n features, we won't be able to use a univariate Gaussian. How does that work?</p>
  
Tags: <variational-autoencoder>
Title: How does a VAE sample the coding layer?
UserDisplayName: desert_ranger
ThreadText: # How does a VAE sample the coding layer?
  
  (Asked by: username_0 on 2022-06-26)
  
  I am reading the book, Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems and came across the following paragraph -
  
  > You can recognize the basic structure of all autoencoders, with an encoder followed by a decoder (in this example, they both have two hidden layers), but there is a twist: instead of directly producing a coding for a given input, the encoder produces a mean coding μ and a standard deviation σ. The actual coding is then sampled randomly from a Gaussian distribution with mean μ and standard deviation σ. After that the decoder decodes the sampled coding normally.
  
  I do not understand how sampling is conducted over here and have the following question -
  
  I understand it is trivial to sample from a univariate Gaussian distribution. However, for a dataset with n features, we won't be able to use a univariate Gaussian. How does that work?
  
  **Tags:** variational-autoencoder
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-06-26. Score: 1 | Accepted answer
  
  The number of features $n$ of an input $\mathbf{x} \in \mathbb{R}^n$ to the VAE may be related to the dimension $m$ of the latent vector $\mathbf{z} \in \mathbb{R}^m \sim \mathcal{N}(\mu, \sigma^2)$, but the formulation of the VAE does not go far enough to tell us how these parameters are related. The only assumption in the VAE is that $\mathbf{x}$ depends on $\mathbf{z}$. Here's the diagram (from the [VAE paper](https://arxiv.org/pdf/1312.6114.pdf)) that illustrates the graphical model they consider
  
  [![enter image description here](https://i.sstatic.net/Mw62M.png)](https://i.sstatic.net/Mw62M.png)
  
  In practice, we may not use a univariate Gaussian, but most likely we use a multivariate Gaussian, i.e. $m > 1$, but $m$ is really a hyper-parameter, so it could also be $1$. Moreover, note that we don't sample $\mathbf{z}$ directly from $\mathcal{N}$, if we use the re-parametrization trick. See [this implementation](https://github.com/pytorch/examples/blob/main/vae/main.py), where $\epsilon$ is actually sampled. $\epsilon$ is just a sample from a standard normal (usually), which we use (along with the learnable mean and variance) to construct $\mathbf{z}$.
  
  
  ---
  
  