Answers.Body: <p>The VAE's encoder is usually implemented to produce the mean and variance. These are vectors, which can also be 1-dimensional (equivalently, scalars). If that's the case, then the latent vector is also 1-dimensional (if I understand your question correctly). See <a href="https://github.com/pytorch/examples/blob/main/vae/main.py#L44" rel="nofollow noreferrer">this implementation</a>, where the size of the mean and variance vector is <span class="math-container">$20$</span>, so the size of <span class="math-container">$z$</span> is also <span class="math-container">$20$</span>, but you can change this to <span class="math-container">$1$</span>.</p>
  <p>(Actually, the (co)variance may not be a vector but a matrix, but I assume a diagonal covariance, like in the linked implementation and the VAE paper).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-26T20:42:30.253Z
Answers.LastActivityDate: 2022-06-26T20:47:35.737Z
Answers.LastEditDate: 2022-06-26T20:47:35.737Z
Answers.UserDisplayName: nbro
Answers.Body: <p>The idea of a VAE is that the code/feature layers produce a probability distribution of features (Gaussian to be exact), by predicting its mean and variance.</p>
  <p>This is the point that requires sampling, since probability distributions are abstract concepts, and sampling is the only way to get numbers from them.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-26T20:58:58.963Z
Answers.LastActivityDate: 2022-06-26T20:58:58.963Z
Answers.UserDisplayName: Dr. Snoopy
Body: <p>While reading the book, Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems, I read that VAEs using a sampling technique to obtain values from the coding layer. However, the output of a neural network activation is just a single value. In that case, what values does the VAE sample?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-26T20:39:54.253Z
Comments.Text: What do you mean by "output of a neural network activation is just a single value"? Which activation are you talking about exactly?
Comments.UserDisplayName: nbro
Tags: <variational-autoencoder>
Title: Why does the coding layer in a VAE have a range of values?
UserDisplayName: desert_ranger
ThreadText: # Why does the coding layer in a VAE have a range of values?
  
  (Asked by: username_0 on 2022-06-26)
  
  While reading the book, Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems, I read that VAEs using a sampling technique to obtain values from the coding layer. However, the output of a neural network activation is just a single value. In that case, what values does the VAE sample?
  
  **Tags:** variational-autoencoder
  
  > **Comments:**
  > **username_1**: What do you mean by "output of a neural network activation is just a single value"? Which activation are you talking about exactly?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-06-26. Score: 1 | Accepted answer
  
  The VAE's encoder is usually implemented to produce the mean and variance. These are vectors, which can also be 1-dimensional (equivalently, scalars). If that's the case, then the latent vector is also 1-dimensional (if I understand your question correctly). See [this implementation](https://github.com/pytorch/examples/blob/main/vae/main.py#L44), where the size of the mean and variance vector is $20$, so the size of $z$ is also $20$, but you can change this to $1$.
  
  (Actually, the (co)variance may not be a vector but a matrix, but I assume a diagonal covariance, like in the linked implementation and the VAE paper).
  
  
  ---
  
  ### Answer by username_2 on 2022-06-26. Score: 2
  
  The idea of a VAE is that the code/feature layers produce a probability distribution of features (Gaussian to be exact), by predicting its mean and variance.
  
  This is the point that requires sampling, since probability distributions are abstract concepts, and sampling is the only way to get numbers from them.
  
  
  ---
  
  