Answers.Body: <p>From my experience (on MNIST digits), even when using a latent space of only <span class="math-container">$10$</span> nodes, the decoded reconstruction was pretty much ok. perhaps the architecture itself lacks the capabilities of encoding/decoding properly. Check out <a href="https://medium.com/dataseries/convolutional-autoencoder-in-pytorch-on-mnist-dataset-d65145c132ac" rel="nofollow noreferrer">this</a> summary and see if you can improve your results using a similar approach.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-04T11:32:38.157Z
Answers.LastActivityDate: 2022-10-04T11:32:38.157Z
Answers.UserDisplayName: Hadar Sharvit
Answers.Body: <p>You are asking about several things here and while related, solving one, will not necessarily &quot;solve&quot; your problem. Let's look at them separately:</p>
  <ol>
  <li>Optimal dimension of the latent space.</li>
  <li>Blurry reconstructions.</li>
  <li>Optimal sample size.</li>
  </ol>
  <p><strong>Optimal dimension of the latent space.</strong></p>
  <p>I'm unaware of a one-fit-all way to find the optimal dimensionality of <span class="math-container">$z$</span> but an easy way is to try with different values and look at the likelihood on the test-set <span class="math-container">$log(p)$</span> - pick the lowest dimensionality that maximises it. This is a solution in tune with Deep Learning spirit :)</p>
  <p>Second solution, maybe a little more grounded, is to decompose your training data with SVD and look at the spectrum of singular values. The number of non trivial (=above some small threshold) values will give you a rough idea of the number of latent dimensions you are going to need.</p>
  <p>Finally, you could allow for a lot of z-dimensions but augment your loss function in such way, that the encoder will be forced to only use what it needs. This is sometimes called Sparsity promoting, L1 or Lasso-type regularisation and is also something that can help with overfitting. Take a look at <code>arXiv:1812.07238</code>.</p>
  <p><strong>Blurry reconstructions</strong></p>
  <p>This is a notorious problem with VAE's and while there are a lot of theories on why this happens, my take is that the reason is two fold.</p>
  <p>First, the loss function. With typical Cross-Entropy or MSE losses, we have this blunt bottom of the function where the minimum is residing allowing for a lot of similar &quot;good solutions&quot;. See <code>arXiv:1511.05440</code> and especially <code>https://openreview.net/forum?id=rkglvsC9Ym</code> for an easy fix that seems to improve the quality/sharpness of the reconstructions.</p>
  <p>Second, the blurriness comes from the Variational formulation itself. Handwavy explained, we are trying to model some very-very complex data (images in your case), with a &quot;simple&quot; isotropic Gaussian. Not surprisingly, the result will be something the best the model can do given this constraint. Recall that the loss function in VAE's is called ELBO - Evidence Lower Bound - which basically tells us that we are trying to model a Lower Bound as best as we can and not the &quot;actual data&quot; distribution. Typically, introduction of a KL-multiplier, <span class="math-container">$\beta$</span>, which relaxes the influence of the Gaussian prior, will give you better reconstructions (see <span class="math-container">$\beta$</span>-VAE's).</p>
  <p>Finally, if you are feeling especially adventurous, take a look at discrete VAE's (VQ-VAE's), which seem to have reconstructions on pair with GAN's. Sampling from them is not trivial, however.</p>
  <p><strong>Optimal sample size</strong></p>
  <p>As for optimal sample size, just choose an architecture that will not overfit. Decrease the number of neurons/layers, check your <span class="math-container">$\log(p)$</span> on the test-set, introduce Dropout, all the usual stuff.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-08T08:17:37.790Z
Answers.Text: Thank you for summarizing and troubleshooting my problem! I was indeed asking about quite a few things because I did not know what is causing the problem. Currently trying your suggestions!
Answers.UserDisplayName: Qingyi Wang
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-04T13:51:24.217Z
Answers.LastActivityDate: 2022-10-04T13:51:24.217Z
Answers.UserDisplayName: Евген
Body: <p>I am training an autoencoder and a variational autoencoder using satellite and streetview images. I have tested my program on standard datasets such as MNIST and CelebA. It seems that the latent space dimension needed for those applications are fairly small. For example, MNIST is 28x28x1 and CelebA is 64x64x3 and for both a latent space bottleneck of 50 would be sufficient to observe reasonably reconstructed image.</p>
  <p>But for the autoencoder I am constructing, I needed a dimension of ~20000 in order to see features. The variational autoencoder is not working, and I only see a few blobs of fuzzy color.</p>
  <p>For those who have experience with training the autoencoders with your own images, what could be the problem? Are certain features easier to compress than others? or does it look like a sample size problem? (i have 20000 images in training) Is there any rule of thumb for the the factor of compression? Thanks!</p>
  <p>Here is a best example of what I have got with my VAE. I am using a ResNeXt architecture and the image dimension is 64x64x3, the latent space dimension is very large (18432).</p>
  <p><a href="https://i.sstatic.net/fuc1E.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fuc1E.png" alt="original image (64x64x3)" /></a>
  <a href="https://i.sstatic.net/EIeMW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/EIeMW.png" alt="VAE reconstructed image (with latent space of 18432)" /></a></p>
  
Tags: <computer-vision><autoencoders><variational-autoencoder>
Title: What is an appropriate size for a latent space of (variational) autoencoders and how it varies with the features of the images?
UserDisplayName: Qingyi Wang
ThreadText: # What is an appropriate size for a latent space of (variational) autoencoders and how it varies with the features of the images?
  
  (Asked by: username_0 on 2022-10-04)
  
  I am training an autoencoder and a variational autoencoder using satellite and streetview images. I have tested my program on standard datasets such as MNIST and CelebA. It seems that the latent space dimension needed for those applications are fairly small. For example, MNIST is 28x28x1 and CelebA is 64x64x3 and for both a latent space bottleneck of 50 would be sufficient to observe reasonably reconstructed image.
  
  But for the autoencoder I am constructing, I needed a dimension of ~20000 in order to see features. The variational autoencoder is not working, and I only see a few blobs of fuzzy color.
  
  For those who have experience with training the autoencoders with your own images, what could be the problem? Are certain features easier to compress than others? or does it look like a sample size problem? (i have 20000 images in training) Is there any rule of thumb for the the factor of compression? Thanks!
  
  Here is a best example of what I have got with my VAE. I am using a ResNeXt architecture and the image dimension is 64x64x3, the latent space dimension is very large (18432).
  
  [![original image (64x64x3)](https://i.sstatic.net/fuc1E.png)](https://i.sstatic.net/fuc1E.png)
  [![VAE reconstructed image (with latent space of 18432)](https://i.sstatic.net/EIeMW.png)](https://i.sstatic.net/EIeMW.png)
  
  **Tags:** computer-vision, autoencoders, variational-autoencoder
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-10-04. Score: 9 | Accepted answer
  
  You are asking about several things here and while related, solving one, will not necessarily "solve" your problem. Let's look at them separately:
  
  1. Optimal dimension of the latent space.
  2. Blurry reconstructions.
  3. Optimal sample size.
  
  **Optimal dimension of the latent space.**
  
  I'm unaware of a one-fit-all way to find the optimal dimensionality of $z$ but an easy way is to try with different values and look at the likelihood on the test-set $log(p)$ - pick the lowest dimensionality that maximises it. This is a solution in tune with Deep Learning spirit :)
  
  Second solution, maybe a little more grounded, is to decompose your training data with SVD and look at the spectrum of singular values. The number of non trivial (=above some small threshold) values will give you a rough idea of the number of latent dimensions you are going to need.
  
  Finally, you could allow for a lot of z-dimensions but augment your loss function in such way, that the encoder will be forced to only use what it needs. This is sometimes called Sparsity promoting, L1 or Lasso-type regularisation and is also something that can help with overfitting. Take a look at `arXiv:1812.07238`.
  
  **Blurry reconstructions**
  
  This is a notorious problem with VAE's and while there are a lot of theories on why this happens, my take is that the reason is two fold.
  
  First, the loss function. With typical Cross-Entropy or MSE losses, we have this blunt bottom of the function where the minimum is residing allowing for a lot of similar "good solutions". See `arXiv:1511.05440` and especially `https://openreview.net/forum?id=rkglvsC9Ym` for an easy fix that seems to improve the quality/sharpness of the reconstructions.
  
  Second, the blurriness comes from the Variational formulation itself. Handwavy explained, we are trying to model some very-very complex data (images in your case), with a "simple" isotropic Gaussian. Not surprisingly, the result will be something the best the model can do given this constraint. Recall that the loss function in VAE's is called ELBO - Evidence Lower Bound - which basically tells us that we are trying to model a Lower Bound as best as we can and not the "actual data" distribution. Typically, introduction of a KL-multiplier, $\beta$, which relaxes the influence of the Gaussian prior, will give you better reconstructions (see $\beta$-VAE's).
  
  Finally, if you are feeling especially adventurous, take a look at discrete VAE's (VQ-VAE's), which seem to have reconstructions on pair with GAN's. Sampling from them is not trivial, however.
  
  **Optimal sample size**
  
  As for optimal sample size, just choose an architecture that will not overfit. Decrease the number of neurons/layers, check your $\log(p)$ on the test-set, introduce Dropout, all the usual stuff.
  
  > **Comments:**
  > **username_0**: Thank you for summarizing and troubleshooting my problem! I was indeed asking about quite a few things because I did not know what is causing the problem. Currently trying your suggestions!  
  
  ---
  
  ### Answer by username_1 on 2022-10-04. Score: 0
  
  From my experience (on MNIST digits), even when using a latent space of only $10$ nodes, the decoded reconstruction was pretty much ok. perhaps the architecture itself lacks the capabilities of encoding/decoding properly. Check out [this](https://medium.com/dataseries/convolutional-autoencoder-in-pytorch-on-mnist-dataset-d65145c132ac) summary and see if you can improve your results using a similar approach.
  
  
  ---
  
  