Answers.Body: <p>Old question but hopefully OP is still active.</p>
  <p>I also agree with recessive that perhaps it is struggling to create a continuous representation of your data in the latent space. Have you looked at your KL loss and your reconstruction loss independantly?</p>
  <p>I was wondering: have you tried changing your input? If you reduce it from 2048 dimensions to 1024, does the dip in your curve also shift proportionally?</p>
  <p>Have you also looked at what the latent space and the reconstructions looks like? what sort of errors is it making?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-19T16:10:26.100Z
Answers.Text: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-28T09:49:00.493Z
Answers.Text: It's been some time now so I don't know what I did of the code so I won't be able to test your suggestions. However, if I remember correctly, I solved the problem by using tanh activation functions instead of relu that I was using originally. I still have no idea what the origin of this behavior is or why changing the activation function fixes it tho.
Answers.UserDisplayName: tyassine
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-19T13:07:59.410Z
Answers.LastActivityDate: 2022-07-19T13:55:52.040Z
Answers.LastEditDate: 2022-07-19T13:55:52.040Z
Answers.UserDisplayName: Whitehot
Body: <p>I'm training a VAE to reconstruct some input (channels picked up by some MIMO BS for context) and I ran an experiment on the training set to see how the performance improves with the latent space dimension.</p>
  <p>My VAE structure is as follows : Input : 2048 -&gt; 1024 -&gt; 512 -&gt; Latent space dimension -&gt; 512 -&gt; 1024 -&gt; Output : 2048</p>
  <p>Here is what I get in terms of relative error when the latent space dimension goes from 2 to 100 :
  <a href="https://i.sstatic.net/hnG3X.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hnG3X.png" alt="enter image description here" /></a></p>
  <p>Everything works as expected at the beginning, but the error starts rising up at around 50 and I have no idea why. With a large latent space dimension, the output is orders of magnitude smaller than the input, which explains the relative error of value 1.</p>
  <p>Here is the same figure when I run the exact same experiment but with a normal autoencoder this time.<a href="https://i.sstatic.net/390qh.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/390qh.png" alt="enter image description here" /></a></p>
  <p>This time the results are consistent.</p>
  <p>What's wrong with my VAE ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-12T04:54:14.137Z
Comments.Text: This is a really interesting phenomena, my guess would be the network is unable to learn a continuous representation in the number of epochs you provided for learning. Try upping the number of epochs in the VAE for learning and see if the error drops.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-12T07:20:33.820Z
Comments.Text: @Recessive I already tried that. Went from 50 epochs to 100. Nothing changed, had the same exact behavior.
Comments.UserDisplayName: tyassine
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-18T18:32:39.543Z
Comments.Text: The question of how the latent space size influences the performance is answered in [this](https://stats.stackexchange.com/questions/450229/how-does-the-bottleneck-z-dimension-affect-the-reconstruction-loss-in-vaes) & [this](https://stats.stackexchange.com/questions/327966/variational-autoencoder-%e2%88%92-dimension-of-the-latent-space?rq=1) Cross Validated posts. In short: the amount of training data upper bounds the latent dimension, and a too large latent space leads to overfitting. This however seems opposite to what you report, maybe your model becomes too large and is thus underfitting?
Comments.UserDisplayName: Mariusmarten
Tags: <autoencoders><variational-autoencoder>
Title: VAE giving near zero output when latent space dimension is large
UserDisplayName: tyassine
ThreadText: # VAE giving near zero output when latent space dimension is large
  
  (Asked by: username_0 on 2021-01-11)
  
  I'm training a VAE to reconstruct some input (channels picked up by some MIMO BS for context) and I ran an experiment on the training set to see how the performance improves with the latent space dimension.
  
  My VAE structure is as follows : Input : 2048 -> 1024 -> 512 -> Latent space dimension -> 512 -> 1024 -> Output : 2048
  
  Here is what I get in terms of relative error when the latent space dimension goes from 2 to 100 :
  [![enter image description here](https://i.sstatic.net/hnG3X.png)](https://i.sstatic.net/hnG3X.png)
  
  Everything works as expected at the beginning, but the error starts rising up at around 50 and I have no idea why. With a large latent space dimension, the output is orders of magnitude smaller than the input, which explains the relative error of value 1.
  
  Here is the same figure when I run the exact same experiment but with a normal autoencoder this time.[![enter image description here](https://i.sstatic.net/390qh.png)](https://i.sstatic.net/390qh.png)
  
  This time the results are consistent.
  
  What's wrong with my VAE ?
  
  **Tags:** autoencoders, variational-autoencoder
  
  > **Comments:**
  > **username_1**: This is a really interesting phenomena, my guess would be the network is unable to learn a continuous representation in the number of epochs you provided for learning. Try upping the number of epochs in the VAE for learning and see if the error drops.  
  > **username_0**: @username_1 I already tried that. Went from 50 epochs to 100. Nothing changed, had the same exact behavior.  
  > **username_2**: The question of how the latent space size influences the performance is answered in [this](https://stats.stackexchange.com/questions/450229/how-does-the-bottleneck-z-dimension-affect-the-reconstruction-loss-in-vaes) & [this](https://stats.stackexchange.com/questions/327966/variational-autoencoder-%e2%88%92-dimension-of-the-latent-space?rq=1) Cross Validated posts. In short: the amount of training data upper bounds the latent dimension, and a too large latent space leads to overfitting. This however seems opposite to what you report, maybe your model becomes too large and is thus underfitting?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2022-07-19. Score: 0
  
  Old question but hopefully OP is still active.
  
  I also agree with username_1 that perhaps it is struggling to create a continuous representation of your data in the latent space. Have you looked at your KL loss and your reconstruction loss independantly?
  
  I was wondering: have you tried changing your input? If you reduce it from 2048 dimensions to 1024, does the dip in your curve also shift proportionally?
  
  Have you also looked at what the latent space and the reconstructions looks like? what sort of errors is it making?
  
  > **Comments:**
  > **username_4**: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  > **username_0**: It's been some time now so I don't know what I did of the code so I won't be able to test your suggestions. However, if I remember correctly, I solved the problem by using tanh activation functions instead of relu that I was using originally. I still have no idea what the origin of this behavior is or why changing the activation function fixes it tho.  
  
  ---
  
  