Answers.Body: <p>I am not sure about the VAE in particular, but the convnext presented <a href="https://arxiv.org/pdf/2201.03545.pdf" rel="nofollow noreferrer">here</a> uses the &quot;inverse bottleneck&quot; (i.e. internal representations being higher dimensional than inputs) as one of the core changes that leads to an increase in its performance when compared to prior convolutional networks. This and some other changes make the convnet be more competitive against the visual transformer (that is pretrained in a large dataset).</p>
  <p>But for really low-dimensional data (as in your example) and for VAEs, I am not sure if the same idea would prove useful.</p>
  <p>P.S. I wanted to add this as a comment but I do not have enough reputation to do so.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-16T07:42:59.387Z
Answers.LastActivityDate: 2022-03-16T07:42:59.387Z
Answers.UserDisplayName: avio11
Body: <p>Could it make any sense to choose a larger dimension for the <em>latent space</em> of the VAE with respect to the original input?</p>
  <p>For example, we may want to learn how to reconstruct a relatively low-dimensional input (let's say <span class="math-container">$20$</span> dimensions), then could I define my encoder and decoder to have <span class="math-container">$64,256,512...$</span> hidden neurons before bringing back the reconstruction?</p>
  <p>EDIT:
  Well I've thought about that and I think it would still be reasonable as in latent-variable models we are actually assuming that our original observations are generated from unseen 'hidden' variables. And (I think) the lower dimension of the latent space is only assumed for an original dimensionality-reduction purpose.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-15T14:05:07.520Z
Comments.Text: I recommend that you put your **specific** question in the title. Here, your question is very specific and it's not just "Dimension of the latent space in VAE". Anyway, one thing is not clear to me. It's clear that you're suggesting to improve the dimensionality of the layers of the encoder, but what about the decoder? Let's say the encoder produces a latent vector of dimension 512, which is bigger than the input, then the decoder should do what? Should it do like the usual role of the encoder and convert this latent vector to a lower-dimensional reconstructed input? Is this what you mean?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-15T15:14:03.067Z
Comments.Text: Apologizes, I've edited the question title with a more specific one. Anyway yes, shortly speaking we would invert the encoder/decoder role in such a situation. Well I've thought about that and I think it would still be reasonable as in latent-variable models we are actually assuming that our original observations are generated from unseen 'hidden' variables. And (I think) the lower dimension of the latent space is only assumed for an original dimensionality-reduction purpose.
Comments.UserDisplayName: James Arten
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-15T16:46:47.513Z
Comments.Text: I would recommend that you add this useful info in your comment to your post directly (because comments are temporary so they might get deleted later)!
Comments.UserDisplayName: nbro
Tags: <variational-autoencoder><hyper-parameters><latent-variable>
Title: Could it make any sense to choose a larger dimension for the latent space of the VAE with respect to the original input?
UserDisplayName: James Arten
ThreadText: # Could it make any sense to choose a larger dimension for the latent space of the VAE with respect to the original input?
  
  (Asked by: username_0 on 2022-03-15)
  
  Could it make any sense to choose a larger dimension for the *latent space* of the VAE with respect to the original input?
  
  For example, we may want to learn how to reconstruct a relatively low-dimensional input (let's say $20$ dimensions), then could I define my encoder and decoder to have $64,256,512...$ hidden neurons before bringing back the reconstruction?
  
  EDIT:
  Well I've thought about that and I think it would still be reasonable as in latent-variable models we are actually assuming that our original observations are generated from unseen 'hidden' variables. And (I think) the lower dimension of the latent space is only assumed for an original dimensionality-reduction purpose.
  
  **Tags:** variational-autoencoder, hyper-parameters, latent-variable
  > **Comments:**
  > **username_1**: I recommend that you put your **specific** question in the title. Here, your question is very specific and it's not just "Dimension of the latent space in VAE". Anyway, one thing is not clear to me. It's clear that you're suggesting to improve the dimensionality of the layers of the encoder, but what about the decoder? Let's say the encoder produces a latent vector of dimension 512, which is bigger than the input, then the decoder should do what? Should it do like the usual role of the encoder and convert this latent vector to a lower-dimensional reconstructed input? Is this what you mean?  
  > **username_0**: Apologizes, I've edited the question title with a more specific one. Anyway yes, shortly speaking we would invert the encoder/decoder role in such a situation. Well I've thought about that and I think it would still be reasonable as in latent-variable models we are actually assuming that our original observations are generated from unseen 'hidden' variables. And (I think) the lower dimension of the latent space is only assumed for an original dimensionality-reduction purpose.  
  > **username_1**: I would recommend that you add this useful info in your comment to your post directly (because comments are temporary so they might get deleted later)!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-03-16. Score: 1
  
  I am not sure about the VAE in particular, but the convnext presented [here](https://arxiv.org/pdf/2201.03545.pdf) uses the "inverse bottleneck" (i.e. internal representations being higher dimensional than inputs) as one of the core changes that leads to an increase in its performance when compared to prior convolutional networks. This and some other changes make the convnet be more competitive against the visual transformer (that is pretrained in a large dataset).
  
  But for really low-dimensional data (as in your example) and for VAEs, I am not sure if the same idea would prove useful.
  
  P.S. I wanted to add this as a comment but I do not have enough reputation to do so.
  
  
  ---
  
  