Answers.Body: <p>The problem with autoencoders (conditional or not) is usually the poor structure of the learned latent space. The reconstruction loss guide the decoder to map from <span class="math-container">$\mathcal{N}(0, 1)$</span> to <span class="math-container">$\mathcal{D}$</span> (being your target distrbution) but it does not provide any information about independent features being present within <span class="math-container">$\mathcal{D}$</span> and no geometric structure is therefore learned, as your PCA visualisation show.</p>
  <p>You can try two approaches to improve your model with the specific intent of generating unseen data:</p>
  <ul>
  <li><p>Add classes to enforce the latent space to learn independent features per each class. In this scenario cross entropy can be use to train the encoder to classify the given input. This obviously require having classes information available along with the input data. (in the picture the upper part)</p>
  </li>
  <li><p>Move to an adversarial approach. GANs are better than VAE in generating unseen data, and fortunately the two are not mutually exclusive. I strongly advise to read <a href="https://arxiv.org/abs/2004.04467" rel="nofollow noreferrer">Adversarial Latent Autoencoders</a>. This approach differs from traditional GANs cause it leverages adversarial training to learn proper latent representation rather than proper generated data, putting a patch on the weak point of VAEs.</p>
  </li>
  </ul>
  <p><a href="https://i.sstatic.net/XFW33.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/XFW33.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-19T14:51:53.970Z
Answers.Text: Thanks for the answer, actually I think the latent space of a cVAE model is not required to disperse in classes like the normal VAE model as the class conditions are already given. Hence the geometric structure of my PCA may not be all wrong. What do you think about this? I referred this blog post : `https://stats.stackexchange.com/questions/428599/why-doesnt-conditional-variational-autoencoderscvae-cluster-data-like-the-vani`
Answers.UserDisplayName: Formal_this
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-19T14:52:39.660Z
Answers.Text: Also how to introduce the cross entropy loss in the encoder which you've mentioned?
Answers.UserDisplayName: Formal_this
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-19T15:24:46.863Z
Answers.Text: I agree that your PCA is not wrong. What I tried to explain (maybe poorly, if so I can rewrite the answer) is that succeeding in training a VAE does not necessarily imply the possibility of generating good new data. You just managed to compress and decompress the data, which is the only thing VAE is meant for by design.
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-19T15:27:51.297Z
Answers.Text: by explicitly conditioning on classes you can enforce a smooth transition between them, like in conditional GANs when changing the value of a single latent dimension cause a change of a specific feature in the generated output (like glasses or hairs or age of the same face).
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-19T15:27:57.260Z
Answers.Text: To include the cross entropy you can just change the output of your encoder to be z+c (z=latent dimension or $\mu$ and $\sigma$, c=num classes). You can then compute the usual reconstruction loss using z and cross entropy using c, your final loss would be the sum of both.
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-15T17:35:02.040Z
Answers.LastActivityDate: 2023-05-15T17:35:02.040Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I am using a cVAE model to generate new structures of molecules. After successfully training the VAE model I am able to get proper reconstructions of the training set. While I am also able to generate decent structures  (near correct) when I use the unseen testing dataset. (i.e given a test molecule, I first pass it into the encoder; generate the latent vector and then produce a reconstruction from the decoder.)</p>
  <p>But when I am randomly sampling my latent vector (z) from a Gaussian distribution (normal Gaussian), then the decoded results are not correct. i.e. sample a random vector <code>torch.randn(dim(z))</code> of the dimension of the latent space and pass it through the decoder. The outputs of this result are nowhere valid.</p>
  <p>I have tried to visualise the latent space according to the class conditional as well, and the latent space looks fairly good, pretty much the same as cVAEs. i.e. distributed uniformly across the origin in all directions. The 2-Dimensional and 3-Dimensional latent space distribution looks like the following.</p>
  <p><strong>Any suggestion on what I should try to get this cVAE model working for random generation of new molecules?</strong></p>
  <p><a href="https://i.sstatic.net/PO7n9.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/PO7n9.png" alt="enter image description here" /></a><a href="https://i.sstatic.net/N6axW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/N6axW.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-04T22:20:19.343Z
Comments.Text: Are you sure your latent space is actually normally distributed? Compute a large number of latent vectors from inputs and measure the mean/var of your learned latents
Comments.UserDisplayName: Karl
Tags: <deep-learning><variational-autoencoder>
Title: Any suggestion on what I should try to get this cVAE model working for random generation of new molecules?
UserDisplayName: Formal_this
ThreadText: # Any suggestion on what I should try to get this cVAE model working for random generation of new molecules?
  
  (Asked by: username_0 on 2023-05-15)
  
  I am using a cVAE model to generate new structures of molecules. After successfully training the VAE model I am able to get proper reconstructions of the training set. While I am also able to generate decent structures (near correct) when I use the unseen testing dataset. (i.e given a test molecule, I first pass it into the encoder; generate the latent vector and then produce a reconstruction from the decoder.)
  
  But when I am randomly sampling my latent vector (z) from a Gaussian distribution (normal Gaussian), then the decoded results are not correct. i.e. sample a random vector `torch.randn(dim(z))` of the dimension of the latent space and pass it through the decoder. The outputs of this result are nowhere valid.
  
  I have tried to visualise the latent space according to the class conditional as well, and the latent space looks fairly good, pretty much the same as cVAEs. i.e. distributed uniformly across the origin in all directions. The 2-Dimensional and 3-Dimensional latent space distribution looks like the following.
  
  **Any suggestion on what I should try to get this cVAE model working for random generation of new molecules?**
  
  [![enter image description here](https://i.sstatic.net/PO7n9.png)](https://i.sstatic.net/PO7n9.png)[![enter image description here](https://i.sstatic.net/N6axW.png)](https://i.sstatic.net/N6axW.png)
  
  **Tags:** deep-learning, variational-autoencoder
  
  > **Comments:**
  > **username_1**: Are you sure your latent space is actually normally distributed? Compute a large number of latent vectors from inputs and measure the mean/var of your learned latents  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-05-15. Score: 2 | Accepted answer
  
  The problem with autoencoders (conditional or not) is usually the poor structure of the learned latent space. The reconstruction loss guide the decoder to map from $\mathcal{N}(0, 1)$ to $\mathcal{D}$ (being your target distrbution) but it does not provide any information about independent features being present within $\mathcal{D}$ and no geometric structure is therefore learned, as your PCA visualisation show.
  
  You can try two approaches to improve your model with the specific intent of generating unseen data:
  
  * Add classes to enforce the latent space to learn independent features per each class. In this scenario cross entropy can be use to train the encoder to classify the given input. This obviously require having classes information available along with the input data. (in the picture the upper part)
  * Move to an adversarial approach. GANs are better than VAE in generating unseen data, and fortunately the two are not mutually exclusive. I strongly advise to read [Adversarial Latent Autoencoders](https://arxiv.org/abs/2004.04467). This approach differs from traditional GANs cause it leverages adversarial training to learn proper latent representation rather than proper generated data, putting a patch on the weak point of VAEs.
  
  [![enter image description here](https://i.sstatic.net/XFW33.png)](https://i.sstatic.net/XFW33.png)
  
  > **Comments:**
  > **username_0**: Thanks for the answer, actually I think the latent space of a cVAE model is not required to disperse in classes like the normal VAE model as the class conditions are already given. Hence the geometric structure of my PCA may not be all wrong. What do you think about this? I referred this blog post : `https://stats.stackexchange.com/questions/428599/why-doesnt-conditional-variational-autoencoderscvae-cluster-data-like-the-vani`  
  > **username_0**: Also how to introduce the cross entropy loss in the encoder which you've mentioned?  
  > **username_2**: I agree that your PCA is not wrong. What I tried to explain (maybe poorly, if so I can rewrite the answer) is that succeeding in training a VAE does not necessarily imply the possibility of generating good new data. You just managed to compress and decompress the data, which is the only thing VAE is meant for by design.  
  > **username_2**: by explicitly conditioning on classes you can enforce a smooth transition between them, like in conditional GANs when changing the value of a single latent dimension cause a change of a specific feature in the generated output (like glasses or hairs or age of the same face).  
  > **username_2**: To include the cross entropy you can just change the output of your encoder to be z+c (z=latent dimension or $\mu$ and $\sigma$, c=num classes). You can then compute the usual reconstruction loss using z and cross entropy using c, your final loss would be the sum of both.  
  
  ---
  
  