Answers.Body: <p>I noticed the same issue as well. I was working in conditioned generation of spectrogram, in which it works well with spectrogram. But after pipe-in a pre-trained VAE into the latent space and training on the latent space, the generation output become blank and the loss doesn't visibly decrease.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-09T01:39:03.860Z
Answers.LastActivityDate: 2024-10-09T01:39:03.860Z
Answers.UserDisplayName: anu
Body: <p>I'm currently playing around with LDMs on the MNIST-Fashion dataset. I thought the VQVAEs used in the original paper were a bit overkill for what I'm doing (and I don't fully understand how they construct the discretized codebook latent space), so I went with a simple convolutional autoencoder with a kl-regularizer to map to an approximately gaussian latent space. I've run this model a few times and verified that it does reconstruct the original image inputs fairly well.</p>
  <p>I run into issues when I try to use this model with my LDM implementation. I first made sure standard diffusion directly in image space works, which it does. I then tried latent diffusion with my trained autoencoder and I can't get the loss to drop below a certain threshold (~1.0), and the image outputs are pretty much gaussian noise still.</p>
  <p>As stated in the paper, I sample latent vectors from the encoder part of the autoencoder and scale them with stats collected from the first batch of data like so:</p>
  <pre><code>                    batch = autoencoder.encode(batch).sample().detach()
                      # rescale the embeddings to be unit variance
                      if epoch == 0 and step == 0:
                          print(&quot;Calculating scale factor...&quot;)
                          std = batch.flatten().std()
                          scale_factor = 1. / std
                          cfg.scale_factor = scale_factor.item()
                      batch *= scale_factor
  </code></pre>
  <p>And then pretty much everything else (applying noise, calculating loss, etc) is the same as standard diffusion. Am I missing something, or is the latent space of my simple conv-autoencoder hard to learn for some reason? I would think that, since an autoencoder is just a deterministic mapping from image space to a lower dimensional one, that the LDM should be able to learn the latter just fine, or is there something inherent in the original paper's transformer architectures for the encoder / codebook latent space that is important to learning for the LDM?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-27T11:26:37.837Z
Comments.Text: Rather than writing "**Question on** Autoencoders Used in Latent Diffusion Models", can you please just put your **specific question** in the title?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-03T16:18:01.083Z
Comments.Text: Two things come to my mind: 1. LDM uses a quite deep convolutional auto encoder, this may not required for Fashion-MNIST, but have you tried changing the depth and see how your result changes? 2. LDM uses the attention mechanism to condition the diffusion. How are you currently conditioning your diffusion process?
Comments.UserDisplayName: Ciodar
Tags: <deep-learning><computer-vision><transformer><autoencoders><diffusion-models>
Title: Latent Diffusion Model Can't Learn the Latent Space of a VAE for the MNIST-Fashion Dataset
UserDisplayName: sb3
ThreadText: # Latent Diffusion Model Can't Learn the Latent Space of a VAE for the MNIST-Fashion Dataset
  
  (Asked by: username_0 on 2023-02-24)
  
  I'm currently playing around with LDMs on the MNIST-Fashion dataset. I thought the VQVAEs used in the original paper were a bit overkill for what I'm doing (and I don't fully understand how they construct the discretized codebook latent space), so I went with a simple convolutional autoencoder with a kl-regularizer to map to an approximately gaussian latent space. I've run this model a few times and verified that it does reconstruct the original image inputs fairly well.
  
  I run into issues when I try to use this model with my LDM implementation. I first made sure standard diffusion directly in image space works, which it does. I then tried latent diffusion with my trained autoencoder and I can't get the loss to drop below a certain threshold (~1.0), and the image outputs are pretty much gaussian noise still.
  
  As stated in the paper, I sample latent vectors from the encoder part of the autoencoder and scale them with stats collected from the first batch of data like so:
  
  ```
                      batch = autoencoder.encode(batch).sample().detach()
                      # rescale the embeddings to be unit variance
                      if epoch == 0 and step == 0:
                          print("Calculating scale factor...")
                          std = batch.flatten().std()
                          scale_factor = 1. / std
                          cfg.scale_factor = scale_factor.item()
                      batch *= scale_factor
  
  ```
  
  And then pretty much everything else (applying noise, calculating loss, etc) is the same as standard diffusion. Am I missing something, or is the latent space of my simple conv-autoencoder hard to learn for some reason? I would think that, since an autoencoder is just a deterministic mapping from image space to a lower dimensional one, that the LDM should be able to learn the latter just fine, or is there something inherent in the original paper's transformer architectures for the encoder / codebook latent space that is important to learning for the LDM?
  
  **Tags:** deep-learning, computer-vision, transformer, autoencoders, diffusion-models
  
  > **Comments:**
  > **username_1**: Rather than writing "**Question on** Autoencoders Used in Latent Diffusion Models", can you please just put your **specific question** in the title?  
  > **username_2**: Two things come to my mind: 1. LDM uses a quite deep convolutional auto encoder, this may not required for Fashion-MNIST, but have you tried changing the depth and see how your result changes? 2. LDM uses the attention mechanism to condition the diffusion. How are you currently conditioning your diffusion process?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2024-10-09. Score: 0
  
  I noticed the same issue as well. I was working in conditioned generation of spectrogram, in which it works well with spectrogram. But after pipe-in a pre-trained VAE into the latent space and training on the latent space, the generation output become blank and the loss doesn't visibly decrease.
  
  
  ---
  
  