Body: <p>I've trained a VQ-VAE on images that I want to use to condition another model (latent diffusion). The shape of my encoder's output is (4, 256, 56, 56) and I'm getting it using:</p>
  <pre><code>z_e = model.encode(input)
  _, argmin = model.emb(z_e, weight_sg=True)
  </code></pre>
  <p>My understanding is that <code>argmin</code> is my latent code (which seems to be supported by the fact that this is an integer tensor). However, I'm noticing that this tensor contains duplicates. I'm not necessarily assuming this as a problem, but I wonder if I should remove them before sending them to the latent diffusion model? The issue there is that <em>that</em> model will presumably expect a fixed shape... Should I just pass the tensor on and not worry about the duplicates?</p>
  <p>One other thing I'm wondering is whether I should normalize the <code>argmin</code> tensor before sending it to the latent diffusion model?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-06T14:20:39.753Z
Comments.Text: I guess by "tensor" you mean that you're getting several vectors, some of which are duplicates. Is that because you give the VQ-VAE several input samples? Whether you remove duplicated depends on what goals you're pursuing by processing those several samples independently in the VQ-VAE and jointly in the diffusion model. "Fixed shape" might be a red flag - why fix the number of samples, especially if you're considering removing duplicates dynamically? Also note that if the order of samples doesn't matter, the diffusion model should be invariant under changes of that order (permutations).
Comments.UserDisplayName: root
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-11T15:07:46.973Z
Comments.Text: Searching around I was able to confirm that the duplicates are important to keep, so I won't mess with them. These are repetitions of individual codebook indices, not whole vectors, afaik. The tensor I'm talking about is giving me the codebook indices, some of which are repetitions. But since the structure is represented by these indices, they're clearly all necessary to keep. I had seen an example online that filtered the repetitions out, but perhaps there was another reason for that, like visualizing overall codebook usage... not sure.
Comments.UserDisplayName: jbm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-11T15:46:29.333Z
Comments.Text: Indices are an inappropriate input data representation for neural-network layers such as attention. Such layers tend to assign similar meaning to similar input values and dissimilar meaning to dissimilar input values, which in general is not the case for indices.
Comments.UserDisplayName: root
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-12T16:05:59.223Z
Comments.Text: As I understand it, the VQ-VAE uses the indices to recall the embeddings, which are the actual inputs to the decoder.
Comments.UserDisplayName: jbm
Tags: <variational-autoencoder><embeddings>
Title: Using a VQ-VAE encoder's output to condition another model
UserDisplayName: jbm
ThreadText: # Using a VQ-VAE encoder's output to condition another model
  
  (Asked by: username_0 on 2023-02-01)
  
  I've trained a VQ-VAE on images that I want to use to condition another model (latent diffusion). The shape of my encoder's output is (4, 256, 56, 56) and I'm getting it using:
  z_e = model.encode(input)
  _, argmin = model.emb(z_e, weight_sg=True)
  
  My understanding is that argmin is my latent code (which seems to be supported by the fact that this is an integer tensor). However, I'm noticing that this tensor contains duplicates. I'm not necessarily assuming this as a problem, but I wonder if I should remove them before sending them to the latent diffusion model? The issue there is that that model will presumably expect a fixed shape... Should I just pass the tensor on and not worry about the duplicates?
  One other thing I'm wondering is whether I should normalize the argmin tensor before sending it to the latent diffusion model?
  
  **Tags:** variational-autoencoder, embeddings
  > **Comments:**
  > **username_1**: I guess by "tensor" you mean that you're getting several vectors, some of which are duplicates. Is that because you give the VQ-VAE several input samples? Whether you remove duplicated depends on what goals you're pursuing by processing those several samples independently in the VQ-VAE and jointly in the diffusion model. "Fixed shape" might be a red flag - why fix the number of samples, especially if you're considering removing duplicates dynamically? Also note that if the order of samples doesn't matter, the diffusion model should be invariant under changes of that order (permutations).  
  > **username_0**: Searching around I was able to confirm that the duplicates are important to keep, so I won't mess with them. These are repetitions of individual codebook indices, not whole vectors, afaik. The tensor I'm talking about is giving me the codebook indices, some of which are repetitions. But since the structure is represented by these indices, they're clearly all necessary to keep. I had seen an example online that filtered the repetitions out, but perhaps there was another reason for that, like visualizing overall codebook usage... not sure.  
  > **username_1**: Indices are an inappropriate input data representation for neural-network layers such as attention. Such layers tend to assign similar meaning to similar input values and dissimilar meaning to dissimilar input values, which in general is not the case for indices.  
  > **username_0**: As I understand it, the VQ-VAE uses the indices to recall the embeddings, which are the actual inputs to the decoder.  
  
  ---
  
  ## 0 answers
  
  