Answers.Body: <p>There's not really a restriction on the shape for variational autoencoders. If you really wanted a 1D vector, you could just flatten the matrix and get a vector of size <code>64 * 64 * 3</code>.</p>
  <p>The 64x64x3 size makes sense if you consider why they encode images into a latent dimension in the first place.</p>
  <p>Consider this passage from the <a href="https://arxiv.org/abs/2112.10752" rel="nofollow noreferrer">latent diffusion paper</a>:</p>
  <blockquote>
  <p>(i) By leaving the high-dimensional image space, we obtain DMs which are computationally much more efficient because sampling is performed on a low-dimensional space. (ii) We exploit the inductive bias of DMs inherited from their UNet architecture [71], which makes them particularly effective for data with spatial structure and therefore alleviates the need for aggressive, quality-reducing compression levels as required by previous approaches [23, 66].</p>
  </blockquote>
  <p>and</p>
  <blockquote>
  <p>This is in contrast to previous works [23, 66], which relied on an arbitrary 1D ordering of the learned space z to model its distribution autoregressively and thereby ignored much of the inherent structure of z.</p>
  </blockquote>
  <p>The purpose of the latent encoding is to compress images to a lower-dimensional space that's easier to work with---but you also want to make sure that you aren't losing information during the compression process.</p>
  <p>A 2D representation would better models spatial structures that subsequent UNet layers are designed to work with.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-11T09:30:09.527Z
Answers.Text: Thank you for the feedback! But if we consider 3D (binary voxel) data instead of images, then what structure of latent representation would be useful: 2D or 3D? The issue is that I can't find any 3D latent diffusion implementations.
Answers.UserDisplayName: Renat Abdrakhmanov
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-11T18:18:30.307Z
Answers.Text: Yeah, I'm not too familiar with this as well. Although this paper (https://snap-research.github.io/3DVADER/paper.pdf) seems to do things in a different way. Their latent has three dimensions (whereas you'd expect four if you were to extend the 2D diffusion model).
Answers.UserDisplayName: Alexander Wan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-10T21:35:11.507Z
Answers.LastActivityDate: 2024-02-10T21:35:11.507Z
Answers.UserDisplayName: Alexander Wan
Body: <p>Since the encoding is performed by a Variational Autoencoder, the VAE encoder must output some <code>mean</code> and <code>log variance</code> that we can use to sample a latent vector of shape <code>(latent_dim,)</code>. But then how does Stable Diffusion use a vector of shape <code>64x64x3</code>?</p>
  <p>Reference (<a href="https://arxiv.org/pdf/2112.10752.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/2112.10752.pdf</a>, page 24, table 12):
  <a href="https://i.sstatic.net/6II6x.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6II6x.png" alt="Table of latent vector dimensions for diffusion models" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-10T16:19:33.803Z
Comments.Text: Can you provide a reference that tells that the latent space has the dimensionality you say?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-10T18:20:46.117Z
Comments.Text: check updated question
Comments.UserDisplayName: Renat Abdrakhmanov
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-11T01:17:45.223Z
Comments.Text: Can you provide a citation/link to the paper where you got the table? I assume it's from Rombach et al., 2022?
Comments.UserDisplayName: Alexander Wan
Tags: <keras><generative-adversarial-networks><pytorch><variational-autoencoder><diffusion-models>
Title: Why does the latent space in Stable Diffusion have a shape of 64x64x3?
UserDisplayName: Renat Abdrakhmanov
ThreadText: # Why does the latent space in Stable Diffusion have a shape of 64x64x3?
  
  (Asked by: username_0 on 2024-02-09)
  
  Since the encoding is performed by a Variational Autoencoder, the VAE encoder must output some `mean` and `log variance` that we can use to sample a latent vector of shape `(latent_dim,)`. But then how does Stable Diffusion use a vector of shape `64x64x3`?
  
  Reference (<https://arxiv.org/pdf/2112.10752.pdf>, page 24, table 12):
  [![Table of latent vector dimensions for diffusion models](https://i.sstatic.net/6II6x.png)](https://i.sstatic.net/6II6x.png)
  
  **Tags:** keras, generative-adversarial-networks, pytorch, variational-autoencoder, diffusion-models
  > **Comments:**
  > **username_1**: Can you provide a reference that tells that the latent space has the dimensionality you say?  
  > **username_0**: check updated question  
  > **username_2**: Can you provide a citation/link to the paper where you got the table? I assume it's from Rombach et al., 2022?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-02-10. Score: 2 | Accepted answer
  
  There's not really a restriction on the shape for variational autoencoders. If you really wanted a 1D vector, you could just flatten the matrix and get a vector of size `64 * 64 * 3`.
  
  The 64x64x3 size makes sense if you consider why they encode images into a latent dimension in the first place.
  
  Consider this passage from the [latent diffusion paper](https://arxiv.org/abs/2112.10752):
  
  > (i) By leaving the high-dimensional image space, we obtain DMs which are computationally much more efficient because sampling is performed on a low-dimensional space. (ii) We exploit the inductive bias of DMs inherited from their UNet architecture [71], which makes them particularly effective for data with spatial structure and therefore alleviates the need for aggressive, quality-reducing compression levels as required by previous approaches [23, 66].
  
  and
  
  > This is in contrast to previous works [23, 66], which relied on an arbitrary 1D ordering of the learned space z to model its distribution autoregressively and thereby ignored much of the inherent structure of z.
  
  The purpose of the latent encoding is to compress images to a lower-dimensional space that's easier to work with---but you also want to make sure that you aren't losing information during the compression process.
  
  A 2D representation would better models spatial structures that subsequent UNet layers are designed to work with.
  
  > **Comments:**
  > **username_0**: Thank you for the feedback! But if we consider 3D (binary voxel) data instead of images, then what structure of latent representation would be useful: 2D or 3D? The issue is that I can't find any 3D latent diffusion implementations.  
  > **username_2**: Yeah, I'm not too familiar with this as well. Although this paper (https://snap-research.github.io/3DVADER/paper.pdf) seems to do things in a different way. Their latent has three dimensions (whereas you'd expect four if you were to extend the 2D diffusion model).  
  
  ---
  
  