Answers.Body: <p>Section 4.3.2 of the paper (on p. 7 in v2) answers this question:</p>
  <blockquote>
  <p>By concatenating spatially aligned conditioning information to the input of <span class="math-container">$\epsilon_θ$</span>, LDMs can serve as efficient general purpose image-to-image translation models. We use this to train models for semantic synthesis, super-resolution
  (Sec. 4.4) and inpainting (Sec. 4.5).</p>
  </blockquote>
  <p>In other words, they switch the network based on the application.  If the input is a class or text, they use the cross-attention mechanism.  If the input is an image, they use the concatenation mechanism.</p>
  <p>This makes a lot of sense.  Textual and class guidance don't have spatial information, and thus cannot be concatenated to the spatial input to the denoising U-nets.</p>
  <p>But class maps and images with to-be-inpainted regions are already spatial, and thus can be concatenated to the input to the U-Net.  Cross attention would lose the spatial information these inputs provide.</p>
  <p>This leads to a clear answer to your question: If you were guiding image generation both with text and with an existing image, you would likely use both paths.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-02T02:06:08.900Z
Answers.Text: Let's say there is an image captioned "a man riding a horse" pretrained in the network... the prompt asks for "an astronaut riding a horse", so the weights push the output towards "a man riding a horse" image, segment out "man" using the u-net, and replaces the cutout with an "astronaut" image pretrained in the network? But how does the posture and size of the "astronaut" on file (who is probably not in a horse-riding posture) be matched to the "man" who *is* in a horse-riding posture?
Answers.UserDisplayName: James
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-02T13:55:28.540Z
Answers.Text: @James That's a new question. Feel free to create a new question and link to it from here.
Answers.UserDisplayName: Josiah Yoder
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-01T18:37:22.800Z
Answers.LastActivityDate: 2024-05-01T18:37:22.800Z
Answers.UserDisplayName: Josiah Yoder
Body: <p>Consider the following diagram from the paper titled <strong><a href="https://openaccess.thecvf.com/content/CVPR2022/papers/Rombach_High-Resolution_Image_Synthesis_With_Latent_Diffusion_Models_CVPR_2022_paper.pdf" rel="nofollow noreferrer">High-Resolution Image Synthesis with Latent Diffusion Models</a></strong> by Robin Rombach et. al.,</p>
  <p><a href="https://i.sstatic.net/zfyNp.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zfyNp.png" alt="enter image description here" /></a></p>
  <p>In the context of this diagram, I'm uncertain about the functionality of a particular component referred to as the &quot;switch.&quot; Based on my understanding, the conditioning information always flows to the denoising step and is directed either to the cross-attention module or to concatenate with <span class="math-container">$z_{T}$</span>, but not to both simultaneously. Is my understanding correct? Could you explain why the conditioning information cannot be passed to both components at the same time? Is there a specific reason or mechanism described in the paper for this design choice?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-24T22:33:21.810Z
Comments.Text: Good question. I also do not understand how the attention output is being used inside the UNET.
Comments.UserDisplayName: Nathan G
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-23T18:43:54.773Z
Comments.Text: I have only skimmed the paper so far, but my understanding is that you are correct: "not both simultaneously"
Comments.UserDisplayName: Josiah Yoder
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-23T20:21:10.410Z
Comments.Text: If the convolutional self-attention is implemented as discussed [here](https://ai.stackexchange.com/q/43249/15487), then perhaps the choice is between self-attention and cross attention.  I can see why one couldn't do both.
Comments.UserDisplayName: Josiah Yoder
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-01T18:41:51.183Z
Comments.Text: Typo to correct: et. al. should be replaced with et al.  et is a complete Latin word meaning "and", not an abbreviation.
Comments.UserDisplayName: Josiah Yoder
Tags: <papers><diffusion-models>
Title: Understanding the functionality of the switch in the latent diffusion models: Does conditioning information pass to both cross attention and $z_{T}$?
UserDisplayName: hanugm
ThreadText: # Understanding the functionality of the switch in the latent diffusion models: Does conditioning information pass to both cross attention and $z_{T}$?
  
  (Asked by: username_0 on 2023-08-09)
  
  Consider the following diagram from the paper titled **[High-Resolution Image Synthesis with Latent Diffusion Models](https://openaccess.thecvf.com/content/CVPR2022/papers/Rombach_High-Resolution_Image_Synthesis_With_Latent_Diffusion_Models_CVPR_2022_paper.pdf)** by Robin Rombach et. al.,
  
  [![enter image description here](https://i.sstatic.net/zfyNp.png)](https://i.sstatic.net/zfyNp.png)
  
  In the context of this diagram, I'm uncertain about the functionality of a particular component referred to as the "switch." Based on my understanding, the conditioning information always flows to the denoising step and is directed either to the cross-attention module or to concatenate with $z\_{T}$, but not to both simultaneously. Is my understanding correct? Could you explain why the conditioning information cannot be passed to both components at the same time? Is there a specific reason or mechanism described in the paper for this design choice?
  
  **Tags:** papers, diffusion-models
  > **Comments:**
  > **username_1**: Good question. I also do not understand how the attention output is being used inside the UNET.  
  > **username_2**: I have only skimmed the paper so far, but my understanding is that you are correct: "not both simultaneously"  
  > **username_2**: If the convolutional self-attention is implemented as discussed [here](https://ai.stackexchange.com/q/43249/15487), then perhaps the choice is between self-attention and cross attention.  I can see why one couldn't do both.  
  > **username_2**: Typo to correct: et. al. should be replaced with et al.  et is a complete Latin word meaning "and", not an abbreviation.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-05-01. Score: 1
  
  Section 4.3.2 of the paper (on p. 7 in v2) answers this question:
  
  > By concatenating spatially aligned conditioning information to the input of $\epsilon\_θ$, LDMs can serve as efficient general purpose image-to-image translation models. We use this to train models for semantic synthesis, super-resolution
  > (Sec. 4.4) and inpainting (Sec. 4.5).
  
  In other words, they switch the network based on the application. If the input is a class or text, they use the cross-attention mechanism. If the input is an image, they use the concatenation mechanism.
  
  This makes a lot of sense. Textual and class guidance don't have spatial information, and thus cannot be concatenated to the spatial input to the denoising U-nets.
  
  But class maps and images with to-be-inpainted regions are already spatial, and thus can be concatenated to the input to the U-Net. Cross attention would lose the spatial information these inputs provide.
  
  This leads to a clear answer to your question: If you were guiding image generation both with text and with an existing image, you would likely use both paths.
  
  > **Comments:**
  > **username_3**: Let's say there is an image captioned "a man riding a horse" pretrained in the network... the prompt asks for "an astronaut riding a horse", so the weights push the output towards "a man riding a horse" image, segment out "man" using the u-net, and replaces the cutout with an "astronaut" image pretrained in the network? But how does the posture and size of the "astronaut" on file (who is probably not in a horse-riding posture) be matched to the "man" who *is* in a horse-riding posture?  
  > **username_2**: @James That's a new question. Feel free to create a new question and link to it from here.  
  
  ---
  
  