<p>Section 4.3.2 of the paper (on p. 7 in v2) answers this question:</p>
<blockquote>
<p>By concatenating spatially aligned conditioning information to the input of <span class="math-container">$\epsilon_θ$</span>, LDMs can serve as efficient general purpose image-to-image translation models. We use this to train models for semantic synthesis, super-resolution
(Sec. 4.4) and inpainting (Sec. 4.5).</p>
</blockquote>
<p>In other words, they switch the network based on the application.  If the input is a class or text, they use the cross-attention mechanism.  If the input is an image, they use the concatenation mechanism.</p>
<p>This makes a lot of sense.  Textual and class guidance don't have spatial information, and thus cannot be concatenated to the spatial input to the denoising U-nets.</p>
<p>But class maps and images with to-be-inpainted regions are already spatial, and thus can be concatenated to the input to the U-Net.  Cross attention would lose the spatial information these inputs provide.</p>
<p>This leads to a clear answer to your question: If you were guiding image generation both with text and with an existing image, you would likely use both paths.</p>

CC BY-SA 4.0
2024-05-02T02:06:08.900Z
Let's say there is an image captioned "a man riding a horse" pretrained in the network... the prompt asks for "an astronaut riding a horse", so the weights push the output towards "a man riding a horse" image, segment out "man" using the u-net, and replaces the cutout with an "astronaut" image pretrained in the network? But how does the posture and size of the "astronaut" on file (who is probably not in a horse-riding posture) be matched to the "man" who *is* in a horse-riding posture?
James
CC BY-SA 4.0
2024-05-02T13:55:28.540Z
@James That's a new question. Feel free to create a new question and link to it from here.
Josiah Yoder
CC BY-SA 4.0
2024-05-01T18:37:22.800Z
2024-05-01T18:37:22.800Z
Josiah Yoder
<p>Consider the following diagram from the paper titled <strong><a href="https://openaccess.thecvf.com/content/CVPR2022/papers/Rombach_High-Resolution_Image_Synthesis_With_Latent_Diffusion_Models_CVPR_2022_paper.pdf" rel="nofollow noreferrer">High-Resolution Image Synthesis with Latent Diffusion Models</a></strong> by Robin Rombach et. al.,</p>
<p><a href="https://i.sstatic.net/zfyNp.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zfyNp.png" alt="enter image description here" /></a></p>
<p>In the context of this diagram, I'm uncertain about the functionality of a particular component referred to as the &quot;switch.&quot; Based on my understanding, the conditioning information always flows to the denoising step and is directed either to the cross-attention module or to concatenate with <span class="math-container">$z_{T}$</span>, but not to both simultaneously. Is my understanding correct? Could you explain why the conditioning information cannot be passed to both components at the same time? Is there a specific reason or mechanism described in the paper for this design choice?</p>

CC BY-SA 4.0
2023-12-24T22:33:21.810Z
Good question. I also do not understand how the attention output is being used inside the UNET.
Nathan G
CC BY-SA 4.0
2024-03-23T18:43:54.773Z
I have only skimmed the paper so far, but my understanding is that you are correct: "not both simultaneously"
Josiah Yoder
CC BY-SA 4.0
2024-03-23T20:21:10.410Z
If the convolutional self-attention is implemented as discussed [here](https://ai.stackexchange.com/q/43249/15487), then perhaps the choice is between self-attention and cross attention.  I can see why one couldn't do both.
Josiah Yoder
CC BY-SA 4.0
2024-05-01T18:41:51.183Z
Typo to correct: et. al. should be replaced with et al.  et is a complete Latin word meaning "and", not an abbreviation.
Josiah Yoder
<papers><diffusion-models>
Understanding the functionality of the switch in the latent diffusion models: Does conditioning information pass to both cross attention and $z_{T}$?
hanugm
# Understanding the functionality of the switch in the latent diffusion models: Does conditioning information pass to both cross attention and $z_{T}$?

(Asked by: username_0 on 2023-08-09)

Consider the following diagram from the paper titled **[High-Resolution Image Synthesis with Latent Diffusion Models](https://openaccess.thecvf.com/content/CVPR2022/papers/Rombach_High-Resolution_Image_Synthesis_With_Latent_Diffusion_Models_CVPR_2022_paper.pdf)** by Robin Rombach et. al.,

[![enter image description here](https://i.sstatic.net/zfyNp.png)](https://i.sstatic.net/zfyNp.png)

In the context of this diagram, I'm uncertain about the functionality of a particular component referred to as the "switch." Based on my understanding, the conditioning information always flows to the denoising step and is directed either to the cross-attention module or to concatenate with $z\_{T}$, but not to both simultaneously. Is my understanding correct? Could you explain why the conditioning information cannot be passed to both components at the same time? Is there a specific reason or mechanism described in the paper for this design choice?

**Tags:** papers, diffusion-models

> **Comments:**
> **username_1**: Good question. I also do not understand how the attention output is being used inside the UNET.  
> **username_2**: I have only skimmed the paper so far, but my understanding is that you are correct: "not both simultaneously"  
> **username_2**: If the convolutional self-attention is implemented as discussed [here](https://ai.stackexchange.com/q/43249/15487), then perhaps the choice is between self-attention and cross attention.  I can see why one couldn't do both.  
> **username_2**: Typo to correct: et. al. should be replaced with et al.  et is a complete Latin word meaning "and", not an abbreviation.  

---

## 1 answers

### Answer by username_2 on 2024-05-01. Score: 1

Section 4.3.2 of the paper (on p. 7 in v2) answers this question:

> By concatenating spatially aligned conditioning information to the input of $\epsilon\_θ$, LDMs can serve as efficient general purpose image-to-image translation models. We use this to train models for semantic synthesis, super-resolution
> (Sec. 4.4) and inpainting (Sec. 4.5).

In other words, they switch the network based on the application. If the input is a class or text, they use the cross-attention mechanism. If the input is an image, they use the concatenation mechanism.

This makes a lot of sense. Textual and class guidance don't have spatial information, and thus cannot be concatenated to the spatial input to the denoising U-nets.

But class maps and images with to-be-inpainted regions are already spatial, and thus can be concatenated to the input to the U-Net. Cross attention would lose the spatial information these inputs provide.

This leads to a clear answer to your question: If you were guiding image generation both with text and with an existing image, you would likely use both paths.

> **Comments:**
> **username_3**: Let's say there is an image captioned "a man riding a horse" pretrained in the network... the prompt asks for "an astronaut riding a horse", so the weights push the output towards "a man riding a horse" image, segment out "man" using the u-net, and replaces the cutout with an "astronaut" image pretrained in the network? But how does the posture and size of the "astronaut" on file (who is probably not in a horse-riding posture) be matched to the "man" who *is* in a horse-riding posture?  
> **username_2**: @username_3 That's a new question. Feel free to create a new question and link to it from here.  

---

