<p><a href="https://github.com/facebookresearch/audiocraft/blob/adf0b04a4452f171970028fcf80f101dd5e26e19/audiocraft/modules/transformer.py#L569" rel="nofollow noreferrer">https://github.com/facebookresearch/audiocraft/blob/adf0b04a4452f171970028fcf80f101dd5e26e19/audiocraft/modules/transformer.py#L569</a></p>
<blockquote>
<p>Is it like when the text condition is not present the middle sub-block is skipped with an if statement?</p>
</blockquote>
<p>Answer is Yes. See the link</p>

CC BY-SA 4.0
2024-07-16T13:19:36.153Z
2024-07-16T13:19:36.153Z
qmzp
<p>Cross-attention reduces to self-attention</p>

CC BY-SA 4.0
2024-06-17T15:07:10.820Z
Hi @Sahil Goyal, and welcome to AI Stack Exchange! It may help the OP if you would add some more details to this answer. Thank you for posting, and we hope to see more of your posts here in the near future!
DeepQZero
CC BY-SA 4.0
2024-06-19T14:25:03.667Z
Can you provide references from previous works?
qmzp
CC BY-SA 4.0
2024-06-19T17:48:10.397Z
A recent reference that I remember is, https://arxiv.org/pdf/2301.00704 (see section 2.7).
Sahil Goyal
CC BY-SA 4.0
2024-07-16T16:40:34.497Z
Yes, Thanks! Also In below link (music gen using Transformers), there's a if statement as I asked in the question.
https://github.com/facebookresearch/audiocraft/blob/adf0b04a4452f171970028fcf80f101dd5e26e19/audiocraft/modules/transformer.py#L569
qmzp
CC BY-SA 4.0
2024-06-17T13:29:16.927Z
2024-06-17T13:29:16.927Z
Sahil Goyal
<p><a href="https://i.sstatic.net/v8te0AAo.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/v8te0AAo.png" alt="enter image description here" /></a></p>
<p>I'm working on music generation using transformers.
Using the decoder part for the audio tokens with text conditioning by the T5 encoder</p>
<p><a href="https://i.sstatic.net/fztWztX6.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fztWztX6.png" alt="enter image description here" /></a>
In Classifier-Free-Guidance, the text conditioning randomly dropped with a probability of <span class="math-container">$p_{uncond}$</span>. So when it's dropped, what should the cross-attention input be, or how is the transformer architecture modified?</p>
<p>Is it like when the text condition is not present the middle sub-block is skipped with an if statement?</p>

<deep-learning><natural-language-processing><transformer>
Classifier-Free-Guidance with Transformers
qmzp
# Classifier-Free-Guidance with Transformers

(Asked by: username_0 on 2024-06-09)

[![enter image description here](https://i.sstatic.net/v8te0AAo.png)](https://i.sstatic.net/v8te0AAo.png)

I'm working on music generation using transformers.
Using the decoder part for the audio tokens with text conditioning by the T5 encoder

[![enter image description here](https://i.sstatic.net/fztWztX6.png)](https://i.sstatic.net/fztWztX6.png)
In Classifier-Free-Guidance, the text conditioning randomly dropped with a probability of $p\_{uncond}$. So when it's dropped, what should the cross-attention input be, or how is the transformer architecture modified?

Is it like when the text condition is not present the middle sub-block is skipped with an if statement?

**Tags:** deep-learning, natural-language-processing, transformer


---

## 2 answers

### Answer by username_0 on 2024-07-16. Score: 0 | Accepted answer

<https://github.com/facebookresearch/audiocraft/blob/adf0b04a4452f171970028fcf80f101dd5e26e19/audiocraft/modules/transformer.py#L569>

> Is it like when the text condition is not present the middle sub-block is skipped with an if statement?

Answer is Yes. See the link


---

### Answer by username_1 on 2024-06-17. Score: 0

Cross-attention reduces to self-attention

> **Comments:**
> **username_2**: Hi @username_1, and welcome to AI Stack Exchange! It may help the OP if you would add some more details to this answer. Thank you for posting, and we hope to see more of your posts here in the near future!  
> **username_0**: Can you provide references from previous works?  
> **username_1**: A recent reference that I remember is, https://arxiv.org/pdf/2301.00704 (see section 2.7).  
> **username_0**: Yes, Thanks! Also In below link (music gen using Transformers), there's a if statement as I asked in the question. https://github.com/facebookresearch/audiocraft/blob/adf0b04a4452f171970028fcf80f101dd5e26e19/audiocraft/modules/transformer.py#L569  

---

