Answers.Body: <p><a href="https://github.com/facebookresearch/audiocraft/blob/adf0b04a4452f171970028fcf80f101dd5e26e19/audiocraft/modules/transformer.py#L569" rel="nofollow noreferrer">https://github.com/facebookresearch/audiocraft/blob/adf0b04a4452f171970028fcf80f101dd5e26e19/audiocraft/modules/transformer.py#L569</a></p>
  <blockquote>
  <p>Is it like when the text condition is not present the middle sub-block is skipped with an if statement?</p>
  </blockquote>
  <p>Answer is Yes. See the link</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-16T13:19:36.153Z
Answers.LastActivityDate: 2024-07-16T13:19:36.153Z
Answers.UserDisplayName: qmzp
Answers.Body: <p>Cross-attention reduces to self-attention</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-17T15:07:10.820Z
Answers.Text: Hi @Sahil Goyal, and welcome to AI Stack Exchange! It may help the OP if you would add some more details to this answer. Thank you for posting, and we hope to see more of your posts here in the near future!
Answers.UserDisplayName: DeepQZero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-19T14:25:03.667Z
Answers.Text: Can you provide references from previous works?
Answers.UserDisplayName: qmzp
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-19T17:48:10.397Z
Answers.Text: A recent reference that I remember is, https://arxiv.org/pdf/2301.00704 (see section 2.7).
Answers.UserDisplayName: Sahil Goyal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-16T16:40:34.497Z
Answers.Text: Yes, Thanks! Also In below link (music gen using Transformers), there's a if statement as I asked in the question.
  https://github.com/facebookresearch/audiocraft/blob/adf0b04a4452f171970028fcf80f101dd5e26e19/audiocraft/modules/transformer.py#L569
Answers.UserDisplayName: qmzp
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-17T13:29:16.927Z
Answers.LastActivityDate: 2024-06-17T13:29:16.927Z
Answers.UserDisplayName: Sahil Goyal
Body: <p><a href="https://i.sstatic.net/v8te0AAo.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/v8te0AAo.png" alt="enter image description here" /></a></p>
  <p>I'm working on music generation using transformers.
  Using the decoder part for the audio tokens with text conditioning by the T5 encoder</p>
  <p><a href="https://i.sstatic.net/fztWztX6.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fztWztX6.png" alt="enter image description here" /></a>
  In Classifier-Free-Guidance, the text conditioning randomly dropped with a probability of <span class="math-container">$p_{uncond}$</span>. So when it's dropped, what should the cross-attention input be, or how is the transformer architecture modified?</p>
  <p>Is it like when the text condition is not present the middle sub-block is skipped with an if statement?</p>
  
Tags: <deep-learning><natural-language-processing><transformer>
Title: Classifier-Free-Guidance with Transformers
UserDisplayName: qmzp
ThreadText: # Classifier-Free-Guidance with Transformers
  
  (Asked by: username_0 on 2024-06-09)
  
  [![enter image description here](https://i.sstatic.net/v8te0AAo.png)](https://i.sstatic.net/v8te0AAo.png)
  
  I'm working on music generation using transformers.
  Using the decoder part for the audio tokens with text conditioning by the T5 encoder
  
  [![enter image description here](https://i.sstatic.net/fztWztX6.png)](https://i.sstatic.net/fztWztX6.png)
  In Classifier-Free-Guidance, the text conditioning randomly dropped with a probability of $p\_{uncond}$. So when it's dropped, what should the cross-attention input be, or how is the transformer architecture modified?
  
  Is it like when the text condition is not present the middle sub-block is skipped with an if statement?
  
  **Tags:** deep-learning, natural-language-processing, transformer
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2024-07-16. Score: 0 | Accepted answer
  
  <https://github.com/facebookresearch/audiocraft/blob/adf0b04a4452f171970028fcf80f101dd5e26e19/audiocraft/modules/transformer.py#L569>
  
  > Is it like when the text condition is not present the middle sub-block is skipped with an if statement?
  
  Answer is Yes. See the link
  
  
  ---
  
  ### Answer by username_1 on 2024-06-17. Score: 0
  
  Cross-attention reduces to self-attention
  
  > **Comments:**
  > **username_2**: Hi @username_1, and welcome to AI Stack Exchange! It may help the OP if you would add some more details to this answer. Thank you for posting, and we hope to see more of your posts here in the near future!  
  > **username_0**: Can you provide references from previous works?  
  > **username_1**: A recent reference that I remember is, https://arxiv.org/pdf/2301.00704 (see section 2.7).  
  > **username_0**: Yes, Thanks! Also In below link (music gen using Transformers), there's a if statement as I asked in the question. https://github.com/facebookresearch/audiocraft/blob/adf0b04a4452f171970028fcf80f101dd5e26e19/audiocraft/modules/transformer.py#L569  
  
  ---
  
  