Answers.Body: <p>The self-attention part 'enriches' the motion sequence. The cross-attention module mixes two different embedding sequences. In a language translation model, the cross-attention module is the part where the actual translation happens. In the MotionDiffuse paper, this is the part where the two modalities (motion sequence and text) interact.</p>
  <p><a href="https://i.sstatic.net/h6hjs.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/h6hjs.png" alt="minimax game" /></a></p>
  <p>The original attention mechanism however does not scale very well. Computing the attention weights leads to quadratically increasing costs. The MotionDiffuse paper makes use of the speed-ups introduced in the</p>
  <blockquote>
  <p>Efficient Attention paper (Shen et al., 2021). Instead of calculating pair-wise attention weights, efficient attention generates global
  feature map.</p>
  </blockquote>
  <p>See this <a href="https://www.youtube.com/watch?v=EixI6t5oif0" rel="nofollow noreferrer">video</a> for a detailed explanation of the original transformer architecture.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-02T22:26:56.763Z
Answers.Text: Thanks, could this model be considered a latent diffusion model? I just finished reading [this](https://theaisummer.com/diffusion-models/#stable-diffusion-latent-diffusion-models) and it seemed to fit the description.
Answers.UserDisplayName: Nikita Belooussov
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-02T21:24:56.293Z
Answers.LastActivityDate: 2023-01-02T21:24:56.293Z
Answers.UserDisplayName: Mariusmarten
Body: <p>So I am trying to understand how a Cross-Modality Linear Transformer is different from an a basic transformer. I found the transformer mentioned in this <a href="https://arxiv.org/pdf/2208.15001.pdf" rel="nofollow noreferrer">paper</a>. Am I correct in understanding that, the transformer  is cross-modality because it is going from one medium to another. In this case, text to motion. Then linear because the time complexity to calculate the attetion weights are linear? I have troubles with understanding transformers. I just was able to understand the basic concept of them. So any help would be great. Thanks</p>
  
Tags: <transformer>
Title: Explanation of Cross-Modality Linear Transformer
UserDisplayName: Nikita Belooussov
ThreadText: # Explanation of Cross-Modality Linear Transformer
  
  (Asked by: username_0 on 2022-12-31)
  
  So I am trying to understand how a Cross-Modality Linear Transformer is different from an a basic transformer. I found the transformer mentioned in this [paper](https://arxiv.org/pdf/2208.15001.pdf). Am I correct in understanding that, the transformer is cross-modality because it is going from one medium to another. In this case, text to motion. Then linear because the time complexity to calculate the attetion weights are linear? I have troubles with understanding transformers. I just was able to understand the basic concept of them. So any help would be great. Thanks
  
  **Tags:** transformer
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-01-02. Score: 1 | Accepted answer
  
  The self-attention part 'enriches' the motion sequence. The cross-attention module mixes two different embedding sequences. In a language translation model, the cross-attention module is the part where the actual translation happens. In the MotionDiffuse paper, this is the part where the two modalities (motion sequence and text) interact.
  
  [![minimax game](https://i.sstatic.net/h6hjs.png)](https://i.sstatic.net/h6hjs.png)
  
  The original attention mechanism however does not scale very well. Computing the attention weights leads to quadratically increasing costs. The MotionDiffuse paper makes use of the speed-ups introduced in the
  
  > Efficient Attention paper (Shen et al., 2021). Instead of calculating pair-wise attention weights, efficient attention generates global
  > feature map.
  
  See this [video](https://www.youtube.com/watch?v=EixI6t5oif0) for a detailed explanation of the original transformer architecture.
  
  > **Comments:**
  > **username_0**: Thanks, could this model be considered a latent diffusion model? I just finished reading [this](https://theaisummer.com/diffusion-models/#stable-diffusion-latent-diffusion-models) and it seemed to fit the description.  
  
  ---
  
  