Answers.Body: <p>The original paper that first introduced the Multi-Head Attention is called <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">&quot;Attention Is All You Need&quot;</a>.</p>
  <p>There are also a couple of tutorials that give more didactic introduction into MHA and Transofrmer architecture:</p>
  <ul>
  <li>Andrej Karpathy <a href="https://www.youtube.com/watch?v=kCc8FmEb1nY" rel="nofollow noreferrer">&quot;Let's build GPT: from scratch&quot;</a> video and <a href="https://github.com/karpathy/minGPT" rel="nofollow noreferrer">miniGPT</a> github repository.</li>
  <li>Jay Alammar: <a href="http://jalammar.github.io/illustrated-transformer/" rel="nofollow noreferrer">&quot;Illustrated Transformer&quot;</a> blog post.</li>
  <li>Michael Phi: <a href="https://www.youtube.com/watch?v=4Bdc55j80l8" rel="nofollow noreferrer">&quot;Illustrated Guide to Transformers&quot;</a> video.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-31T22:47:14.660Z
Answers.Text: The question is about **backpropagation** of the transformers multi-head layer not about the transformers or multi-head attention. So, we can conclude that the OP wants to know how backpropagation works in these layers specifically. It's also true they didn't provide more context. You don't mention BP in your answer. Does any of your links talk about this?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-29T23:13:00.933Z
Answers.LastActivityDate: 2023-05-29T23:13:00.933Z
Answers.UserDisplayName: Kostya
Body: <p>Is there any reference about backpropagation of the Transformer's multi-head layer or multi-head attention (MHA)? I have searched various journals but have not found one yet.</p>
  
Tags: <reference-request><transformer><backpropagation>
Title: Is there any reference about backpropagation of the Transformer's multi-head layer?
UserDisplayName: poglhar
ThreadText: # Is there any reference about backpropagation of the Transformer's multi-head layer?
  
  (Asked by: username_0 on 2023-05-29)
  
  Is there any reference about backpropagation of the Transformer's multi-head layer or multi-head attention (MHA)? I have searched various journals but have not found one yet.
  
  **Tags:** reference-request, transformer, backpropagation
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-05-29. Score: 0
  
  The original paper that first introduced the Multi-Head Attention is called ["Attention Is All You Need"](https://arxiv.org/abs/1706.03762).
  
  There are also a couple of tutorials that give more didactic introduction into MHA and Transofrmer architecture:
  
  * Andrej Karpathy ["Let's build GPT: from scratch"](https://www.youtube.com/watch?v=kCc8FmEb1nY) video and [miniGPT](https://github.com/karpathy/minGPT) github repository.
  * Jay Alammar: ["Illustrated Transformer"](http://jalammar.github.io/illustrated-transformer/) blog post.
  * Michael Phi: ["Illustrated Guide to Transformers"](https://www.youtube.com/watch?v=4Bdc55j80l8) video.
  
  > **Comments:**
  > **username_2**: The question is about **backpropagation** of the transformers multi-head layer not about the transformers or multi-head attention. So, we can conclude that the OP wants to know how backpropagation works in these layers specifically. It's also true they didn't provide more context. You don't mention BP in your answer. Does any of your links talk about this?  
  
  ---
  
  