Body: <p><a href="https://i.sstatic.net/jzvCW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/jzvCW.png" alt="Self Attention Diagram" /></a></p>
  <p>In this example l’Afrique is x^3 and the attention is being computed for this word A^3(l’Afrique). In the image above Andrew Ng indicates that the word with the biggest wieght in the computation of A^3 will be visite not l’Afrique itself because visite gives more context to l’Afrique. So when all the V values are summed A^3 will contatin more of visite embedding than l’Afrique embedding. So my question is this theoretically if vanishing gradients did not exist would a residual block still be required for a transformer network to work because it would need to maintain the information of the original word x^3 and combine that with A^3 since A^3 has more information about X^2 than X^3 ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-28T01:07:42.010Z
Comments.Text: Please, edit your post to format the formulas with mathjax.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-28T12:25:06.807Z
Comments.Text: Also, please, put your specific question in the title. Thanks.
Comments.UserDisplayName: nbro
Tags: <neural-networks><transformer><attention>
Title: Do I need residual block in a transformer model if vanishing gradients don't exist?
UserDisplayName: Stef
ThreadText: # Do I need residual block in a transformer model if vanishing gradients don't exist?
  
  (Asked by: username_0 on 2023-12-27)
  
  [![Self Attention Diagram](https://i.sstatic.net/jzvCW.png)](https://i.sstatic.net/jzvCW.png)
  
  In this example l’Afrique is x^3 and the attention is being computed for this word A^3(l’Afrique). In the image above Andrew Ng indicates that the word with the biggest wieght in the computation of A^3 will be visite not l’Afrique itself because visite gives more context to l’Afrique. So when all the V values are summed A^3 will contatin more of visite embedding than l’Afrique embedding. So my question is this theoretically if vanishing gradients did not exist would a residual block still be required for a transformer network to work because it would need to maintain the information of the original word x^3 and combine that with A^3 since A^3 has more information about X^2 than X^3 ?
  
  **Tags:** neural-networks, transformer, attention
  
  > **Comments:**
  > **username_1**: Please, edit your post to format the formulas with mathjax.  
  > **username_1**: Also, please, put your specific question in the title. Thanks.  
  
  ---
  
  ## 0 answers
  
  