<p><a href="https://i.sstatic.net/jzvCW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/jzvCW.png" alt="Self Attention Diagram" /></a></p>
<p>In this example l’Afrique is x^3 and the attention is being computed for this word A^3(l’Afrique). In the image above Andrew Ng indicates that the word with the biggest wieght in the computation of A^3 will be visite not l’Afrique itself because visite gives more context to l’Afrique. So when all the V values are summed A^3 will contatin more of visite embedding than l’Afrique embedding. So my question is this theoretically if vanishing gradients did not exist would a residual block still be required for a transformer network to work because it would need to maintain the information of the original word x^3 and combine that with A^3 since A^3 has more information about X^2 than X^3 ?</p>

CC BY-SA 4.0
2023-12-28T01:07:42.010Z
Please, edit your post to format the formulas with mathjax.
nbro
CC BY-SA 4.0
2023-12-28T12:25:06.807Z
Also, please, put your specific question in the title. Thanks.
nbro
<neural-networks><transformer><attention>
Do I need residual block in a transformer model if vanishing gradients don't exist?
Stef
# Do I need residual block in a transformer model if vanishing gradients don't exist?

(Asked by: username_0 on 2023-12-27)

[![Self Attention Diagram](https://i.sstatic.net/jzvCW.png)](https://i.sstatic.net/jzvCW.png)

In this example l’Afrique is x^3 and the attention is being computed for this word A^3(l’Afrique). In the image above Andrew Ng indicates that the word with the biggest wieght in the computation of A^3 will be visite not l’Afrique itself because visite gives more context to l’Afrique. So when all the V values are summed A^3 will contatin more of visite embedding than l’Afrique embedding. So my question is this theoretically if vanishing gradients did not exist would a residual block still be required for a transformer network to work because it would need to maintain the information of the original word x^3 and combine that with A^3 since A^3 has more information about X^2 than X^3 ?

**Tags:** neural-networks, transformer, attention

> **Comments:**
> **username_1**: Please, edit your post to format the formulas with mathjax.  
> **username_1**: Also, please, put your specific question in the title. Thanks.  

---

## 0 answers

