Answers.Body: <p>This goes back to the purpose of self-attention. While the Vaswani et als Attention is All You Need paper does not explicitly define self-attention, I am using it here as being synonymous to what it refers to as scaled dot-product attention when all three inputs are identical.</p>
  <p>Measure between word-vectors is generally computed through cosine-similarity because in the dimensions word tokens exist, it's highly unlikely for two words to be colinear even if they are trained to be closer in value if they are similar. However, two trained tokens will have higher cosine-similarity if they are semantically closer to each other than two completely unrelated words.</p>
  <p>This fact is exploited by scaled dot-product attention; After several of these matrix multiplications, the dissimilar words will zero out or become negative due to the dot product between them, and the similar words will stand out in the resulting matrix.</p>
  <p>So, self attention can be viewed as a weighted average, where less similar words become averaged out faster (toward the zero vector, on average), thereby achieving groupings of important and unimportant words (i.e. attention). The weighting happens through the dot product. If input vectors were normalized, the weights would be exactly the cosine similarities.</p>
  <p>The important thing to take into consideration is that within the self-attention mechanism, there are no parameters; Those linear operations are just there to capture the relationship between the different vectors by using the properties of the vectors used to represent them.</p>
  <p>Read <a href="http://peterbloem.nl/blog/transformers" rel="nofollow noreferrer">this blog post by Peter Bloem</a> for a more in-depth explanation of self-attention.</p>
  <hr />
  <h2>Edit</h2>
  <p>I should add that this explanation is less satisfactory considering how Transformers also seem to work for tasks without learned embeddings, like time-series forecasting. I have no idea why that is. However, the model was originally used for NLP, and they did use learned embeddings. So, I bet that's why that particular architecture looks the way it does.</p>
  <p>Bloem, in the blog post above, does discuss the mathematical properties of self-attention without bringing up the fact that the original architecture does have learned embeddings.</p>
  <p>All this shows is that having learned embeddings does not matter that much; The layers following the multi-headed attention will learn the relationships between the vectors. The general point about the properties of the dot-product being exploited does stand.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-17T22:28:51.177Z
Answers.Text: It seems like people are applying transformers to things without these embeddings, as you say.  Without embeddings that have a priori meaningful relationships, all the attention stuff does not seem applicable.  Since there is no activation, the same transformation can be done with a single layer if I'm not mistaken.
Answers.UserDisplayName: Mastiff
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-21T16:49:00.713Z
Answers.Text: @Mastiff Well, if you want a more general answer, this is the best alternative I can come up with: 
  There are a number of papers, starting with Metaformer, that suggest that any "token mixer" may work. As a specific example see FNet which uses Fourier transforms instead.
  
  Per these papers, attention, FFTs and even convolutions and poolings are examples of token mixers. So, my explanation still holds but for a different, possibly deeper, reason.
  I never added this since it's still pretty much in early research stage. Do you feel like I should?
Answers.UserDisplayName: Avatrin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-05T10:09:18.660Z
Answers.Text: `there are no parameters`  -  When said **parameters** we mean something that is learnable by machine leaning, right? then [the Wq Wk Wv Wo](https://ai.stackexchange.com/q/30491/87742) that learned from training should be think of as parameters, maybe?
Answers.UserDisplayName: Wuaner
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-05T11:04:07.103Z
Answers.Text: @Wuaner I have noticed there is an issue of definition that leads to this confusion. There is a learned layer right before the matrix multiplications. If you consider the weights from that layer as a part of attention, then yes, the illustration you are linking to is correct. However, the Attention is All You Need paper does not includue those weights as a part of scaled dot-product attention, but it does include those in multi-head attention (look at figure 2 in that paper). My answer refers to self-attention. Vaswani is not clear how he defines that.
Answers.UserDisplayName: Avatrin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-05T11:32:19.100Z
Answers.Text: @Wuaner I have edited the answer to make that clear. Understanding dot products and how they have historically been used in NLP is essential to understand attention as a whole. So, my answer focuses on that.
Answers.UserDisplayName: Avatrin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-06T01:58:14.133Z
Answers.Text: @Avatrin I see, that make sense.
Answers.UserDisplayName: Wuaner
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-26T18:05:58.800Z
Answers.LastActivityDate: 2024-10-12T22:45:42.090Z
Answers.LastEditDate: 2024-10-12T22:45:42.090Z
Answers.UserDisplayName: Avatrin
Body: <p>I was hoping someone could explain to me why in the transformer model from the &quot;Attention is all you need&quot; paper there is no activation applied after both the multihead attention layer and to the residual connections.  It seems to me that there are multiple linear layers in a row, and I have always been under the impression that you should have an activation between linear layers.</p>
  <p>For instance when I look at the different flavors of resnet they always apply some sort of non linearity following a linear layer.  For instance a residual block might look something like...</p>
  <p>Input -&gt; Conv -&gt; BN -&gt; Relu -&gt; Conv -&gt; (+ Input) -&gt; BN -&gt; Relu</p>
  <p>or in the case of pre-activation...</p>
  <p>Input -&gt; BN -&gt; Relu -&gt; Conv -&gt; BN -&gt; Relu -&gt; Conv -&gt; (+ Input)</p>
  <p>In all the resnet flavors I have seen, they never allow two linear layers to be connected without a relu in-between.</p>
  <p>However in the the transformer...</p>
  <p>Input -&gt; Multihead-Attn -&gt; Add/Norm -&gt; Feed Forward(Dense Layer -&gt; Relu -&gt; Dense Layer) -&gt; Add/Norm</p>
  <p>In the multihead attention layer it performs the attention mechanism and then applies a fully connected layer to project back to the dimension of its input.  However, there is no non linearity between that and feed forward network (except for maybe the softmax used in part of the attention.)  A model like this would make more sense to me...</p>
  <p>Input -&gt; Multihead-Attn -&gt; Add/Norm -&gt; <strong>Relu</strong> -&gt; Feed Forward(Dense Layer -&gt; Relu -&gt; Dense Layer) -&gt; Add/Norm -&gt; <strong>Relu</strong></p>
  <p>or something like the pre-activated resnet...</p>
  <p>Input -&gt; Relu -&gt; Multihead-Attn -&gt; Add/Norm -&gt; Input2 -&gt; <strong>Relu</strong> -&gt; Feed Forward(Dense Layer -&gt; Relu -&gt; Dense Layer) -&gt; Add/Norm(Input2)</p>
  <p>Can anyone explain why the transformer is the way it is?</p>
  <p>I have asked a similar question when I was looking at the architecture of wavenet on another forum but I never really got a clear answer. In that case it did not make sense to me again why there was no activation applied to the residual connections.
  (<a href="https://www.reddit.com/r/MachineLearning/comments/njbjfb/d_is_there_a_point_to_having_layers_with_just_a/" rel="noreferrer">https://www.reddit.com/r/MachineLearning/comments/njbjfb/d_is_there_a_point_to_having_layers_with_just_a/</a>)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-06T17:18:14.640Z
Comments.Text: FWIW, the final operation of each attention head is a weighted sum of values where the weights are computed as a softmax. Softmax is non-linear, and its shape is sometimes thought of as a multidimensional sigmoid. In some sense, the softmax-output weights serve as a sort of activation function. So maybe it's not bizarre to follow it up with the 1x1 convolutions of the position-wise feed forward layer (with layer normalization and residual connections in between).
Comments.UserDisplayName: Alexander Guyer
Tags: <transformer><attention>
Title: Why does a transformer not use an activation function following the multi-head attention layer?
UserDisplayName: chasep255
ThreadText: # Why does a transformer not use an activation function following the multi-head attention layer?
  
  (Asked by: username_0 on 2021-08-24)
  
  I was hoping someone could explain to me why in the transformer model from the "Attention is all you need" paper there is no activation applied after both the multihead attention layer and to the residual connections. It seems to me that there are multiple linear layers in a row, and I have always been under the impression that you should have an activation between linear layers.
  
  For instance when I look at the different flavors of resnet they always apply some sort of non linearity following a linear layer. For instance a residual block might look something like...
  
  Input -> Conv -> BN -> Relu -> Conv -> (+ Input) -> BN -> Relu
  
  or in the case of pre-activation...
  
  Input -> BN -> Relu -> Conv -> BN -> Relu -> Conv -> (+ Input)
  
  In all the resnet flavors I have seen, they never allow two linear layers to be connected without a relu in-between.
  
  However in the the transformer...
  
  Input -> Multihead-Attn -> Add/Norm -> Feed Forward(Dense Layer -> Relu -> Dense Layer) -> Add/Norm
  
  In the multihead attention layer it performs the attention mechanism and then applies a fully connected layer to project back to the dimension of its input. However, there is no non linearity between that and feed forward network (except for maybe the softmax used in part of the attention.) A model like this would make more sense to me...
  
  Input -> Multihead-Attn -> Add/Norm -> **Relu** -> Feed Forward(Dense Layer -> Relu -> Dense Layer) -> Add/Norm -> **Relu**
  
  or something like the pre-activated resnet...
  
  Input -> Relu -> Multihead-Attn -> Add/Norm -> Input2 -> **Relu** -> Feed Forward(Dense Layer -> Relu -> Dense Layer) -> Add/Norm(Input2)
  
  Can anyone explain why the transformer is the way it is?
  
  I have asked a similar question when I was looking at the architecture of wavenet on another forum but I never really got a clear answer. In that case it did not make sense to me again why there was no activation applied to the residual connections.
  (<https://www.reddit.com/r/MachineLearning/comments/njbjfb/d_is_there_a_point_to_having_layers_with_just_a/>)
  
  **Tags:** transformer, attention
  
  > **Comments:**
  > **username_1**: FWIW, the final operation of each attention head is a weighted sum of values where the weights are computed as a softmax. Softmax is non-linear, and its shape is sometimes thought of as a multidimensional sigmoid. In some sense, the softmax-output weights serve as a sort of activation function. So maybe it's not bizarre to follow it up with the 1x1 convolutions of the position-wise feed forward layer (with layer normalization and residual connections in between).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-08-26. Score: 10 | Accepted answer
  
  This goes back to the purpose of self-attention. While the Vaswani et als Attention is All You Need paper does not explicitly define self-attention, I am using it here as being synonymous to what it refers to as scaled dot-product attention when all three inputs are identical.
  
  Measure between word-vectors is generally computed through cosine-similarity because in the dimensions word tokens exist, it's highly unlikely for two words to be colinear even if they are trained to be closer in value if they are similar. However, two trained tokens will have higher cosine-similarity if they are semantically closer to each other than two completely unrelated words.
  
  This fact is exploited by scaled dot-product attention; After several of these matrix multiplications, the dissimilar words will zero out or become negative due to the dot product between them, and the similar words will stand out in the resulting matrix.
  
  So, self attention can be viewed as a weighted average, where less similar words become averaged out faster (toward the zero vector, on average), thereby achieving groupings of important and unimportant words (i.e. attention). The weighting happens through the dot product. If input vectors were normalized, the weights would be exactly the cosine similarities.
  
  The important thing to take into consideration is that within the self-attention mechanism, there are no parameters; Those linear operations are just there to capture the relationship between the different vectors by using the properties of the vectors used to represent them.
  
  Read [this blog post by Peter Bloem](http://peterbloem.nl/blog/transformers) for a more in-depth explanation of self-attention.
  
  ---
  
  ## Edit
  
  I should add that this explanation is less satisfactory considering how Transformers also seem to work for tasks without learned embeddings, like time-series forecasting. I have no idea why that is. However, the model was originally used for NLP, and they did use learned embeddings. So, I bet that's why that particular architecture looks the way it does.
  
  Bloem, in the blog post above, does discuss the mathematical properties of self-attention without bringing up the fact that the original architecture does have learned embeddings.
  
  All this shows is that having learned embeddings does not matter that much; The layers following the multi-headed attention will learn the relationships between the vectors. The general point about the properties of the dot-product being exploited does stand.
  
  > **Comments:**
  > **username_3**: It seems like people are applying transformers to things without these embeddings, as you say.  Without embeddings that have a priori meaningful relationships, all the attention stuff does not seem applicable.  Since there is no activation, the same transformation can be done with a single layer if I'm not mistaken.  
  > **username_2**: @username_3 Well, if you want a more general answer, this is the best alternative I can come up with:  There are a number of papers, starting with Metaformer, that suggest that any "token mixer" may work. As a specific example see FNet which uses Fourier transforms instead.  Per these papers, attention, FFTs and even convolutions and poolings are examples of token mixers. So, my explanation still holds but for a different, possibly deeper, reason. I never added this since it's still pretty much in early research stage. Do you feel like I should?  
  > **username_4**: `there are no parameters`  -  When said **parameters** we mean something that is learnable by machine leaning, right? then [the Wq Wk Wv Wo](https://ai.stackexchange.com/q/30491/87742) that learned from training should be think of as parameters, maybe?  
  > **username_2**: @username_4 I have noticed there is an issue of definition that leads to this confusion. There is a learned layer right before the matrix multiplications. If you consider the weights from that layer as a part of attention, then yes, the illustration you are linking to is correct. However, the Attention is All You Need paper does not includue those weights as a part of scaled dot-product attention, but it does include those in multi-head attention (look at figure 2 in that paper). My answer refers to self-attention. Vaswani is not clear how he defines that.  
  > **username_2**: @username_4 I have edited the answer to make that clear. Understanding dot products and how they have historically been used in NLP is essential to understand attention as a whole. So, my answer focuses on that.  
  > **username_4**: @username_2 I see, that make sense.  
  
  ---
  
  