Answers.Body: <blockquote>
  <p>It seems like doing this would lead to much-needed nonlinearity, otherwise, we're just doing linear transformations.</p>
  </blockquote>
  <p>Attention is broadly defined as a following operation (<span class="math-container">$\text{softmax}$</span> is sometimes replaced by <span class="math-container">$\tanh$</span>) :</p>
  <p><span class="math-container">$$\text{Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$</span></p>
  <p>Where <span class="math-container">$Q$</span>, <span class="math-container">$K$</span> and <span class="math-container">$V$</span> are matrices that are some functions of the inputs.<br />
  There are <em>three</em> nonlinear operations there:</p>
  <ol>
  <li>The inner projection <span class="math-container">$QK^T$</span> is nonlinear. We have multiplication of two functions of the inputs. For example, in case of self-attention <span class="math-container">$Q=X W_Q$</span> and <span class="math-container">$K = XW_K$</span> are two linear transforms of the same <span class="math-container">$X$</span>, so <span class="math-container">$QK^T = X \left(W_Q W_K^T\right) X^T$</span> is a quadratic function of the inputs.</li>
  <li>The <span class="math-container">$\text{softmax}(x_i) = e^{x_i} /\sum_n e^{x_n} $</span> function is obviously nonlinear (<span class="math-container">$\tanh$</span> as well)</li>
  <li>The final <span class="math-container">$\text{softmax}(\dots) V$</span> product is also nonlinear for the same reasons as (1)</li>
  </ol>
  <p>I would say that it is pretty clear that it is definitely not just a linear transformation - there's quite a lot of nonlinearities in the attention block.</p>
  <hr />
  <blockquote>
  <p>This observation applies to the transformer, additive attention, etc.</p>
  </blockquote>
  <p>Let's see what happens next with the outputs of the attention layers:</p>
  <p><strong>In the <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">transformer</a> model</strong>, outputs of the multi-head-self-attention are fed into a feed-forward network inside each block:</p>
  <p><a href="https://i.sstatic.net/yYZNW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/yYZNW.png" alt="Transformer figure1 cutout" /></a></p>
  <p>&quot;Feed-forward&quot; means that the inputs are multiplied by a weight matrix and then a nonlinear activation function is applied.</p>
  <p><strong>The <a href="https://arxiv.org/abs/1409.0473" rel="nofollow noreferrer">additive attention</a></strong> approach, directly applies another <span class="math-container">$\text{softmax}$</span> on the outputs of what one would call the attention block:</p>
  <p><span class="math-container">$$e_{ij} = v_a^T \tanh\left(W_as_{i-1} + U_a h_j\right)$$</span></p>
  <p><span class="math-container">$$\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_k \exp(e_{ik})}$$</span></p>
  <hr />
  <p>To summarize - I don't think that the premise of the question is correct. Various nonlinearities are present both inside the attention blocks and, typically, are applied after the attention is computed.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-11T23:51:37.293Z
Answers.Text: Self attention does not meat that $Q = K = X$. It only means that $Q=XW_Q$ and $K=XW_K$ or in other words that K and Q are obtained from the same X, as opposed to cross-attention where keys and queries come from different sequences.
Answers.UserDisplayName: hans
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-12T13:33:42.090Z
Answers.Text: @hans I stand corrected, thank you. Edited the answer to reflect that
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-06T15:32:36.313Z
Answers.LastActivityDate: 2023-03-12T13:33:11.777Z
Answers.LastEditDate: 2023-03-12T13:33:11.777Z
Answers.UserDisplayName: Kostya
Answers.Body: <p>here are a couple of reasons why nonlinear activation functions aren't typically used after projecting the query, key, and value vectors in attention mechanisms like those found in transformers:</p>
  <p>Redundancy: The attention mechanism itself already introduces non-linearity through the softmax function. Softmax takes the attention scores (which are linear dot products) and squashes them into probabilities, creating a non-linear relationship between the input and output.</p>
  <p>Later Layers Handle Non-Linearity:  The transformer architecture addresses the need for non-linearity in later stages.  Following the multi-head attention layer, there's a fully-connected feed-forward network (MLP) with one or more hidden layers. These hidden layers typically use ReLU or similar non-linear activation functions, allowing the network to learn complex relationships between the attention outputs and the final prediction.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-24T12:08:30.410Z
Answers.LastActivityDate: 2024-05-24T12:08:30.410Z
Answers.UserDisplayName: Emerson Sanchotene
Body: <p>Why don't people use nonlinear activation functions after projecting the query key value in attention?</p>
  
  <p>It seems like doing this would lead to much-needed nonlinearity, otherwise, we're just doing linear transformations.</p>
  
  <p>This observation applies to the transformer, additive attention, etc.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-03T07:21:41.257Z
Comments.Text: I'm not sure if I got your question right, for the attention model where exactly would you place the non-linearity? Looking at [Graph Attention Networks](https://arxiv.org/pdf/1710.10903.pdf) by Petar Velickovic, they do apply an activation function in eq. 5.
Comments.UserDisplayName: razvanc92
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-04T21:53:52.640Z
Comments.Text: Can you provide an example of someone not using nonlinear activations in their attention?
Comments.UserDisplayName: Philip Raeisghasem
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-23T08:13:40.353Z
Comments.Text: I think what he means is that the queries, keys and values are computed as linear projections, i.e. the input is simply multiplied by a matrix, `q = x * W_q`, `k = x * W_k` and `v = x * W_v` respectively. We could use a non-linear function on each of them, `q = σ(x * W_q)` etc., but it is redundant because later on we use the softmax function and at the end a MLP which also has non-linearities in it.
Comments.UserDisplayName: kuzand
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-25T20:23:10.597Z
Comments.Text: @AndreasK. Related. Why we apply these projections after the embedding layer which is linear? Aren't these two composed linearities redundant?
Comments.UserDisplayName: Antonios Sarikas
Tags: <neural-networks><attention><transformer>
Title: Why don't people use nonlinear activation functions after projecting the query key value in attention?
UserDisplayName: user3180
ThreadText: # Why don't people use nonlinear activation functions after projecting the query key value in attention?
  
  (Asked by: username_0 on 2019-05-03)
  
  Why don't people use nonlinear activation functions after projecting the query key value in attention?
  
  It seems like doing this would lead to much-needed nonlinearity, otherwise, we're just doing linear transformations.
  
  This observation applies to the transformer, additive attention, etc.
  
  **Tags:** neural-networks, attention, transformer
  
  > **Comments:**
  > **username_1**: I'm not sure if I got your question right, for the attention model where exactly would you place the non-linearity? Looking at [Graph Attention Networks](https://arxiv.org/pdf/1710.10903.pdf) by Petar Velickovic, they do apply an activation function in eq. 5.  
  > **username_2**: Can you provide an example of someone not using nonlinear activations in their attention?  
  > **username_3**: I think what he means is that the queries, keys and values are computed as linear projections, i.e. the input is simply multiplied by a matrix, `q = x * W_q`, `k = x * W_k` and `v = x * W_v` respectively. We could use a non-linear function on each of them, `q = σ(x * W_q)` etc., but it is redundant because later on we use the softmax function and at the end a MLP which also has non-linearities in it.  
  > **username_4**: @AndreasK. Related. Why we apply these projections after the embedding layer which is linear? Aren't these two composed linearities redundant?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_5 on 2023-03-06. Score: 5
  
  > It seems like doing this would lead to much-needed nonlinearity, otherwise, we're just doing linear transformations.
  
  Attention is broadly defined as a following operation ($\text{softmax}$ is sometimes replaced by $\tanh$) :
  
  $$\text{Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d\_k}}\right)V $$
  
  Where $Q$, $K$ and $V$ are matrices that are some functions of the inputs.  
  There are *three* nonlinear operations there:
  
  1. The inner projection $QK^T$ is nonlinear. We have multiplication of two functions of the inputs. For example, in case of self-attention $Q=X W\_Q$ and $K = XW\_K$ are two linear transforms of the same $X$, so $QK^T = X \left(W\_Q W\_K^T\right) X^T$ is a quadratic function of the inputs.
  2. The $\text{softmax}(x\_i) = e^{x\_i} /\sum\_n e^{x\_n} $ function is obviously nonlinear ($\tanh$ as well)
  3. The final $\text{softmax}(\dots) V$ product is also nonlinear for the same reasons as (1)
  
  I would say that it is pretty clear that it is definitely not just a linear transformation - there's quite a lot of nonlinearities in the attention block.
  
  ---
  
  > This observation applies to the transformer, additive attention, etc.
  
  Let's see what happens next with the outputs of the attention layers:
  
  **In the [transformer](https://arxiv.org/abs/1706.03762) model**, outputs of the multi-head-self-attention are fed into a feed-forward network inside each block:
  
  [![Transformer figure1 cutout](https://i.sstatic.net/yYZNW.png)](https://i.sstatic.net/yYZNW.png)
  
  "Feed-forward" means that the inputs are multiplied by a weight matrix and then a nonlinear activation function is applied.
  
  **The [additive attention](https://arxiv.org/abs/1409.0473)** approach, directly applies another $\text{softmax}$ on the outputs of what one would call the attention block:
  
  $$e\_{ij} = v\_a^T \tanh\left(W\_as\_{i-1} + U\_a h\_j\right)$$
  
  $$\alpha\_{ij} = \frac{\exp(e\_{ij})}{\sum\_k \exp(e\_{ik})}$$
  
  ---
  
  To summarize - I don't think that the premise of the question is correct. Various nonlinearities are present both inside the attention blocks and, typically, are applied after the attention is computed.
  
  > **Comments:**
  > **username_6**: Self attention does not meat that $Q = K = X$. It only means that $Q=XW_Q$ and $K=XW_K$ or in other words that K and Q are obtained from the same X, as opposed to cross-attention where keys and queries come from different sequences.  
  > **username_5**: @username_6 I stand corrected, thank you. Edited the answer to reflect that  
  
  ---
  
  ### Answer by username_7 on 2024-05-24. Score: 0
  
  here are a couple of reasons why nonlinear activation functions aren't typically used after projecting the query, key, and value vectors in attention mechanisms like those found in transformers:
  
  Redundancy: The attention mechanism itself already introduces non-linearity through the softmax function. Softmax takes the attention scores (which are linear dot products) and squashes them into probabilities, creating a non-linear relationship between the input and output.
  
  Later Layers Handle Non-Linearity: The transformer architecture addresses the need for non-linearity in later stages. Following the multi-head attention layer, there's a fully-connected feed-forward network (MLP) with one or more hidden layers. These hidden layers typically use ReLU or similar non-linear activation functions, allowing the network to learn complex relationships between the attention outputs and the final prediction.
  
  
  ---
  
  