Answers.Body: <p>The attention module can be seen in many way. Indeed, you can just see as a way to construct a set of weights on the fly (in poor words this is what it does):
  <span class="math-container">$$
  A = softmax(QK/\sqrt{d})\\
  output = A\cdot V
  $$</span>
  Indeed, you can see it as a way to aggregate length-independent sequence (a dense layer does not have this property as it requires a fixed size input).<br />
  Such mechanisms can be found in literature as &quot;hyper networks&quot; (you use a network to estimate the weights of another network)</p>
  <p>Nothing stops you from using a different activation function... whoever you will loose the &quot;interpretation&quot; of that matrix being an attention/distribution</p>
  <p>What you gain instead is scalability. Indeed, the problem of the <span class="math-container">$O(n^2)$</span> complexity comes from the fact that you need the normalization of the softmax. However if you use another activation, as ReLU, or even linear, you can go back to the <span class="math-container">$O(n)$</span> beloved complexity</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-07T18:22:29.427Z
Answers.LastActivityDate: 2024-05-07T18:22:29.427Z
Answers.UserDisplayName: Alberto
Body: <p>As far as I understand, the attention model in a LLM is used to mitigate the vanishing gradient problem.</p>
  <p>When using activation functions like the sigmoid function, deep neural networks may lead to gradients that are very close to zero (because you are multiplying tails that are close to zero), which in a way reduces the usefulness of these models.</p>
  <p>However, couldn't you simply use a different activation function like ReLU to avoid this issue?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-05T16:44:01.553Z
Comments.Text: nobody stops you, but you loose the interpretation of attention (not anymore a distribution)… check “linear transformers”
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-06T19:40:40.850Z
Comments.Text: @Alberto feel free to post an answer
Comments.UserDisplayName: A. Darwin
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-07T12:16:10.173Z
Comments.Text: Attention layer in *Attention is All You Need*'s Transformer uses softmax, not the sigmoid function. Do you have something else in mind?
Comments.UserDisplayName: pcpthm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-07T17:05:13.720Z
Comments.Text: @pcpthm I would say that with "like the sigmoid function" he meant "in that family of functions"
Comments.UserDisplayName: Alberto
Tags: <attention><large-language-models><vanishing-gradient-problem>
Title: Can attention models be replaced by non-sigmoid activation functions?
UserDisplayName: A. Darwin
ThreadText: # Can attention models be replaced by non-sigmoid activation functions?
  
  (Asked by: username_0 on 2024-05-05)
  
  As far as I understand, the attention model in a LLM is used to mitigate the vanishing gradient problem.
  
  When using activation functions like the sigmoid function, deep neural networks may lead to gradients that are very close to zero (because you are multiplying tails that are close to zero), which in a way reduces the usefulness of these models.
  
  However, couldn't you simply use a different activation function like ReLU to avoid this issue?
  
  **Tags:** attention, large-language-models, vanishing-gradient-problem
  > **Comments:**
  > **username_1**: nobody stops you, but you loose the interpretation of attention (not anymore a distribution)… check “linear transformers”  
  > **username_0**: @Alberto feel free to post an answer  
  > **username_2**: Attention layer in *Attention is All You Need*'s Transformer uses softmax, not the sigmoid function. Do you have something else in mind?  
  > **username_1**: @pcpthm I would say that with "like the sigmoid function" he meant "in that family of functions"  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-05-07. Score: 0
  
  The attention module can be seen in many way. Indeed, you can just see as a way to construct a set of weights on the fly (in poor words this is what it does):
  $$
  A = softmax(QK/\sqrt{d})\\
  output = A\cdot V
  $$
  Indeed, you can see it as a way to aggregate length-independent sequence (a dense layer does not have this property as it requires a fixed size input).  
  Such mechanisms can be found in literature as "hyper networks" (you use a network to estimate the weights of another network)
  
  Nothing stops you from using a different activation function... whoever you will loose the "interpretation" of that matrix being an attention/distribution
  
  What you gain instead is scalability. Indeed, the problem of the $O(n^2)$ complexity comes from the fact that you need the normalization of the softmax. However if you use another activation, as ReLU, or even linear, you can go back to the $O(n)$ beloved complexity
  
  
  ---
  
  