Answers.Body: <p>Softmax was first used for its properties, as it is differentiable, has not domain problem even though it has a division, and it's gradient is well behaved (and in conjunction with Categorical Cross Entropy it gets simplified a lot becoming linear)</p>
  <p>In the case of attention, many of such properties do not hold, and it's used merely for its ability to emulate a categorical distribution. However, there is not specific reason to use it instead of some other formulation, for example also the following activation will be a valid one:
  <span class="math-container">$$
  \frac{p_i^2}{\sum_j p_j^2}
  $$</span>
  where the <span class="math-container">$p_i$</span> are the row logits from <span class="math-container">$QK^T$</span>. However, even though this formulation is valid, as the sum adds up to 1 and all probabilities are positive, it's much more unstable, as the denominator might get to 0, or close to 0, thus introducing instability during learning</p>
  <p>At the end of the day, any non parametric activation that maps a vector <span class="math-container">$v\in R^n$</span> to a vector <span class="math-container">$u$</span> such that <span class="math-container">$||u||_1=1$</span> and <span class="math-container">$u_i \ge 0\forall i$</span> is indeed a valid &quot;attention&quot; activation</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-07T23:24:26.620Z
Answers.Text: Part of my question is whether the attention mechanism has to generate outputs that sum to 1, creating a valid probability distribution. Would lifting this condition change anything other than the ease of interpretation from our point of view?
Answers.UserDisplayName: Kasia
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-08T10:39:39.170Z
Answers.Text: @Kasia yes, if you relax that assumption (debatable calling it attention then though) you can even solve the $O(n^2)$ problem of transformers, which is what indeed linear transformer do, and it's even used in the paper of _"RWKV: Reinventing RNNs for the Transformer Era"_, since they exploit that using linear attention, you don't need to know the future tokens attention, and thus you can decay to a sort of RNN with additive recurrent layer
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-28T00:12:07.140Z
Answers.Text: Just wanted to add a quite relevant and recent paper on the topic: https://arxiv.org/abs/2410.18613
Answers.UserDisplayName: cjferes
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-07T22:49:39.077Z
Answers.LastActivityDate: 2023-12-07T22:49:39.077Z
Answers.UserDisplayName: Alberto
Body: <p>I’m curious about the mathematical reasoning behind the use of the softmax function as the activation function in self-attention mechanisms within neural networks. Specifically, I’m interested in understanding if there is a theoretical basis that necessitates the use of softmax over other activation functions.</p>
  <p>Softmax is commonly employed to convert raw attention scores into a probability distribution, ensuring that the sum of attention weights equals 1. This normalization allows the model to effectively focus on certain parts of the input sequence. However, I wonder if there are alternative activation functions that could be less constraining and still allow the optimization process to determine the best way to allocate attention, similar to how tanh or other activations work in different layers of a neural network.</p>
  <ol>
  <li>Is there a mathematical justification for the necessity of softmax in self-attention mechanisms?</li>
  <li>Could other activation functions, perhaps with fewer constraints, be used effectively in place of softmax, allowing the optimization process more flexibility?</li>
  </ol>
  <p>Any insights or references to relevant literature would be greatly appreciated.</p>
  
Tags: <deep-learning><transformer><attention><softmax>
Title: Is Softmax Necessary as the Activation Function for Self-Attention Mechanisms?
UserDisplayName: Kasia
ThreadText: # Is Softmax Necessary as the Activation Function for Self-Attention Mechanisms?
  
  (Asked by: username_0 on 2023-12-07)
  
  I’m curious about the mathematical reasoning behind the use of the softmax function as the activation function in self-attention mechanisms within neural networks. Specifically, I’m interested in understanding if there is a theoretical basis that necessitates the use of softmax over other activation functions.
  
  Softmax is commonly employed to convert raw attention scores into a probability distribution, ensuring that the sum of attention weights equals 1. This normalization allows the model to effectively focus on certain parts of the input sequence. However, I wonder if there are alternative activation functions that could be less constraining and still allow the optimization process to determine the best way to allocate attention, similar to how tanh or other activations work in different layers of a neural network.
  
  1. Is there a mathematical justification for the necessity of softmax in self-attention mechanisms?
  2. Could other activation functions, perhaps with fewer constraints, be used effectively in place of softmax, allowing the optimization process more flexibility?
  
  Any insights or references to relevant literature would be greatly appreciated.
  
  **Tags:** deep-learning, transformer, attention, softmax
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-12-07. Score: 0
  
  Softmax was first used for its properties, as it is differentiable, has not domain problem even though it has a division, and it's gradient is well behaved (and in conjunction with Categorical Cross Entropy it gets simplified a lot becoming linear)
  
  In the case of attention, many of such properties do not hold, and it's used merely for its ability to emulate a categorical distribution. However, there is not specific reason to use it instead of some other formulation, for example also the following activation will be a valid one:
  $$
  \frac{p\_i^2}{\sum\_j p\_j^2}
  $$
  where the $p\_i$ are the row logits from $QK^T$. However, even though this formulation is valid, as the sum adds up to 1 and all probabilities are positive, it's much more unstable, as the denominator might get to 0, or close to 0, thus introducing instability during learning
  
  At the end of the day, any non parametric activation that maps a vector $v\in R^n$ to a vector $u$ such that $||u||\_1=1$ and $u\_i \ge 0\forall i$ is indeed a valid "attention" activation
  
  > **Comments:**
  > **username_0**: Part of my question is whether the attention mechanism has to generate outputs that sum to 1, creating a valid probability distribution. Would lifting this condition change anything other than the ease of interpretation from our point of view?  
  > **username_1**: @username_0 yes, if you relax that assumption (debatable calling it attention then though) you can even solve the $O(n^2)$ problem of transformers, which is what indeed linear transformer do, and it's even used in the paper of _"RWKV: Reinventing RNNs for the Transformer Era"_, since they exploit that using linear attention, you don't need to know the future tokens attention, and thus you can decay to a sort of RNN with additive recurrent layer  
  > **username_2**: Just wanted to add a quite relevant and recent paper on the topic: https://arxiv.org/abs/2410.18613  
  
  ---
  
  