Answers.Body: <p>Because what attention does is to control how much of the information in <span class="math-container">$V$</span> to use based on weights computed through the similarity between <span class="math-container">$Q$</span> and <span class="math-container">$K$</span>.</p>
  <p>When we multiply the attention weights by <span class="math-container">$V$</span>, we are doing a weighed sum of the vectors in <span class="math-container">$V$</span> to get a new matrix that better represents <span class="math-container">$Q$</span> contextually within <span class="math-container">$V$</span>. There is no need for variance normalization since its the resulting representation of the attention layer itself, rather than something that we use as a probability distribution. This is in opposition to the product <span class="math-container">$QK^T$</span>, to which we then apply a softmax to obtain dimension-wise pseudo-probabilities.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-30T08:43:23.470Z
Answers.LastActivityDate: 2023-04-30T08:43:23.470Z
Answers.UserDisplayName: Areg Sarvazyan
Body: <p>As this <a href="https://ai.stackexchange.com/q/21237/23811">question</a> says:</p>
  <blockquote>
  <p>In scaled dot product attention, we scale our outputs by dividing the
  dot product by the square root of the dimensionality of the matrix:</p>
  <p><a href="https://i.sstatic.net/wLI4m.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/wLI4m.png" alt="enter image description here" /></a></p>
  <p>The reason why is stated that this constrains the distribution of the weights of the output to have a standard deviation of 1.</p>
  </blockquote>
  <p>My question is why don't we do the same after multiplying to <span class="math-container">$V$</span>(values) for the same reason?</p>
  
Tags: <neural-networks><natural-language-processing><transformer><attention><normalisation>
Title: In the attention mechanism, why don't we normalize after multiplying values?
UserDisplayName: Peyman
ThreadText: # In the attention mechanism, why don't we normalize after multiplying values?
  
  (Asked by: username_0 on 2023-04-29)
  
  As this [question](https://ai.stackexchange.com/q/21237/23811) says:
  
  > In scaled dot product attention, we scale our outputs by dividing the
  > dot product by the square root of the dimensionality of the matrix:
  >
  > [![enter image description here](https://i.sstatic.net/wLI4m.png)](https://i.sstatic.net/wLI4m.png)
  >
  > The reason why is stated that this constrains the distribution of the weights of the output to have a standard deviation of 1.
  
  My question is why don't we do the same after multiplying to $V$(values) for the same reason?
  
  **Tags:** neural-networks, natural-language-processing, transformer, attention, normalisation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-04-30. Score: 2
  
  Because what attention does is to control how much of the information in $V$ to use based on weights computed through the similarity between $Q$ and $K$.
  
  When we multiply the attention weights by $V$, we are doing a weighed sum of the vectors in $V$ to get a new matrix that better represents $Q$ contextually within $V$. There is no need for variance normalization since its the resulting representation of the attention layer itself, rather than something that we use as a probability distribution. This is in opposition to the product $QK^T$, to which we then apply a softmax to obtain dimension-wise pseudo-probabilities.
  
  
  ---
  
  