<p>Because what attention does is to control how much of the information in <span class="math-container">$V$</span> to use based on weights computed through the similarity between <span class="math-container">$Q$</span> and <span class="math-container">$K$</span>.</p>
<p>When we multiply the attention weights by <span class="math-container">$V$</span>, we are doing a weighed sum of the vectors in <span class="math-container">$V$</span> to get a new matrix that better represents <span class="math-container">$Q$</span> contextually within <span class="math-container">$V$</span>. There is no need for variance normalization since its the resulting representation of the attention layer itself, rather than something that we use as a probability distribution. This is in opposition to the product <span class="math-container">$QK^T$</span>, to which we then apply a softmax to obtain dimension-wise pseudo-probabilities.</p>

CC BY-SA 4.0
2023-04-30T08:43:23.470Z
2023-04-30T08:43:23.470Z
Areg Sarvazyan
<p>As this <a href="https://ai.stackexchange.com/q/21237/23811">question</a> says:</p>
<blockquote>
<p>In scaled dot product attention, we scale our outputs by dividing the
dot product by the square root of the dimensionality of the matrix:</p>
<p><a href="https://i.sstatic.net/wLI4m.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/wLI4m.png" alt="enter image description here" /></a></p>
<p>The reason why is stated that this constrains the distribution of the weights of the output to have a standard deviation of 1.</p>
</blockquote>
<p>My question is why don't we do the same after multiplying to <span class="math-container">$V$</span>(values) for the same reason?</p>

<neural-networks><natural-language-processing><transformer><attention><normalisation>
In the attention mechanism, why don't we normalize after multiplying values?
Peyman
# In the attention mechanism, why don't we normalize after multiplying values?

(Asked by: username_0 on 2023-04-29)

As this [question](https://ai.stackexchange.com/q/21237/23811) says:

> In scaled dot product attention, we scale our outputs by dividing the
> dot product by the square root of the dimensionality of the matrix:
>
> [![enter image description here](https://i.sstatic.net/wLI4m.png)](https://i.sstatic.net/wLI4m.png)
>
> The reason why is stated that this constrains the distribution of the weights of the output to have a standard deviation of 1.

My question is why don't we do the same after multiplying to $V$(values) for the same reason?

**Tags:** neural-networks, natural-language-processing, transformer, attention, normalisation


---

## 1 answers

### Answer by username_1 on 2023-04-30. Score: 2

Because what attention does is to control how much of the information in $V$ to use based on weights computed through the similarity between $Q$ and $K$.

When we multiply the attention weights by $V$, we are doing a weighed sum of the vectors in $V$ to get a new matrix that better represents $Q$ contextually within $V$. There is no need for variance normalization since its the resulting representation of the attention layer itself, rather than something that we use as a probability distribution. This is in opposition to the product $QK^T$, to which we then apply a softmax to obtain dimension-wise pseudo-probabilities.


---

