Answers.Body: <p>Entries of an attention mask are typically either <span class="math-container">$0$</span> or <span class="math-container">$-\infty$</span>.</p>
  <p>So, adding such a mask gives either the original entry of <span class="math-container">$QK^T$</span> or <span class="math-container">$-\infty$</span>.</p>
  <p>The issue with entrywise multiplication with a binary matrix, is that <span class="math-container">$0$</span> values still contribute to softmax.</p>
  <p><span class="math-container">$$softmax(z)_i = \frac{e^{z_i}}{\sum_k e^{z_k}}$$</span></p>
  <p><span class="math-container">$e^0$</span> is <span class="math-container">$1$</span>, so element-wise product before softmax is not really a mask. After softmax doesn't work either, since the output is no longer a probability distribution. To ensure the masked elements do not contribute at all, you need them to be <span class="math-container">$-\infty$</span>, which mask addition does.</p>
  <p>If you know that certain indices of the input of are padded, that is, they are all zero vectors, then the padding tokens are already ignored by matmul (since they are zero). The issue is the rows of <span class="math-container">$QK^T$</span> corresponding to the pad tokens are zero and not <span class="math-container">$-\infty$</span>.</p>
  <p>To ignore pad tokens, you would create a mask with <span class="math-container">$-\infty$</span> along columns corresponding to the location of padding, and add this to <span class="math-container">$QK^T$</span> before softmax. So it's just the same attention formula, different value of <span class="math-container">$M$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-20T19:36:17.373Z
Answers.Text: Thank you for your answer. You're totally right about the multiplication didn't give it much thought. But I have a question about this type of masking for the padding tokens. If we put $-\infty$ in rows corresponding to the pad tokens, the softmax will take in rows that are all $-\infty$ and will then output a row of nan values. I reckon that we can turn those nan values into $0$ but I wonder if it is how it's being done in deep learning frameworks like PyTorch etc.
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-20T19:55:28.297Z
Answers.Text: @Daviiid In the case of TensorFlow, they mask for infinite values (floats with full exponent and empty mantissa) in implementing exp. The other frameworks probably do the same thing.
Answers.UserDisplayName: Venna Banana
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-20T20:04:57.003Z
Answers.Text: thank you for your answer. Can I ask you if that is equivalent mathematically to replacing those rows where we should have nan values, in a normal implementation, with 0? I'm trying to understand the effect of having a row full of $-\infty$ (the rows corresponding to padding tokens) on the multiplication of the values $V$, by the softmax, since having such rows will lead in a "normal" setting to an undefined value output from the softmax because of $0/0$
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-20T21:10:39.583Z
Answers.Text: Well, you will only see a $0$ in the denominator if all entries are $-\infty$, not just some rows. That is, your entire sequence would have to be padding. In this case, the framework will return some NaN symbol, and the network evaluation will not work. I think the formula I used for softmax in the answer could be clearer. Edited.
Answers.UserDisplayName: Venna Banana
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-20T21:27:00.613Z
Answers.Text: Yeah you're right. Well, we do apply the softmax per row right? And we put $-\infty$ in the rows corresponding to pad token positions right? So we get a whole row of $-\infty$ that's fed to a softmax per row. Maybe this last part is wrong. If $Q$ is of shape (bsz, seq len, d_m) and the same for $K$, we get $QK^{T}$ of shape (bsz, seq len, seq len), will the padding mask, of shape (seq len, seq len) have $-\infty$ column-wise? E.g. if all sentences have a pad token at the end then will the last column of the padding mask be $-\infty$, or, will the last row be all $-\infty$?
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-20T21:36:17.853Z
Answers.Text: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/147362/discussion-between-venna-banana-and-daviiid).
Answers.UserDisplayName: Venna Banana
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-20T18:58:53.313Z
Answers.LastActivityDate: 2023-07-20T22:11:59.037Z
Answers.LastEditDate: 2023-07-20T22:11:59.037Z
Answers.UserDisplayName: Venna Banana
Answers.Body: <p>For a given sequence <code>x</code> you calculate the attention scores using the formula:</p>
  <p><span class="math-container">\begin{equation}
  A = \frac{(xQ K^Tx^T)}{\sqrt{d_k}},
  \end{equation}</span>
  where <span class="math-container">$Q, K$</span> are the query and key matrices of the attention layer.</p>
  <p>The result is a square matrix of size <span class="math-container">$T \times T$</span> where <span class="math-container">$T$</span> is the length of the sequence <span class="math-container">$x$</span>. The entry <span class="math-container">$A_{i,j}$</span> gives the attention score between <span class="math-container">$x_{i}$</span> and <span class="math-container">$x_{j}$</span> (note <span class="math-container">$A_{i,j} \neq A_{j,i}$</span>). So basically row <span class="math-container">$i$</span> gives you the attention scores for token <span class="math-container">$x_i$</span>, which other tokens of the sequence it should attend to. However, you actually want to use these scores to perform a weighted average over the value encodings given by <span class="math-container">$xV$</span> (<span class="math-container">$V$</span> is the value matrix of the attention layer). But the scores can be arbitrary real numbers, and you want to have positive weights that sum to <span class="math-container">$1.$</span> That is why we apply a <code>softmax</code> layer to convert the scores into attention weights.</p>
  <p>Now, if your sequence contains pad tokens you don't want <span class="math-container">$x_i$</span> to attend to them. So you want to &quot;remove&quot; the attention between <span class="math-container">$x_i$</span> and the pad tokens. You could set the attention weights directly to <span class="math-container">$0.$</span>, but then the sum will not be <span class="math-container">$1.$</span> You actually want to mask the attention before applying the softmax function. Thus, you set the attention scores between <span class="math-container">$x_i$</span> and the pad tokens to a large negative number. Now applying the softmax will produce <span class="math-container">$0.$</span> attention weights. You can set the scores to <code>-float(&quot;inf&quot;)</code>, but I think setting them to <span class="math-container">$-1e9$</span> is more than enough.</p>
  <p>For a concrete example see this <a href="https://github.com/pi-tau/transformer/blob/3d4bbc055dd695eccd7ca97d6646d6a81a239cb9/multihead_attn.py#L121" rel="nofollow noreferrer">github repo</a>. Also here is an extensive <a href="https://pi-tau.github.io/posts/transformer/" rel="nofollow noreferrer">blog post</a> that I wrote about the Transformer, you might like it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-25T12:25:54.360Z
Answers.Text: Sorry for the late reply. Thank you for your answer!
Answers.UserDisplayName: Daviiid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-23T15:24:03.403Z
Answers.LastActivityDate: 2023-07-23T15:24:03.403Z
Answers.UserDisplayName: pi-tau
Body: <p>I have been looking for the answer in other questions but no one tackled that. I want to ask you how is the padding mask considered in the formula of attention?</p>
  <p>The attention formula taking into account a causal mask is:
  <span class="math-container">$Attention(Q, K, V) = softmax(\frac{QK^{T} + CausalMask}{\sqrt{d_{k}}})V$</span></p>
  <p>But how do we add the padding mask? The aim of a padding mask is to mask the padding positions as they're used just to make the batching feasible. But I don't know how this mask is added in the Attention formula.</p>
  <p>Does it make sense if we do element-wise multiplication of the Attention matrix with a tensor of ones of shape (batch size, sequence length, <span class="math-container">$d_{model}$</span>) and whatever sentence <span class="math-container">$s$</span> in that batch and wherever position <span class="math-container">$p$</span> is a padding token then the tensor[s, p, :] is zeros?</p>
  <p>Thank you in advance for your help!</p>
  
Tags: <natural-language-processing><transformer><attention><sequence-modeling><padding>
Title: How is the padding mask incorporated in the attention formula?
UserDisplayName: Daviiid
ThreadText: # How is the padding mask incorporated in the attention formula?
  
  (Asked by: username_0 on 2023-07-19)
  
  I have been looking for the answer in other questions but no one tackled that. I want to ask you how is the padding mask considered in the formula of attention?
  
  The attention formula taking into account a causal mask is:
  $Attention(Q, K, V) = softmax(\frac{QK^{T} + CausalMask}{\sqrt{d\_{k}}})V$
  
  But how do we add the padding mask? The aim of a padding mask is to mask the padding positions as they're used just to make the batching feasible. But I don't know how this mask is added in the Attention formula.
  
  Does it make sense if we do element-wise multiplication of the Attention matrix with a tensor of ones of shape (batch size, sequence length, $d\_{model}$) and whatever sentence $s$ in that batch and wherever position $p$ is a padding token then the tensor[s, p, :] is zeros?
  
  Thank you in advance for your help!
  
  **Tags:** natural-language-processing, transformer, attention, sequence-modeling, padding
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-07-20. Score: 2 | Accepted answer
  
  Entries of an attention mask are typically either $0$ or $-\infty$.
  
  So, adding such a mask gives either the original entry of $QK^T$ or $-\infty$.
  
  The issue with entrywise multiplication with a binary matrix, is that $0$ values still contribute to softmax.
  
  $$softmax(z)\_i = \frac{e^{z\_i}}{\sum\_k e^{z\_k}}$$
  
  $e^0$ is $1$, so element-wise product before softmax is not really a mask. After softmax doesn't work either, since the output is no longer a probability distribution. To ensure the masked elements do not contribute at all, you need them to be $-\infty$, which mask addition does.
  
  If you know that certain indices of the input of are padded, that is, they are all zero vectors, then the padding tokens are already ignored by matmul (since they are zero). The issue is the rows of $QK^T$ corresponding to the pad tokens are zero and not $-\infty$.
  
  To ignore pad tokens, you would create a mask with $-\infty$ along columns corresponding to the location of padding, and add this to $QK^T$ before softmax. So it's just the same attention formula, different value of $M$.
  
  > **Comments:**
  > **username_0**: Thank you for your answer. You're totally right about the multiplication didn't give it much thought. But I have a question about this type of masking for the padding tokens. If we put $-\infty$ in rows corresponding to the pad tokens, the softmax will take in rows that are all $-\infty$ and will then output a row of nan values. I reckon that we can turn those nan values into $0$ but I wonder if it is how it's being done in deep learning frameworks like PyTorch etc.  
  > **username_1**: @username_0 In the case of TensorFlow, they mask for infinite values (floats with full exponent and empty mantissa) in implementing exp. The other frameworks probably do the same thing.  
  > **username_0**: thank you for your answer. Can I ask you if that is equivalent mathematically to replacing those rows where we should have nan values, in a normal implementation, with 0? I'm trying to understand the effect of having a row full of $-\infty$ (the rows corresponding to padding tokens) on the multiplication of the values $V$, by the softmax, since having such rows will lead in a "normal" setting to an undefined value output from the softmax because of $0/0$  
  > **username_1**: Well, you will only see a $0$ in the denominator if all entries are $-\infty$, not just some rows. That is, your entire sequence would have to be padding. In this case, the framework will return some NaN symbol, and the network evaluation will not work. I think the formula I used for softmax in the answer could be clearer. Edited.  
  > **username_0**: Yeah you're right. Well, we do apply the softmax per row right? And we put $-\infty$ in the rows corresponding to pad token positions right? So we get a whole row of $-\infty$ that's fed to a softmax per row. Maybe this last part is wrong. If $Q$ is of shape (bsz, seq len, d_m) and the same for $K$, we get $QK^{T}$ of shape (bsz, seq len, seq len), will the padding mask, of shape (seq len, seq len) have $-\infty$ column-wise? E.g. if all sentences have a pad token at the end then will the last column of the padding mask be $-\infty$, or, will the last row be all $-\infty$?  
  > **username_1**: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/147362/discussion-between-username_1-username_1-and-username_0).  
  
  ---
  
  ### Answer by username_2 on 2023-07-23. Score: 2
  
  For a given sequence `x` you calculate the attention scores using the formula:
  
  \begin{equation}
  A = \frac{(xQ K^Tx^T)}{\sqrt{d\_k}},
  \end{equation}
  where $Q, K$ are the query and key matrices of the attention layer.
  
  The result is a square matrix of size $T \times T$ where $T$ is the length of the sequence $x$. The entry $A\_{i,j}$ gives the attention score between $x\_{i}$ and $x\_{j}$ (note $A\_{i,j} \neq A\_{j,i}$). So basically row $i$ gives you the attention scores for token $x\_i$, which other tokens of the sequence it should attend to. However, you actually want to use these scores to perform a weighted average over the value encodings given by $xV$ ($V$ is the value matrix of the attention layer). But the scores can be arbitrary real numbers, and you want to have positive weights that sum to $1.$ That is why we apply a `softmax` layer to convert the scores into attention weights.
  
  Now, if your sequence contains pad tokens you don't want $x\_i$ to attend to them. So you want to "remove" the attention between $x\_i$ and the pad tokens. You could set the attention weights directly to $0.$, but then the sum will not be $1.$ You actually want to mask the attention before applying the softmax function. Thus, you set the attention scores between $x\_i$ and the pad tokens to a large negative number. Now applying the softmax will produce $0.$ attention weights. You can set the scores to `-float("inf")`, but I think setting them to $-1e9$ is more than enough.
  
  For a concrete example see this [github repo](https://github.com/username_2/transformer/blob/3d4bbc055dd695eccd7ca97d6646d6a81a239cb9/multihead_attn.py#L121). Also here is an extensive [blog post](https://username_2.github.io/posts/transformer/) that I wrote about the Transformer, you might like it.
  
  > **Comments:**
  > **username_0**: Sorry for the late reply. Thank you for your answer!  
  
  ---
  
  