Answers.Body: <p>Answer to <strong>Q1)</strong> <em>If sampling for next token do you need to apply mask during inference?</em>  </p>
  
  <p>Yes you do! The models ability to transfer information across positions was trained in this manner, and changing it up will have unpredictable consequences. Let my try to give an example:  </p>
  
  <p><strong>Tokens:</strong> 1:sally, 2:sold, 3:seashells, 4:on, 5:the, 6:____<br>
  In the above you are trying to predict 6 from {1:5}  </p>
  
  <p>Denote <span class="math-container">$n^{(m)}$</span> as the set of tokens the <span class="math-container">$n^{th}$</span> positional embedding has info from at the <span class="math-container">$m^{th}$</span> layer.  </p>
  
  <p>In both cases we see that <span class="math-container">$n^{(0)} = \{n\} \ \  \forall n$</span>. Now though with a mask we get <span class="math-container">$n^{(i)} = \{k\}_{k\leq n} \ \  \forall n \ \ s.t. \ \ i \geq 1$</span> but without we see <span class="math-container">$n^{(i)} = \{k\}_{k \in [1:N]} \ \  \forall n$</span>. This difference means at the final layer the mebeddings going in will differ completely, and unless we train for such an approach it will cause  error  </p>
  
  <p>Answer to <strong>Q2)</strong> <em>What is the <strong>sample</strong> dimension?</em>  </p>
  
  <p>It took me a couple reads to understand what youre asking for but I think I understand. The sample at each step is drawn from a distribution where its logits are linearly associated to a single embedding of dimension <span class="math-container">$d_{(model)}$</span> therefore that is our upper bound: <span class="math-container">$dim(sample) \leq d_{(model)}$</span> which in the example you gave is 768.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-16T09:35:20.627Z
Answers.Text: Hi, about your example, i think you are describing how mask works but not why it's needed, right? as said in the question, in inference mode, there are no future tokens, so there is no tokens for k>n, and so k \in [1:N] is the same as k <= n.  it seems no need mask in inference. is there anything wrong?
Answers.UserDisplayName: dingx
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-18T11:33:06.607Z
Answers.Text: Hi @mshlis, regarding **Q1** have you experimented with and without causal masking during inference to see if the model really performs bad without? Also if you know of any papers that discuss this topic, then links or references would be greatly appreciated.
Answers.UserDisplayName: pi-tau
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-14T14:40:26.407Z
Answers.LastActivityDate: 2019-11-14T14:40:26.407Z
Answers.UserDisplayName: mshlis
Answers.Body: <p>Yes, <strong>it is still needed</strong>. In fact, @mshlis has given a clear reason in the Answer to Q1. As I am new to the community and have no reputation to directly comment, let me supply his/her answer by opening up my answer.</p>
  <h3>Empirical Verification</h3>
  <p>First, in response to @pi-tau, yes, the model performs <strong>really worse without</strong> causal masking than with causal masking during inference according to my observations. For example, assuming you are using Python, one can comment out the related lines in the used package (typically located in a directory with a path like <code>anaconda3/lib/python3.7/site-packages/transformers/models/gpt2/modeling_gpt2.py</code> and similar to the ones shown below, whose exact forms depend on your own environment and torch version) to experiment without causal masking.</p>
  <pre><code>if not self.is_cross_attention:
      # if only &quot;normal&quot; attention layer implements causal mask
      query_length, key_length = query.size(-2), key.size(-2)
      causal_mask = self.bias[:, :, key_length - query_length : key_length, :key_length].bool()
      attn_weights = torch.where(causal_mask, attn_weights, self.masked_bias.to(attn_weights.dtype))
  </code></pre>
  <p><em>One may also refer to the official implementation <a href="https://github.com/huggingface/transformers/blob/8211c59b9a8fe84d2861446b26542f89a0260e64/src/transformers/models/gpt2/modeling_gpt2.py#L194" rel="nofollow noreferrer">here</a></em>.</p>
  <h3>Intuition</h3>
  <p>Second, regarding the confusion of @dingx, yes, there was something wrong. Let me explain it with <strong>a concrete example</strong> while reusing the notation introduced by @mshlis's answer. We will look into the case where the GPT-2 now takes 5 tokens (''sally sold seashells on the'') as input to predict the 6-th token (assuming each word is a token).</p>
  <p>Without causal masking, for each layer, the information that each embedding has can be listed as follows:</p>
  <ul>
  <li>1st layer: <span class="math-container">$1^{(1)} = \{1^{(0)}, 2^{(0)}, 3^{(0)}, 4^{(0)}, 5^{(0)}\}$</span>, <span class="math-container">$2^{(1)} = \{1^{(0)}, 2^{(0)}, 3^{(0)}, 4^{(0)}, 5^{(0)}\}$</span>, ..., <span class="math-container">$5^{(1)} = \{1^{(0)}, 2^{(0)}, 3^{(0)}, 4^{(0)}, 5^{(0)}\}$</span></li>
  <li>2nd layer: <span class="math-container">$1^{(2)} = \{1^{(1)}, 2^{(1)}, 3^{(1)}, 4^{(1)}, 5^{(1)}\}$</span>, <span class="math-container">$2^{(2)} = \{1^{(1)}, 2^{(1)}, 3^{(1)}, 4^{(1)}, 5^{(1)}\}$</span>, ..., <span class="math-container">$5^{(2)} = \{1^{(1)}, 2^{(1)}, 3^{(1)}, 4^{(1)}, 5^{(1)}\}$</span></li>
  <li>...</li>
  <li>6th layer: <span class="math-container">$1^{(6)} = \{1^{(5)}, 2^{(5)}, 3^{(5)}, 4^{(5)}, 5^{(5)}\}$</span>, <span class="math-container">$2^{(6)} = \{1^{(5)}, 2^{(5)}, 3^{(5)}, 4^{(5)}, 5^{(5)}\}$</span>, ..., <span class="math-container">$5^{(6)} = \{1^{(5)}, 2^{(5)}, 3^{(5)}, 4^{(5)}, 5^{(5)}\}$</span></li>
  <li>Predicting the 6-th token uses</li>
  </ul>
  <p><span class="math-container">\begin{align*}
  5^{(6)} &amp;= \{1^{(5)}, 2^{(5)}, 3^{(5)}, 4^{(5)}, 5^{(5)}\} \\
  &amp;= \{
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\}
  \} \\
  &amp;= \dots
  \end{align*}</span></p>
  <p>With causal masking, things are quite different:</p>
  <ul>
  <li>1st layer: <span class="math-container">$1^{(1)} = \{1\}$</span>, <span class="math-container">$2^{(1)} = \{1, 2\}$</span>, ..., <span class="math-container">$5^{(1)} = \{1, 2, 3, 4, 5\}$</span></li>
  <li>2nd layer: <span class="math-container">$1^{(2)} = \{1\}$</span>, <span class="math-container">$2^{(2)} = \{1, 2\}$</span>, ..., <span class="math-container">$5^{(1)} = \{1, 2, 3, 4, 5\}$</span></li>
  <li>...</li>
  <li>6th layer: <span class="math-container">$1^{(6)} = \{1\}$</span>, <span class="math-container">$2^{(6)} = \{1, 2\}$</span>, ..., <span class="math-container">$5^{(6)} = \{1, 2, 3, 4, 5\}$</span></li>
  <li>Predicting the 6-th token uses</li>
  </ul>
  <p><span class="math-container">\begin{align*}
  5^{(6)} &amp;= \{1^{(5)}, 2^{(5)}, 3^{(5)}, 4^{(5)}, 5^{(5)}\} \\
  &amp;= \{
  \{1^{(4)}\},
  \{1^{(4)}, 2^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\}
  \} \\
  &amp;= \dots
  \end{align*}</span></p>
  <p>While both extensions are not complete, one should first be clear about (1) that there indeed <strong>exists a difference</strong> in the prediction result between the two methods, and (2) that this difference <strong>has nothing to do with</strong> whether the GPT-2 model can see the so-called &quot;future tokens&quot; that have not been generated (i.e., <span class="math-container">$6^{(m)}$</span> has never appeared in the above equations for any <span class="math-container">$m$</span>).</p>
  <p>But why the result inferred with causal masking is preferred over that without? It is because that way is <strong>exactly how the shipped version of GPT-2 is pre-trained</strong>. In other words, it is more comfortable to predict the 6th token with</p>
  <p><span class="math-container">$\{
  \{1^{(4)}\},
  \{1^{(4)}, 2^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\}
  \},$</span></p>
  <p>instead of</p>
  <p><span class="math-container">$\{
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\}
  \}.$</span></p>
  <p>Finally, one might wonder why the GPT2 has to be trained in this way. It is about efficiency. For prediction in the former way, all the <span class="math-container">$1^{(4)}$</span>'s, <span class="math-container">$2^{(4)}$</span>'s, <span class="math-container">$3^{(4)}$</span>'s, and <span class="math-container">$4^{(4)}$</span>'s are identical to what were generated in previous iterations and thus can be cached and reused. However, this is not the case for the latter way. To see that, note that all the <span class="math-container">$1^{(4)}$</span>'s, <span class="math-container">$2^{(4)}$</span>'s, <span class="math-container">$3^{(4)}$</span>'s, and <span class="math-container">$4^{(4)}$</span>'s in the latter way contain information about the 5th token, which are not possible to compute in previous iterations.</p>
  <blockquote>
  <p>In my understanding, causal masking is indeed for hiding future
  tokens. However, &quot;future tokens&quot; do not narrowly refer to what has not
  been generated yet. Instead, for a particular token, they refer to all
  the succeeding tokens <strong>including those that have already been
  generated</strong> and not.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-31T06:44:22.317Z
Answers.LastActivityDate: 2023-10-31T06:44:22.317Z
Answers.UserDisplayName: Zhifeng Jiang
Answers.Body: <p>Following the above comment by @Zhifeng , I don't get why are you assuming that the causal masking will make the first token not see the prebious prompt tokens? How will it generate then?</p>
  <p>And if it's not the prompt token then those tokens have already been generated, i.e the querry will still see them, that's how decoder only model was trained on the first place. I don't get it why you have only put 1 token in layer 1.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-06T18:08:33.870Z
Answers.LastActivityDate: 2024-11-06T18:08:33.870Z
Answers.UserDisplayName: Anshuman Sinha
Body: <p>My understanding is that masked self-attention is necessary during training of GPT-2, as otherwise it would be able to directly see the correct next output at each iteration. My question is whether the attention mask is necessary, or even possible, during inference. As GPT-2 will only be producing one token at a time, it doesn't make sense to mask out future tokens that haven't been inferred yet.</p>
  
Tags: <natural-language-processing><attention><transformer><gpt><inference>
Title: Is the Mask Needed for Masked Self-Attention During Inference with GPT-2
UserDisplayName: D_s
ThreadText: # Is the Mask Needed for Masked Self-Attention During Inference with GPT-2
  
  (Asked by: username_0 on 2019-11-14)
  
  My understanding is that masked self-attention is necessary during training of GPT-2, as otherwise it would be able to directly see the correct next output at each iteration. My question is whether the attention mask is necessary, or even possible, during inference. As GPT-2 will only be producing one token at a time, it doesn't make sense to mask out future tokens that haven't been inferred yet.
  
  **Tags:** natural-language-processing, attention, transformer, gpt, inference
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2019-11-14. Score: 3
  
  Answer to **Q1)** *If sampling for next token do you need to apply mask during inference?*
  
  Yes you do! The models ability to transfer information across positions was trained in this manner, and changing it up will have unpredictable consequences. Let my try to give an example:
  
  **Tokens:** 1:sally, 2:sold, 3:seashells, 4:on, 5:the, 6:\_\_\_\_  
  In the above you are trying to predict 6 from {1:5}
  
  Denote $n^{(m)}$ as the set of tokens the $n^{th}$ positional embedding has info from at the $m^{th}$ layer.
  
  In both cases we see that $n^{(0)} = \{n\} \ \ \forall n$. Now though with a mask we get $n^{(i)} = \{k\}\_{k\leq n} \ \ \forall n \ \ s.t. \ \ i \geq 1$ but without we see $n^{(i)} = \{k\}\_{k \in [1:N]} \ \ \forall n$. This difference means at the final layer the mebeddings going in will differ completely, and unless we train for such an approach it will cause error
  
  Answer to **Q2)** *What is the **sample** dimension?*
  
  It took me a couple reads to understand what youre asking for but I think I understand. The sample at each step is drawn from a distribution where its logits are linearly associated to a single embedding of dimension $d\_{(model)}$ therefore that is our upper bound: $dim(sample) \leq d\_{(model)}$ which in the example you gave is 768.
  
  > **Comments:**
  > **username_2**: Hi, about your example, i think you are describing how mask works but not why it's needed, right? as said in the question, in inference mode, there are no future tokens, so there is no tokens for k>n, and so k \in [1:N] is the same as k <= n.  it seems no need mask in inference. is there anything wrong?  
  > **username_3**: Hi @mshlis, regarding **Q1** have you experimented with and without causal masking during inference to see if the model really performs bad without? Also if you know of any papers that discuss this topic, then links or references would be greatly appreciated.  
  
  ---
  
  ### Answer by username_4 on 2023-10-31. Score: 1
  
  Yes, it is still needed. In fact, @mshlis has given a clear reason in the Answer to Q1. As I am new to the community and have no reputation to directly comment, let me supply his/her answer by opening up my answer.
  Empirical Verification
  First, in response to @pi-tau, yes, the model performs really worse without causal masking than with causal masking during inference according to my observations. For example, assuming you are using Python, one can comment out the related lines in the used package (typically located in a directory with a path like anaconda3/lib/python3.7/site-packages/transformers/models/gpt2/modeling_gpt2.py and similar to the ones shown below, whose exact forms depend on your own environment and torch version) to experiment without causal masking.
  if not self.is_cross_attention:
      # if only "normal" attention layer implements causal mask
      query_length, key_length = query.size(-2), key.size(-2)
      causal_mask = self.bias[:, :, key_length - query_length : key_length, :key_length].bool()
      attn_weights = torch.where(causal_mask, attn_weights, self.masked_bias.to(attn_weights.dtype))
  
  One may also refer to the official implementation here.
  Intuition
  Second, regarding the confusion of @dingx, yes, there was something wrong. Let me explain it with a concrete example while reusing the notation introduced by @mshlis's answer. We will look into the case where the GPT-2 now takes 5 tokens (''sally sold seashells on the'') as input to predict the 6-th token (assuming each word is a token).
  Without causal masking, for each layer, the information that each embedding has can be listed as follows:
  
  1st layer: $1^{(1)} = \{1^{(0)}, 2^{(0)}, 3^{(0)}, 4^{(0)}, 5^{(0)}\}$, $2^{(1)} = \{1^{(0)}, 2^{(0)}, 3^{(0)}, 4^{(0)}, 5^{(0)}\}$, ..., $5^{(1)} = \{1^{(0)}, 2^{(0)}, 3^{(0)}, 4^{(0)}, 5^{(0)}\}$
  2nd layer: $1^{(2)} = \{1^{(1)}, 2^{(1)}, 3^{(1)}, 4^{(1)}, 5^{(1)}\}$, $2^{(2)} = \{1^{(1)}, 2^{(1)}, 3^{(1)}, 4^{(1)}, 5^{(1)}\}$, ..., $5^{(2)} = \{1^{(1)}, 2^{(1)}, 3^{(1)}, 4^{(1)}, 5^{(1)}\}$
  ...
  6th layer: $1^{(6)} = \{1^{(5)}, 2^{(5)}, 3^{(5)}, 4^{(5)}, 5^{(5)}\}$, $2^{(6)} = \{1^{(5)}, 2^{(5)}, 3^{(5)}, 4^{(5)}, 5^{(5)}\}$, ..., $5^{(6)} = \{1^{(5)}, 2^{(5)}, 3^{(5)}, 4^{(5)}, 5^{(5)}\}$
  Predicting the 6-th token uses
  
  \begin{align*}
  5^{(6)} &= \{1^{(5)}, 2^{(5)}, 3^{(5)}, 4^{(5)}, 5^{(5)}\} \\
  &= \{
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\}
  \} \\
  &= \dots
  \end{align*}
  With causal masking, things are quite different:
  
  1st layer: $1^{(1)} = \{1\}$, $2^{(1)} = \{1, 2\}$, ..., $5^{(1)} = \{1, 2, 3, 4, 5\}$
  2nd layer: $1^{(2)} = \{1\}$, $2^{(2)} = \{1, 2\}$, ..., $5^{(1)} = \{1, 2, 3, 4, 5\}$
  ...
  6th layer: $1^{(6)} = \{1\}$, $2^{(6)} = \{1, 2\}$, ..., $5^{(6)} = \{1, 2, 3, 4, 5\}$
  Predicting the 6-th token uses
  
  \begin{align*}
  5^{(6)} &= \{1^{(5)}, 2^{(5)}, 3^{(5)}, 4^{(5)}, 5^{(5)}\} \\
  &= \{
  \{1^{(4)}\},
  \{1^{(4)}, 2^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\}
  \} \\
  &= \dots
  \end{align*}
  While both extensions are not complete, one should first be clear about (1) that there indeed exists a difference in the prediction result between the two methods, and (2) that this difference has nothing to do with whether the GPT-2 model can see the so-called "future tokens" that have not been generated (i.e., $6^{(m)}$ has never appeared in the above equations for any $m$).
  But why the result inferred with causal masking is preferred over that without? It is because that way is exactly how the shipped version of GPT-2 is pre-trained. In other words, it is more comfortable to predict the 6th token with
  $\{
  \{1^{(4)}\},
  \{1^{(4)}, 2^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\}
  \},$
  instead of
  $\{
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\},
  \{1^{(4)}, 2^{(4)}, 3^{(4)}, 4^{(4)}, 5^{(4)}\}
  \}.$
  Finally, one might wonder why the GPT2 has to be trained in this way. It is about efficiency. For prediction in the former way, all the $1^{(4)}$'s, $2^{(4)}$'s, $3^{(4)}$'s, and $4^{(4)}$'s are identical to what were generated in previous iterations and thus can be cached and reused. However, this is not the case for the latter way. To see that, note that all the $1^{(4)}$'s, $2^{(4)}$'s, $3^{(4)}$'s, and $4^{(4)}$'s in the latter way contain information about the 5th token, which are not possible to compute in previous iterations.
  
  In my understanding, causal masking is indeed for hiding future
  tokens. However, "future tokens" do not narrowly refer to what has not
  been generated yet. Instead, for a particular token, they refer to all
  the succeeding tokens including those that have already been
  generated and not.
  
  
  ---
  
  ### Answer by username_5 on 2024-11-06. Score: 0
  
  Following the above comment by @Zhifeng , I don't get why are you assuming that the causal masking will make the first token not see the prebious prompt tokens? How will it generate then?
  
  And if it's not the prompt token then those tokens have already been generated, i.e the querry will still see them, that's how decoder only model was trained on the first place. I don't get it why you have only put 1 token in layer 1.
  
  
  ---
  
  