Answers.Body: <p>I don't see why you should use a causal mask during inference.</p>
  <p>The whole point is to &quot;avoid cheating&quot; when you already have the answer (during training), so that you can parallelize the training without having to do it step by step</p>
  <p>During inference, there is obviously no concept of &quot;cheating&quot;, as you generate a token, you append it to the already generated sentence, and you then feed this new sequence to the decoder to generate the next token</p>
  <p>In my opinion, the only point for that causal masking, is either for code reuse, so you have to maintain a single transformer decoder code, or for something like padding if you want to do batched inference (not too sure about this in the link you provided)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-18T10:04:18.933Z
Answers.Text: Hi @Alberto, thanks for your answer. This is exactly what I was thinking: no point of masking because there can be no cheating! Regarding code reuse: looking at the implementation from Harvard you can simply pass a mask tensor of all ones. Regarding batched inference: I believe that all sequences in the batch are unrolled together so there is no need to pad anything during inference, but at the end you have to slice shorter outputs. I really couldn't find any reason for having the mask, but yet these "reference" implementations that I linked do use it.
Answers.UserDisplayName: pi-tau
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-18T10:43:03.877Z
Answers.Text: @pi-tau wrt the batched, I was thinking about the padding, as causal mask includes also the padding mask... wrt to the code, yes, you are right...
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-18T09:53:35.343Z
Answers.LastActivityDate: 2023-09-18T09:53:35.343Z
Answers.UserDisplayName: Alberto
Body: <p>I understand how causal masking in the self-attention layer of the decoder works and why we use it <strong>during training</strong>. What I want to ask is: <em>should we use causal masking during inference</em> ?</p>
  <p>Consider a machine translation task where you need to translate the sentence <br />
  <code>[&quot;I&quot;, &quot;am&quot;, &quot;going&quot;, &quot;to&quot;, &quot;the&quot;, &quot;cinema&quot;]</code> <br />
  from english into german. During inference the encoder encodes the input sentence and the decoder starts generating the output sentence token by token. Let's say the following is generated until now:<br />
  <code>[&quot;&lt;START&gt;&quot;, &quot;Ich&quot;, &quot;gehe&quot;, &quot;ins&quot;]</code> <br />
  and you have to generate the next token. What you need to do is forward the currently generated sequence through the decoder and it will output a probability distribution for the next token. The question is: <em><strong>Do we need to use causal masking here</strong></em>?</p>
  <p>Using the causal mask:
  <span class="math-container">$$
  \text{mask} = 
  \begin{pmatrix}
  1 &amp; 0 &amp; 0 &amp; 0 \\
  1 &amp; 1 &amp; 0 &amp; 0 \\
  1 &amp; 1 &amp; 1 &amp; 0 \\
  1 &amp; 1 &amp; 1 &amp; 1
  \end{pmatrix}
  $$</span>
  in the self-attention layer of the decoder would force each of the generated tokens to attend only to previous tokens. However, in my opinion, there is no need to use any masking here. The tokens that are already generated could simply attend to each other in order to better predict the next token.</p>
  <p>However, reference implementations that I have been looking at continue using the causal masking during inference. See for example:</p>
  <ul>
  <li>Annotated Transformer from Harvad<br />
  <a href="http://nlp.seas.harvard.edu/annotated-transformer/#greedy-decoding%5C" rel="nofollow noreferrer">http://nlp.seas.harvard.edu/annotated-transformer/#greedy-decoding\</a>
  Here they explicitly pass the causal mask as a parameter when decoding.</li>
  <li>Andrej Karpathy's minGPT<br />
  <a href="https://github.com/karpathy/minGPT/blob/37baab71b9abea1b76ab957409a1cc2fbfba8a26/mingpt/model.py#L283%5C" rel="nofollow noreferrer">https://github.com/karpathy/minGPT/blob/37baab71b9abea1b76ab957409a1cc2fbfba8a26/mingpt/model.py#L283\</a>
  Here the decoder is called as-is, thus the causal mask will not be deactivated, but will be applied.</li>
  </ul>
  <p>Is there a reason for using causal masking during inference?<br />
  Any thoughts on the matter would be appreciated. If you know of any research papers that discuss this topic or if you have seen somewhere an implementation that does not use casual masking during inference, please share a link.</p>
  
Tags: <transformer><decoder>
Title: Transformer decoder. Causal masking during inference?
UserDisplayName: pi-tau
ThreadText: # Transformer decoder. Causal masking during inference?
  
  (Asked by: username_0 on 2023-09-18)
  
  I understand how causal masking in the self-attention layer of the decoder works and why we use it **during training**. What I want to ask is: *should we use causal masking during inference* ?
  
  Consider a machine translation task where you need to translate the sentence   
  `["I", "am", "going", "to", "the", "cinema"]`   
  from english into german. During inference the encoder encodes the input sentence and the decoder starts generating the output sentence token by token. Let's say the following is generated until now:  
  `["<START>", "Ich", "gehe", "ins"]`   
  and you have to generate the next token. What you need to do is forward the currently generated sequence through the decoder and it will output a probability distribution for the next token. The question is: ***Do we need to use causal masking here***?
  
  Using the causal mask:
  $$
  \text{mask} =
  \begin{pmatrix}
  1 & 0 & 0 & 0 \\
  1 & 1 & 0 & 0 \\
  1 & 1 & 1 & 0 \\
  1 & 1 & 1 & 1
  \end{pmatrix}
  $$
  in the self-attention layer of the decoder would force each of the generated tokens to attend only to previous tokens. However, in my opinion, there is no need to use any masking here. The tokens that are already generated could simply attend to each other in order to better predict the next token.
  
  However, reference implementations that I have been looking at continue using the causal masking during inference. See for example:
  
  * Annotated Transformer from Harvad  
    [http://nlp.seas.harvard.edu/annotated-transformer/#greedy-decoding\](http://nlp.seas.harvard.edu/annotated-transformer/#greedy-decoding%5C)
    Here they explicitly pass the causal mask as a parameter when decoding.
  * Andrej Karpathy's minGPT  
    [https://github.com/karpathy/minGPT/blob/37baab71b9abea1b76ab957409a1cc2fbfba8a26/mingpt/model.py#L283\](https://github.com/karpathy/minGPT/blob/37baab71b9abea1b76ab957409a1cc2fbfba8a26/mingpt/model.py#L283%5C)
    Here the decoder is called as-is, thus the causal mask will not be deactivated, but will be applied.
  
  Is there a reason for using causal masking during inference?  
  Any thoughts on the matter would be appreciated. If you know of any research papers that discuss this topic or if you have seen somewhere an implementation that does not use casual masking during inference, please share a link.
  
  **Tags:** transformer, decoder
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-09-18. Score: 1
  
  I don't see why you should use a causal mask during inference.
  
  The whole point is to "avoid cheating" when you already have the answer (during training), so that you can parallelize the training without having to do it step by step
  
  During inference, there is obviously no concept of "cheating", as you generate a token, you append it to the already generated sentence, and you then feed this new sequence to the decoder to generate the next token
  
  In my opinion, the only point for that causal masking, is either for code reuse, so you have to maintain a single transformer decoder code, or for something like padding if you want to do batched inference (not too sure about this in the link you provided)
  
  > **Comments:**
  > **username_0**: Hi @username_1, thanks for your answer. This is exactly what I was thinking: no point of masking because there can be no cheating! Regarding code reuse: looking at the implementation from Harvard you can simply pass a mask tensor of all ones. Regarding batched inference: I believe that all sequences in the batch are unrolled together so there is no need to pad anything during inference, but at the end you have to slice shorter outputs. I really couldn't find any reason for having the mask, but yet these "reference" implementations that I linked do use it.  
  > **username_1**: @username_0 wrt the batched, I was thinking about the padding, as causal mask includes also the padding mask... wrt to the code, yes, you are right...  
  
  ---
  
  