Answers.Body: <p>During training you use <strong>teacher forcing</strong>. This means that you feed the entire <span class="math-container">$tgt$</span> sequence to the decoder. You want the decoder to look only at <span class="math-container">$tgt[:i]$</span> when predicting <span class="math-container">$y[i]$</span>. You then use <span class="math-container">$y[i]$</span> only to compare it with <span class="math-container">$tgt[i+1]$</span> to compute the loss. Thus for computing the output at position <span class="math-container">$i$</span> you want all the future tokens to be masked, but not erased. So when you calculate the attention scores between <span class="math-container">$tgt[i]$</span> and all the rest you want to zero the attention scores with the future tokens.</p>
  <p>The attention scores is a square matrix of shape <span class="math-container">$l_s \times l_s$</span>, where <span class="math-container">$l_s$</span> is the length of the target sequence. The entry <span class="math-container">$A_{i,j}$</span> gives the attention score between <span class="math-container">$tgt[i]$</span> and <span class="math-container">$tgt[j]$</span>. Basically, row <span class="math-container">$i$</span> gives you the attention scores for token <span class="math-container">$tgt[i]$</span>. So you want to zero out the scores when <span class="math-container">$i&gt;j$</span>. That is the reason why we use a triangular mask - simply zero out the upper triangular part of the attention scores matrix.</p>
  <p>If you want to read a <strong>very detailed</strong> blog post about the Transformer model in general that I wrote, feel free to check this out:<br />
  <a href="https://pi-tau.github.io/posts/transformer" rel="nofollow noreferrer">https://pi-tau.github.io/posts/transformer</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-28T08:38:13.567Z
Answers.LastActivityDate: 2023-09-18T10:18:17.897Z
Answers.LastEditDate: 2023-09-18T10:18:17.897Z
Answers.UserDisplayName: pi-tau
Body: <p>I have some confusion about the implementation of the triangle mask in the transformer decoder. I understand the reasoning for the mask, it prevents the network from 'cheating' by looking ahead at the next token, but the way it's implemented seems strange. First of all, why do we apply the mask to the product <span class="math-container">$QK^T$</span>? Why not just set the rows representing unseen tokens to zero or some special token before sending the matrix to the transformer? Also, I don't understand the reasoning of a triangle mask in general. Suppose we are training with a sequence of length <span class="math-container">$\ell_s$</span>, and we are trying to predict token <span class="math-container">$k$</span> with <span class="math-container">$k &lt; \ell_s$</span>. Why don't we have to mask all entries <span class="math-container">$QK^T_{i,j}$</span> if <span class="math-container">$i &gt; k$</span> or <span class="math-container">$j &gt; k$</span>? With a standard triangle mask it seems like we still incorporate information computed from 'unseen' tokens.</p>
  
Tags: <transformer><language-model>
Title: Confusion About Triangle Mask in Transformer Decoder
UserDisplayName: new2java
ThreadText: # Confusion About Triangle Mask in Transformer Decoder
  
  (Asked by: username_0 on 2023-07-27)
  
  I have some confusion about the implementation of the triangle mask in the transformer decoder. I understand the reasoning for the mask, it prevents the network from 'cheating' by looking ahead at the next token, but the way it's implemented seems strange. First of all, why do we apply the mask to the product $QK^T$? Why not just set the rows representing unseen tokens to zero or some special token before sending the matrix to the transformer? Also, I don't understand the reasoning of a triangle mask in general. Suppose we are training with a sequence of length $\ell\_s$, and we are trying to predict token $k$ with $k < \ell\_s$. Why don't we have to mask all entries $QK^T\_{i,j}$ if $i > k$ or $j > k$? With a standard triangle mask it seems like we still incorporate information computed from 'unseen' tokens.
  
  **Tags:** transformer, language-model
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-28. Score: 1
  
  During training you use **teacher forcing**. This means that you feed the entire $tgt$ sequence to the decoder. You want the decoder to look only at $tgt[:i]$ when predicting $y[i]$. You then use $y[i]$ only to compare it with $tgt[i+1]$ to compute the loss. Thus for computing the output at position $i$ you want all the future tokens to be masked, but not erased. So when you calculate the attention scores between $tgt[i]$ and all the rest you want to zero the attention scores with the future tokens.
  
  The attention scores is a square matrix of shape $l\_s \times l\_s$, where $l\_s$ is the length of the target sequence. The entry $A\_{i,j}$ gives the attention score between $tgt[i]$ and $tgt[j]$. Basically, row $i$ gives you the attention scores for token $tgt[i]$. So you want to zero out the scores when $i>j$. That is the reason why we use a triangular mask - simply zero out the upper triangular part of the attention scores matrix.
  
  If you want to read a **very detailed** blog post about the Transformer model in general that I wrote, feel free to check this out:  
  <https://username_1.github.io/posts/transformer>
  
  
  ---
  
  