Answers.Body: <p>Transformer networks are great, because they can handle variable length inputs, but they also have limitations, concerning the input size. For example BERT (a transformer based language model) only accepts <span class="math-container">$N = 512$</span> input tokens at most. They way transformer models can accept sequences shorter than the <span class="math-container">$N$</span>, is by padding the input sequence with zeros and masking. When inputting a sequence with length <span class="math-container">$n \leq N$</span>, the mask <span class="math-container">$m$</span> is a vector of size <span class="math-container">$N$</span>, where <span class="math-container">$m_i = (i \leq n)$</span>. Say we have a transformer that should translate from English to German which accepts <span class="math-container">$N = 8$</span> tokens at most, and we input the sentence &quot;Today is Tuesday&quot;. Let the words have the following token values:</p>
  <p><span class="math-container">\begin{array} {|r|r|}
  \hline \text{[SOS]} &amp; 1 \\ 
  \hline \text{Today} &amp; 274  \\ 
  \hline \text{is} &amp; 12 \\ 
  \hline \text{Tuesday} &amp; 125 \\ 
  \hline \text{[EOS]} &amp; 2 \\ 
  \hline  \end{array}</span></p>
  <p>The first input will always be occupied by the [SOS] token and each sequence is followed by an [EOS] token.
  Now the input looks like this:</p>
  <p><span class="math-container">\begin{bmatrix}1\\274\\12\\125\\2\\0\\0\\0\end{bmatrix}</span></p>
  <p>And you also supply an attention mask (using integers representing booleans) which looks like this:</p>
  <p><span class="math-container">\begin{bmatrix}1\\1\\1\\1\\1\\0\\0\\0\end{bmatrix}</span></p>
  <p>This means, each position can only attend to positions <span class="math-container">$i$</span> where <span class="math-container">$m_i = 1$</span>. Therefore, positions where the mask is 0, do not affect the output of the model.</p>
  <p>The vector representation of the mask above is the simplified version of the actual attention mask, describing that position at row <span class="math-container">$i$</span> can attend to positions at column <span class="math-container">$j$</span>:</p>
  <p><span class="math-container">\begin{bmatrix}1&amp;1&amp;1&amp;1&amp;1&amp;0&amp;0&amp;0\\1&amp;1&amp;1&amp;1&amp;1&amp;0&amp;0&amp;0\\1&amp;1&amp;1&amp;1&amp;1&amp;0&amp;0&amp;0\\1&amp;1&amp;1&amp;1&amp;1&amp;0&amp;0&amp;0\\1&amp;1&amp;1&amp;1&amp;1&amp;0&amp;0&amp;0\\0&amp;0&amp;0&amp;0&amp;0&amp;0&amp;0&amp;0\\0&amp;0&amp;0&amp;0&amp;0&amp;0&amp;0&amp;0\\0&amp;0&amp;0&amp;0&amp;0&amp;0&amp;0&amp;0\end{bmatrix}</span></p>
  <p>The decoder works a bit differently during training and inference.</p>
  <p><strong>Training:</strong>
  The decoder attention mask is built, so that each position can not attend to future positions. You can describe this by using a matrix:</p>
  <p><span class="math-container">\begin{bmatrix}1&amp;0&amp;0&amp;0&amp;0&amp;0&amp;0&amp;0\\1&amp;1&amp;0&amp;0&amp;0&amp;0&amp;0&amp;0\\1&amp;1&amp;1&amp;0&amp;0&amp;0&amp;0&amp;0\\1&amp;1&amp;1&amp;1&amp;0&amp;0&amp;0&amp;0\\1&amp;1&amp;1&amp;1&amp;1&amp;0&amp;0&amp;0\\1&amp;1&amp;1&amp;1&amp;1&amp;1&amp;0&amp;0\\1&amp;1&amp;1&amp;1&amp;1&amp;1&amp;1&amp;0\\1&amp;1&amp;1&amp;1&amp;1&amp;1&amp;1&amp;1\end{bmatrix}</span></p>
  <p>When using this mask for the decoder, position <span class="math-container">$i$</span> can only attend to positions <span class="math-container">$j, j \leq i$</span>.
  For the task of translation from English to German (e.g. &quot;Today is Tuesday&quot; -&gt; &quot;Heute ist Dienstag&quot;), you can provide the model with the entire input sentence and the entire output sentence. The model will make a prediction for each decoder input position, what the next word will be. And by using the mask, no information leaks into earlier positions (when predicting the next token for the word &quot;ist&quot;, the model can only see &quot;[SOS] Heute ist&quot;).</p>
  <p><strong>Inference:</strong>
  For the inference stage, you would first provide the input sentence for the encoder and for the decoder only the [SOS] token. Now you sample a word from the transformer output (e.g. picking the most likely or sample from the output distribution) at the position of your last decoder input (in this case the position of the [SOS] token, e.g. position 0). Then you do the next inference step, by again providing the input sentence for the encoder, and for the decoder you provide the [SOS] token + the predicted word from the last step. If the previous prediction was &quot;Heute&quot;, your decoder input now is &quot;[SOS] Heute&quot;. You just repeat this process, each time providing the so far predicted sentence to the decoder, until the transformer predicts [EOS].</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-08T23:08:13.053Z
Answers.Text: thanks so much this cleared out a lot of questions i had. One last question, does this mean that for the first step in inference you also input a (n_vocab, n_model) dimensional matrix to the decoder with a an attention mask with just a 1 at the first posiiton (for the first [SOS] token) and then for each step you expand the mask?
Answers.UserDisplayName: Paul Müller
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-09T01:27:05.897Z
Answers.Text: For inference you also just use the lower left triangular matrix as mask, just like in the training stage. This mask makes sure each prediction can't use future positions and due to the nature of it, it "expands" automatically. You could think that it would make more sense if e.g. pos 2 could also attend to pos 4 after predicting the first 5 words, but as during training this was not possible, changing this for the inference stage would not make sense. I think this is also a slight downside to transformers, as information in the decoder flows unidirectional. Glad I was able to help :)
Answers.UserDisplayName: Evator
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-09T08:24:38.307Z
Answers.Text: Thanks so much, I wish I could upvote, I'll come back to this as soon as I get 15 reputation ^^
Answers.UserDisplayName: Paul Müller
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-23T11:59:20.983Z
Answers.Text: Would it be possible to generate an entire German sentence in one go, in multiple passes, by repeatedly feeding back the previous iteration of the sentence into the transformer, with a square/rectangular mask? Would that be useful? Then, the transformer would not be constrained to generate the output one word at a time.
Answers.UserDisplayName: Criticize SE actions means ban
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-24T13:03:38.427Z
Answers.Text: I think what you mean would be equivalent to doing one word at a time/ that is what you are doing during the inference stage of a transformer. If I understand you right you just want to embed this process into the network architecture?
Answers.UserDisplayName: Evator
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-08T15:06:02.087Z
Answers.LastActivityDate: 2022-03-08T15:43:25.300Z
Answers.LastEditDate: 2022-03-08T15:43:25.300Z
Answers.UserDisplayName: Evator
Body: <p>I'm currently studying the Transformer model (<strong>Attention is all you need</strong>) and after reading it I still have some questions for which I get conflicting answers if I google them:</p>
  <ul>
  <li>What exactly are the dimensions of the input to the encoder of a transformer, from what I've seen you can input sentences with dynamic lengths but in the paper it seems like all layers expect the K/Q/V matrices to have dimensions of <code>d_model, d_k</code> or <code>d_v/d_q</code></li>
  <li>Same question for the decoder, what exactly are the input dimensions and how do the attention layers handle dynamic dimensions if its possible</li>
  <li>Another question is is masking only used for training where you input the whole sentence as the input to the decoder or does it have another purpose than that (especially in inference)</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-08T16:02:36.890Z
Comments.Text: [**Ask only one question per post**, even if you have multiple questions and they are related to the same topic!](https://meta.stackexchange.com/a/39224). So, edit your post to leave only one specific question, and put it in the title. "Dimensions of a Transformer model and purpose of masking" is not a question and it's also quite vague.
Comments.UserDisplayName: nbro
Tags: <neural-networks><machine-learning><natural-language-processing><transformer>
Title: Dimensions of a Transformer model and purpose of masking
UserDisplayName: Paul Müller
ThreadText: # Dimensions of a Transformer model and purpose of masking
  
  (Asked by: username_0 on 2022-03-08)
  
  I'm currently studying the Transformer model (**Attention is all you need**) and after reading it I still have some questions for which I get conflicting answers if I google them:
  
  * What exactly are the dimensions of the input to the encoder of a transformer, from what I've seen you can input sentences with dynamic lengths but in the paper it seems like all layers expect the K/Q/V matrices to have dimensions of `d_model, d_k` or `d_v/d_q`
  * Same question for the decoder, what exactly are the input dimensions and how do the attention layers handle dynamic dimensions if its possible
  * Another question is is masking only used for training where you input the whole sentence as the input to the decoder or does it have another purpose than that (especially in inference)
  
  **Tags:** neural-networks, machine-learning, natural-language-processing, transformer
  > **Comments:**
  > **username_1**: [**Ask only one question per post**, even if you have multiple questions and they are related to the same topic!](https://meta.stackexchange.com/a/39224). So, edit your post to leave only one specific question, and put it in the title. "Dimensions of a Transformer model and purpose of masking" is not a question and it's also quite vague.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-03-08. Score: 1 | Accepted answer
  
  Transformer networks are great, because they can handle variable length inputs, but they also have limitations, concerning the input size. For example BERT (a transformer based language model) only accepts $N = 512$ input tokens at most. They way transformer models can accept sequences shorter than the $N$, is by padding the input sequence with zeros and masking. When inputting a sequence with length $n \leq N$, the mask $m$ is a vector of size $N$, where $m\_i = (i \leq n)$. Say we have a transformer that should translate from English to German which accepts $N = 8$ tokens at most, and we input the sentence "Today is Tuesday". Let the words have the following token values:
  
  \begin{array} {|r|r|}
  \hline \text{[SOS]} & 1 \\
  \hline \text{Today} & 274 \\
  \hline \text{is} & 12 \\
  \hline \text{Tuesday} & 125 \\
  \hline \text{[EOS]} & 2 \\
  \hline \end{array}
  
  The first input will always be occupied by the [SOS] token and each sequence is followed by an [EOS] token.
  Now the input looks like this:
  
  \begin{bmatrix}1\\274\\12\\125\\2\\0\\0\\0\end{bmatrix}
  
  And you also supply an attention mask (using integers representing booleans) which looks like this:
  
  \begin{bmatrix}1\\1\\1\\1\\1\\0\\0\\0\end{bmatrix}
  
  This means, each position can only attend to positions $i$ where $m\_i = 1$. Therefore, positions where the mask is 0, do not affect the output of the model.
  
  The vector representation of the mask above is the simplified version of the actual attention mask, describing that position at row $i$ can attend to positions at column $j$:
  
  \begin{bmatrix}1&1&1&1&1&0&0&0\\1&1&1&1&1&0&0&0\\1&1&1&1&1&0&0&0\\1&1&1&1&1&0&0&0\\1&1&1&1&1&0&0&0\\0&0&0&0&0&0&0&0\\0&0&0&0&0&0&0&0\\0&0&0&0&0&0&0&0\end{bmatrix}
  
  The decoder works a bit differently during training and inference.
  
  **Training:**
  The decoder attention mask is built, so that each position can not attend to future positions. You can describe this by using a matrix:
  
  \begin{bmatrix}1&0&0&0&0&0&0&0\\1&1&0&0&0&0&0&0\\1&1&1&0&0&0&0&0\\1&1&1&1&0&0&0&0\\1&1&1&1&1&0&0&0\\1&1&1&1&1&1&0&0\\1&1&1&1&1&1&1&0\\1&1&1&1&1&1&1&1\end{bmatrix}
  
  When using this mask for the decoder, position $i$ can only attend to positions $j, j \leq i$.
  For the task of translation from English to German (e.g. "Today is Tuesday" -> "Heute ist Dienstag"), you can provide the model with the entire input sentence and the entire output sentence. The model will make a prediction for each decoder input position, what the next word will be. And by using the mask, no information leaks into earlier positions (when predicting the next token for the word "ist", the model can only see "[SOS] Heute ist").
  
  **Inference:**
  For the inference stage, you would first provide the input sentence for the encoder and for the decoder only the [SOS] token. Now you sample a word from the transformer output (e.g. picking the most likely or sample from the output distribution) at the position of your last decoder input (in this case the position of the [SOS] token, e.g. position 0). Then you do the next inference step, by again providing the input sentence for the encoder, and for the decoder you provide the [SOS] token + the predicted word from the last step. If the previous prediction was "Heute", your decoder input now is "[SOS] Heute". You just repeat this process, each time providing the so far predicted sentence to the decoder, until the transformer predicts [EOS].
  
  > **Comments:**
  > **username_0**: thanks so much this cleared out a lot of questions i had. One last question, does this mean that for the first step in inference you also input a (n_vocab, n_model) dimensional matrix to the decoder with a an attention mask with just a 1 at the first posiiton (for the first [SOS] token) and then for each step you expand the mask?  
  > **username_2**: For inference you also just use the lower left triangular matrix as mask, just like in the training stage. This mask makes sure each prediction can't use future positions and due to the nature of it, it "expands" automatically. You could think that it would make more sense if e.g. pos 2 could also attend to pos 4 after predicting the first 5 words, but as during training this was not possible, changing this for the inference stage would not make sense. I think this is also a slight downside to transformers, as information in the decoder flows unidirectional. Glad I was able to help :)  
  > **username_0**: Thanks so much, I wish I could upvote, I'll come back to this as soon as I get 15 reputation ^^  
  > **username_3**: Would it be possible to generate an entire German sentence in one go, in multiple passes, by repeatedly feeding back the previous iteration of the sentence into the transformer, with a square/rectangular mask? Would that be useful? Then, the transformer would not be constrained to generate the output one word at a time.  
  > **username_2**: I think what you mean would be equivalent to doing one word at a time/ that is what you are doing during the inference stage of a transformer. If I understand you right you just want to embed this process into the network architecture?  
  
  ---
  
  