Answers.Body: <p>In addition to the answer by Cesar Ruiz:</p>
  <p>In the Transformer model, padding tokens are used to make all sequences the same length. These padding tokens are not actual data and are assigned a logit of <code>-inf</code> so that they have no effect on the model's computations. When the softmax function is applied, the output probability associated with a logit of <code>-inf</code> will be effectively <code>0</code>. This ensures that the padding tokens do not influence the model's predictions.</p>
  <p>For a more detailed explanation, read <a href="https://medium.com/analytics-vidhya/masking-in-transformers-self-attention-mechanism-bad3c9ec235c" rel="nofollow noreferrer">this</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-27T05:49:23.223Z
Answers.LastActivityDate: 2023-12-27T05:49:23.223Z
Answers.UserDisplayName: randomuser94786
Answers.Body: <p>In the implementation of transformers, there are specific methods employed to address this issue. The attention mechanism initially establishes a context length, which refers to the number of tokens each attention head processes. The issue you're describing seems to relate to what happens when the number of tokens generated by the decoder is less than the product of the context length and the number of attention heads. This situation is typically resolved by inserting a placeholder token named <code>'[PAD]'</code> for padding purposes. The network, therefore, functions as intended, and the attention calculations include the embeddings of this special token that is added in the vocabulary for that specific purpose. Consequently, the Q matrix will not contain null or random-valued rows, allowing the process to continue smoothly.</p>
  <p>As a reference, when programming the tokenizer for the transformer with <a href="https://huggingface.co/docs/transformers/index" rel="nofollow noreferrer">Hugginface's Transformers library</a>, you can specify whether to use padding in the <a href="https://huggingface.co/docs/transformers/main_classes/tokenizer" rel="nofollow noreferrer">tokenizer class</a>. This will add the '[PAD]' token to the Lookup table dictionary. This is customizable for different applications; for example, you can choose whether to add the paddings to the right or left side.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-21T16:58:57.000Z
Answers.LastActivityDate: 2023-12-21T16:58:57.000Z
Answers.UserDisplayName: Cesar Ruiz
Body: <br>
  I have a question regarding the original transformer implementation (as in "Attention is all you need").<br>
  Assuming I want to translate English to German.
  <ol>
  <li>In the Decoder part, in the self-attention layers, the input is only the German tokens the model already translated, but never the future tokens.
  So in the input matrices X=K=Q=V, what are the values in those rows? some placeholder null token?</li>
  <li>If so, what happens in the (not self) attention afterwards? How does its Q matrix look like as a result from the previous self attention? What bothers me is that its Q matrix would have null/random-valued rows, which would affect the attention result.</li>
  </ol>
  <p>Much appreciated.</p>
  
Tags: <transformer><attention><encoder-decoder>
Title: Transformers - how do the decoder attention input matrices look like, in terms of future tokens?
UserDisplayName: Ben Lahav
ThreadText: # Transformers - how do the decoder attention input matrices look like, in terms of future tokens?
  
  (Asked by: username_0 on 2023-12-16)
  
  I have a question regarding the original transformer implementation (as in "Attention is all you need").  
  Assuming I want to translate English to German.
  
  1. In the Decoder part, in the self-attention layers, the input is only the German tokens the model already translated, but never the future tokens.
     So in the input matrices X=K=Q=V, what are the values in those rows? some placeholder null token?
  2. If so, what happens in the (not self) attention afterwards? How does its Q matrix look like as a result from the previous self attention? What bothers me is that its Q matrix would have null/random-valued rows, which would affect the attention result.
  
  Much appreciated.
  
  **Tags:** transformer, attention, encoder-decoder
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-12-21. Score: 2 | Accepted answer
  
  In the implementation of transformers, there are specific methods employed to address this issue. The attention mechanism initially establishes a context length, which refers to the number of tokens each attention head processes. The issue you're describing seems to relate to what happens when the number of tokens generated by the decoder is less than the product of the context length and the number of attention heads. This situation is typically resolved by inserting a placeholder token named `'[PAD]'` for padding purposes. The network, therefore, functions as intended, and the attention calculations include the embeddings of this special token that is added in the vocabulary for that specific purpose. Consequently, the Q matrix will not contain null or random-valued rows, allowing the process to continue smoothly.
  
  As a reference, when programming the tokenizer for the transformer with [Hugginface's Transformers library](https://huggingface.co/docs/transformers/index), you can specify whether to use padding in the [tokenizer class](https://huggingface.co/docs/transformers/main_classes/tokenizer). This will add the '[PAD]' token to the Lookup table dictionary. This is customizable for different applications; for example, you can choose whether to add the paddings to the right or left side.
  
  
  ---
  
  ### Answer by username_1 on 2023-12-27. Score: 2
  
  In addition to the answer by Cesar Ruiz:
  
  In the Transformer model, padding tokens are used to make all sequences the same length. These padding tokens are not actual data and are assigned a logit of `-inf` so that they have no effect on the model's computations. When the softmax function is applied, the output probability associated with a logit of `-inf` will be effectively `0`. This ensures that the padding tokens do not influence the model's predictions.
  
  For a more detailed explanation, read [this](https://medium.com/analytics-vidhya/masking-in-transformers-self-attention-mechanism-bad3c9ec235c).
  
  
  ---
  
  