Answers.Body: <p>Well well, this text in the link posted in one of the answers above (<a href="https://www.tensorflow.org/text/tutorials/transformer" rel="nofollow noreferrer">https://www.tensorflow.org/text/tutorials/transformer</a>) gives the answer to the question.</p>
  <blockquote>
  <p>Note: The model is optimized for efficient training and makes a
  next-token prediction for each token in the output simultaneously.
  This is redundant during inference, and only the last prediction is
  used. This model can be made more efficient for inference if you only
  calculate the last prediction when running in inference mode
  (training=False).</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-05T09:17:51.537Z
Answers.LastActivityDate: 2024-05-05T09:17:51.537Z
Answers.UserDisplayName: eddys
Answers.Body: <h3>Edit</h3>
  <p>Based on the comments to the original version of this answer, OP indicated that the use case was translation between two languages.</p>
  <h3>Answer:</h3>
  <p>At sampling time, the last linear layer of the decoder is going to output a sequence whose length is incremented by one each time you apply the encoder-decoder transformer to the input sentence.</p>
  <p>Let's take a practical example, with <span class="math-container">$w$</span> denoting the words in the original sentence and <span class="math-container">$w^{i}$</span> those in the target language after iteration <span class="math-container">$i$</span> of applying the transformer model.</p>
  <p>If you have already sampled a sequence <span class="math-container">$(w^1_1, w^2_2, ..., w^{l}_l)$</span>, the inputs to the encoder and the decoder the next time you apply the model (to get the next token <span class="math-container">$w^{l+1}_{l+1}$</span>) are going to be respectively <span class="math-container">$(w_1, w_2, ..., w_T)$</span> (the original sentence) and <span class="math-container">$(w^1_1, w^2_2, ..., w^l_l)$</span>. The output of the decoder (after sampling) will be <span class="math-container">$(w^{l+1}_1, w^{l+1}_2, ..., w^{l+1}_l, w^{l+1}_{l+1})$</span>, but we're only going to keep <span class="math-container">$w^{l+1}_{l+1}$</span>.</p>
  <p>Then, applying the model again to get <span class="math-container">$w^{l+2}_{l+2}$</span>, the new inputs to the encoder and the decoder are going to be <span class="math-container">$(w_1, w_2, ..., w_T)$</span> (again the original sentence) and <span class="math-container">$(w^1_1, w^2_2, ..., w^l_l, w^{l+1}_{l+1})$</span>. The output of the decoder will be <span class="math-container">$(w^{l+2}_1, w^{l+2}_2, ..., w^{l+2}_l, w^{l+2}_{l+1}, w^{l+2}_{l+2})$</span>, and we'll keep only <span class="math-container">$w^{l+2}_{l+2}$</span>.</p>
  <p>This continues until the sentence is fully translated.</p>
  <p>At each step, we give the whole original sentence to the encoder so that the model can build its translation for the next word by looking at the whole sentence. And we give the sentence translated so far to the decoder so that the translation of the next word can attend to what has already been translated.</p>
  <p>To confirm, you can have a look at what they do in the <a href="https://www.tensorflow.org/text/tutorials/transformer#run_inference" rel="nofollow noreferrer">TensorFlow transformer tutorial</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-11T06:23:07.347Z
Answers.Text: Thanks a lot for your answer. Give me please some time to look at the implementation... In the meanwhile there are some clarifications I would need to better understand.
Answers.UserDisplayName: Thomas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-11T06:25:17.453Z
Answers.Text: (1) You say  "the inputs to the encoder and the decoder the next time you apply the model to the sequence (to get the next token wT+1) is going to be (w1,w2,...,wT)". --> what setting do you have in mind ? I am thinking in this thread about sentence translation. For example I would like to translate in german "The dog is on the table". For my understanding the encoder is used (forward pass) just once with the whole english sentence "The dog is on the table". Instead the decoder, as you say, receives as input the translated sentence. "Der" , than "Der Hund", than "Der Hund ist", and so on...
Answers.UserDisplayName: Thomas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-11T06:31:01.403Z
Answers.Text: (2) "The output of the decoder (after sampling) will be (w2,w3,...,wT+1)."--> Since we already have in your example "(w1,w2,...,wT)". Let's call "(w'2,w'3,...,w'T+1)" the output of the decoder, since there is no guarantee that w2=w2',... and so on... The only word we will use in the following steps is w'T+1, right ? (see also next question)
Answers.UserDisplayName: Thomas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-11T06:32:35.653Z
Answers.Text: (3) Then, applying the model again to get wT+2, the new input to the encoder and the decoder is going to be (w2,w3,...,wT,wT+1) --> According to the last question, I thought that the new input would have been (w1,w2,w3,...,wT,w'T+1) (see also video in section "The Decoder Side" of the link "The illustrated transformer)
Answers.UserDisplayName: Thomas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-11T19:43:45.653Z
Answers.Text: Let me update my question for the translation use case.
Answers.UserDisplayName: Raphael Lopez Kaufman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-12T08:31:10.090Z
Answers.Text: Thanks Raphael. It is very useful your update, so that we can communicate on the same lines. Before asking for some clarifications ( I may update my question ), I have an other preparatory point on what you wrote.
Answers.UserDisplayName: Thomas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-12T08:34:33.457Z
Answers.Text: (4) If you have already sampled a sequence (w11,w22,...,wll), the inputs to the encoder and the decoder --> why do you say that at each step we pass the sentence throught the encoder at testing/sampling time ? For my understanding, the sentence is passed throught the encoder just once. It is the output of the encoder that is given as an additional input to the decoder at each translation/sampling step. Performing the encoding of the sentence at each step would mean repeating the same operation at each step of the translation, since the output of the encoder does not depend on tranlsation steps
Answers.UserDisplayName: Thomas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-12T18:32:35.570Z
Answers.Text: Indeed, as an optimisation you can reuse the output of the encoder rather than recalculating it. Depending on your implementation, it might be easy or hard to do. Also, it doesn't change the overall time complexity of the algorithm (although in practice speed might be important!)
Answers.UserDisplayName: Raphael Lopez Kaufman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-31T19:45:52.660Z
Answers.Text: Is there anything which is still unclear? Otherwise, could you accept my answer?
Answers.UserDisplayName: Raphael Lopez Kaufman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-02T05:53:03.757Z
Answers.Text: Hello Raphael. Actually, if I have to be honest, I did not accept your answer because we spent quite some time to clarify what you wrote ( a useful exercise for me though )... and at the end I could not focus on clarifying my question... I did not get actually the answer to what I was really asking and therefore I was planning to make my question more explicit. But this requires me to find some time to modify my question, and I ended up being quite busy lately...
Answers.UserDisplayName: Thomas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-10T21:40:31.700Z
Answers.LastActivityDate: 2022-08-11T21:04:03.943Z
Answers.LastEditDate: 2022-08-11T21:04:03.943Z
Answers.UserDisplayName: Raphael Lopez Kaufman
Answers.Body: <p>I agree with this:</p>
  <blockquote>
  <p>PS: I conjectured that the last linear layer is using just the last
  vector, because than I would understand what happens in training time,
  one would just use in that case all the output vectors from the
  decoder, instead of just the last one, to have a parallelized
  prediction.</p>
  </blockquote>
  <p>Indeed when you look at the code of Tensorflow <a href="https://ai.stackexchange.com/a/36695/47183">shared previusly</a> you see this line</p>
  <pre><code># select the last token from the seq_len dimension
    predictions = predictions[:, -1:, :]  # (batch_size, 1, vocab_size)
  </code></pre>
  <p>where they take the last token of the sequence as a prediction for the next token. And this is consistent with the fact that the input sequence to the decoder is shifted to the right by adding a &lt;START&gt; token at the beginning (see transformers <a href="https://arxiv.org/abs/1706.03762" rel="noreferrer">original paper</a> section 3.1-Decoder).</p>
  <p>So to correct the previous answer, the input and output sequence of the decoder have the same <strong>length</strong> and the last vector is used for the next token prediction which is then added at the end of the input sequence.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-31T13:26:36.043Z
Answers.LastActivityDate: 2023-01-31T13:26:36.043Z
Answers.UserDisplayName: bguetarni
Body: <p>I am learning the transformers architecture from these two sources:</p>
  <p><a href="https://arxiv.org/pdf/1706.03762.pdf" rel="noreferrer">https://arxiv.org/pdf/1706.03762.pdf</a></p>
  <p><a href="https://jalammar.github.io/illustrated-transformer/" rel="noreferrer">https://jalammar.github.io/illustrated-transformer/</a></p>
  <p>I just wanted to ask about the final step in the decoder. Let's fix testing time. As I understand, the decoder starts with an input of dimension <span class="math-container">$(N_{words},d_{emb})$</span>, where <span class="math-container">$N_{words}$</span> is the number of words already predicted and <span class="math-container">$d_{emb}$</span> is the embedding dimension.</p>
  <p>Now if we &quot;follow&quot; the following decoder steps, at each step (after e.g. the attention layers) we should have a vector of dimension <span class="math-container">$(N_{words},d_{model})$</span> where <span class="math-container">$d_{model}$</span> is the model dimension. In other words, up to the final linear layer we have <span class="math-container">$N_{words}$</span> vectors which are <span class="math-container">$d_{model}$</span>-dimensional.</p>
  <p>Are all these <span class="math-container">$N_{words}$</span> vectors fed into the last linear layer (before the softmax) or, as I suspect, only the last of these vectors is used ? In the latter case the last linear layer would be a matrix of dimension <span class="math-container">$d_{model}\times N_{vocab}$</span>, where <span class="math-container">$N_{vocab}$</span> is the vocabulary dimension.</p>
  <p>Is this correct ? Are there any issues in what I wrote ? Unluckily from the online sources I was not able to clarify this point...</p>
  <p>PS: I conjectured that the last linear layer is using just the last vector, because than I would understand what happens in training time, one would just use in that case all the output vectors from the decoder, instead of just the last one, to have a parallelized prediction.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-10T12:35:14.393Z
Comments.Text: Anyway I am starting to be quite convinced that I am right. If the final linear layer were processing all the vectors output of a decoder than the number of parameters of this dense layer would be dependent on the sequence length, which would give problems in handling inputs with different lengths (which we need to handle for language translation)
Comments.UserDisplayName: Thomas
Tags: <deep-learning><transformer>
Title: Last linear layer of the decoder of a transformer
UserDisplayName: Thomas
ThreadText: # Last linear layer of the decoder of a transformer
  
  (Asked by: username_0 on 2022-08-10)
  
  I am learning the transformers architecture from these two sources:
  
  <https://arxiv.org/pdf/1706.03762.pdf>
  
  <https://jalammar.github.io/illustrated-transformer/>
  
  I just wanted to ask about the final step in the decoder. Let's fix testing time. As I understand, the decoder starts with an input of dimension $(N\_{words},d\_{emb})$, where $N\_{words}$ is the number of words already predicted and $d\_{emb}$ is the embedding dimension.
  
  Now if we "follow" the following decoder steps, at each step (after e.g. the attention layers) we should have a vector of dimension $(N\_{words},d\_{model})$ where $d\_{model}$ is the model dimension. In other words, up to the final linear layer we have $N\_{words}$ vectors which are $d\_{model}$-dimensional.
  
  Are all these $N\_{words}$ vectors fed into the last linear layer (before the softmax) or, as I suspect, only the last of these vectors is used ? In the latter case the last linear layer would be a matrix of dimension $d\_{model}\times N\_{vocab}$, where $N\_{vocab}$ is the vocabulary dimension.
  
  Is this correct ? Are there any issues in what I wrote ? Unluckily from the online sources I was not able to clarify this point...
  
  PS: I conjectured that the last linear layer is using just the last vector, because than I would understand what happens in training time, one would just use in that case all the output vectors from the decoder, instead of just the last one, to have a parallelized prediction.
  
  **Tags:** deep-learning, transformer
  
  > **Comments:**
  > **username_0**: Anyway I am starting to be quite convinced that I am right. If the final linear layer were processing all the vectors output of a decoder than the number of parameters of this dense layer would be dependent on the sequence length, which would give problems in handling inputs with different lengths (which we need to handle for language translation)  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2023-01-31. Score: 7 | Accepted answer
  
  I agree with this:
  
  > PS: I conjectured that the last linear layer is using just the last
  > vector, because than I would understand what happens in training time,
  > one would just use in that case all the output vectors from the
  > decoder, instead of just the last one, to have a parallelized
  > prediction.
  
  Indeed when you look at the code of Tensorflow [shared previusly](https://ai.stackexchange.com/a/36695/47183) you see this line
  
  ```
  # select the last token from the seq_len dimension
    predictions = predictions[:, -1:, :]  # (batch_size, 1, vocab_size)
  
  ```
  
  where they take the last token of the sequence as a prediction for the next token. And this is consistent with the fact that the input sequence to the decoder is shifted to the right by adding a <START> token at the beginning (see transformers [original paper](https://arxiv.org/abs/1706.03762) section 3.1-Decoder).
  
  So to correct the previous answer, the input and output sequence of the decoder have the same **length** and the last vector is used for the next token prediction which is then added at the end of the input sequence.
  
  
  ---
  
  ### Answer by username_1 on 2024-05-05. Score: 1
  
  Well well, this text in the link posted in one of the answers above (<https://www.tensorflow.org/text/tutorials/transformer>) gives the answer to the question.
  
  > Note: The model is optimized for efficient training and makes a
  > next-token prediction for each token in the output simultaneously.
  > This is redundant during inference, and only the last prediction is
  > used. This model can be made more efficient for inference if you only
  > calculate the last prediction when running in inference mode
  > (training=False).
  
  
  ---
  
  ### Answer by username_2 on 2022-08-10. Score: 1
  
  ### Edit
  
  Based on the comments to the original version of this answer, OP indicated that the use case was translation between two languages.
  
  ### Answer:
  
  At sampling time, the last linear layer of the decoder is going to output a sequence whose length is incremented by one each time you apply the encoder-decoder transformer to the input sentence.
  
  Let's take a practical example, with $w$ denoting the words in the original sentence and $w^{i}$ those in the target language after iteration $i$ of applying the transformer model.
  
  If you have already sampled a sequence $(w^1\_1, w^2\_2, ..., w^{l}\_l)$, the inputs to the encoder and the decoder the next time you apply the model (to get the next token $w^{l+1}\_{l+1}$) are going to be respectively $(w\_1, w\_2, ..., w\_T)$ (the original sentence) and $(w^1\_1, w^2\_2, ..., w^l\_l)$. The output of the decoder (after sampling) will be $(w^{l+1}\_1, w^{l+1}\_2, ..., w^{l+1}\_l, w^{l+1}\_{l+1})$, but we're only going to keep $w^{l+1}\_{l+1}$.
  
  Then, applying the model again to get $w^{l+2}\_{l+2}$, the new inputs to the encoder and the decoder are going to be $(w\_1, w\_2, ..., w\_T)$ (again the original sentence) and $(w^1\_1, w^2\_2, ..., w^l\_l, w^{l+1}\_{l+1})$. The output of the decoder will be $(w^{l+2}\_1, w^{l+2}\_2, ..., w^{l+2}\_l, w^{l+2}\_{l+1}, w^{l+2}\_{l+2})$, and we'll keep only $w^{l+2}\_{l+2}$.
  
  This continues until the sentence is fully translated.
  
  At each step, we give the whole original sentence to the encoder so that the model can build its translation for the next word by looking at the whole sentence. And we give the sentence translated so far to the decoder so that the translation of the next word can attend to what has already been translated.
  
  To confirm, you can have a look at what they do in the [TensorFlow transformer tutorial](https://www.tensorflow.org/text/tutorials/transformer#run_inference).
  
  > **Comments:**
  > **username_0**: Thanks a lot for your answer. Give me please some time to look at the implementation... In the meanwhile there are some clarifications I would need to better understand.  
  > **username_0**: (1) You say  "the inputs to the encoder and the decoder the next time you apply the model to the sequence (to get the next token wT+1) is going to be (w1,w2,...,wT)". --> what setting do you have in mind ? I am thinking in this thread about sentence translation. For example I would like to translate in german "The dog is on the table". For my understanding the encoder is used (forward pass) just once with the whole english sentence "The dog is on the table". Instead the decoder, as you say, receives as input the translated sentence. "Der" , than "Der Hund", than "Der Hund ist", and so on...  
  > **username_0**: (2) "The output of the decoder (after sampling) will be (w2,w3,...,wT+1)."--> Since we already have in your example "(w1,w2,...,wT)". Let's call "(w'2,w'3,...,w'T+1)" the output of the decoder, since there is no guarantee that w2=w2',... and so on... The only word we will use in the following steps is w'T+1, right ? (see also next question)  
  > **username_0**: (3) Then, applying the model again to get wT+2, the new input to the encoder and the decoder is going to be (w2,w3,...,wT,wT+1) --> According to the last question, I thought that the new input would have been (w1,w2,w3,...,wT,w'T+1) (see also video in section "The Decoder Side" of the link "The illustrated transformer)  
  > **username_2**: Let me update my question for the translation use case.  
  > **username_0**: Thanks username_2. It is very useful your update, so that we can communicate on the same lines. Before asking for some clarifications ( I may update my question ), I have an other preparatory point on what you wrote.  
  > **username_0**: (4) If you have already sampled a sequence (w11,w22,...,wll), the inputs to the encoder and the decoder --> why do you say that at each step we pass the sentence throught the encoder at testing/sampling time ? For my understanding, the sentence is passed throught the encoder just once. It is the output of the encoder that is given as an additional input to the decoder at each translation/sampling step. Performing the encoding of the sentence at each step would mean repeating the same operation at each step of the translation, since the output of the encoder does not depend on tranlsation steps  
  > **username_2**: Indeed, as an optimisation you can reuse the output of the encoder rather than recalculating it. Depending on your implementation, it might be easy or hard to do. Also, it doesn't change the overall time complexity of the algorithm (although in practice speed might be important!)  
  > **username_2**: Is there anything which is still unclear? Otherwise, could you accept my answer?  
  > **username_0**: Hello username_2. Actually, if I have to be honest, I did not accept your answer because we spent quite some time to clarify what you wrote ( a useful exercise for me though )... and at the end I could not focus on clarifying my question... I did not get actually the answer to what I was really asking and therefore I was planning to make my question more explicit. But this requires me to find some time to modify my question, and I ended up being quite busy lately...  
  
  ---
  
  