Answers.Body: <p><strong>Can the decoder in a transformer model be parallelized like the encoder?</strong></p>
  <p>The correct answer is: computation in a Transformer decoder can be parallelized during training, but not during actual translation (or, in a wider sense, generating output sequences for new input sequences during a testing phase).</p>
  <p><strong>What exactly is parallelized?</strong></p>
  <p>Also, it's worth mentioning that &quot;parallelization&quot; in this case means to compute encoder or decoder states in paralllel <strong>for all positions</strong> of the input sequence. Parallelization over several layers is not possible: the first layer of a multi-layer encoder or decoder still needs to finish computing all positions in parallel before the second layer can start computing.</p>
  <p><strong>Why can the decoder be parallelized position-wise during training?</strong></p>
  <p>For each position in the input sequence, a Transformer decoder produces a decoder state as an output. (The decoder state is then used to eventually predict a token in the target sequence.)</p>
  <p>In order to compute one decoder state for a particular position in the sequence of states, the network consumes as inputs: 1) the entire input sequence and 2) <strong>the target words that were generated previously</strong>.</p>
  <p>During training, the target words generated previously are known, since they are taken from the target side of our parallel training data. This is the reason why computation can be factored over positions.</p>
  <p>During inference (also called &quot;testing&quot;, or &quot;translation&quot;), the target words previously generated are predicted by the model, and computing decoder states must be performed sequentially for this reason.</p>
  <p><strong>Comparison to RNN models</strong></p>
  <p>While Transformers can parallelize over input positions during training, an encoder-decoder model based on RNNs cannot parallelize positions. This means that Transformers are generally faster to train, while RNNs are faster for inference.</p>
  <p>This observation leads to the nowadays common practice of training Transformer models and then using <a href="https://arxiv.org/abs/1606.07947" rel="noreferrer">sequence-level distillation</a> to learn an RNN model that mimicks the trained Transformer, for faster inference.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-12T10:35:24.347Z
Answers.Text: it seems your definition of "parallelization" seems to be very different from the actual definition of Parallelization. We say MLP and CNN are Parallelizable because the ith output node produces output independent of other output nodes. That doesn't happen in RNN/LSTM/decoder of transformer. Hence the architecture cannot produce parallel outputs, hence not Parallelizable. Though data/batches can be fed to such architectures in parallel. But that is technically "data parallization", not Parallelization.
Answers.UserDisplayName: Ritwik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-12T12:38:57.613Z
Answers.Text: @Ritwik As I explain in my answer, all elements in an output sequence inside a particular layer of a Transformer decoder are produced in parallel **during training, but not during sequence generation**. What is your "actual definition of parallelization"?
Answers.UserDisplayName: Mathias Müller
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-12T16:40:37.440Z
Answers.Text: isn't sequence generation part of the training process only? and architecture is "parallelizable" if it generates outputs by simple mathematical operations like matmul, max, min, pool, etc. In RNN/decoder of transformer an additional loop is needed for each output word, next output depends on previous output, hence not parallelizable. I will share the reference to this, in some time.
Answers.UserDisplayName: Ritwik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-12T16:57:46.203Z
Answers.Text: @Ritwik "isn't sequence generation part of the training process only?" then what do you call producing a sequence after training is finished? "and architecture is "parallelizable" if it generates outputs by simple mathematical operations like matmul, max, min, pool, etc." - no, that is not an accurate definition of parallelizable. Parallelizable in the context of Transformers here means that a certain operation can be run independently for each item in a sequence. It does not matter at all what kind of mathematical operation is performed.
Answers.UserDisplayName: Mathias Müller
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-12T16:58:30.860Z
Answers.Text: "RNN/decoder of transformer an additional loop is needed for each output word, next output depends on previous output, hence not parallelizable." - yes, but for a Transformer decoder that is _not_ true during training.
Answers.UserDisplayName: Mathias Müller
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-13T06:38:11.637Z
Answers.Text: if sequence generation is _not_ part of training then how exactly loss is calculated in the decoder (while training)?
Answers.UserDisplayName: Ritwik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-13T07:27:59.960Z
Answers.Text: @Ritwik What I meant is: yes, you can think of sequence generation to _also_ happen during training, but during training a decoder is usually not fed its own previous predictions when it has to generate the output distribution for the next time step. Here: "during training, but not during sequence generation" I use "sequence generation" in the sense of "producing sequences after training is finished".
Answers.UserDisplayName: Mathias Müller
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-13T09:39:15.667Z
Answers.Text: quoting directly from the paper "Given z,  the decoder then generates an output sequence of symbols one element at a time. At each step the model is auto-regressive, consuming the previously generated symbols as additional input when generating the next." The paper doesn't differentiate between the training and testing paradigms of the decoder.
Answers.UserDisplayName: Ritwik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-13T11:07:16.250Z
Answers.Text: @Ritwik If you are referring to the paper "Attention is All You Need": even if in this particular sentence, there is no distinction between training and testing, this does not mean that they are identical. In most cases, Transformer decoders are trained with a technique called "teacher forcing" that does not feed the decoder its own previous outputs during training. You are claiming that this is not true without any evidence.
Answers.UserDisplayName: Mathias Müller
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-18T07:08:56.453Z
Answers.Text: @Ritwik "Attention is All You Need" definitely uses teacher forcing in training, even though it's not obvious from the sentence you quoted. For example, take a look at https://github.com/tensorflow/tensor2tensor/issues/695 (I now it doesn't prove that's what the original paper did, but this is the codebase referenced in the paper).
Answers.UserDisplayName: max
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-19T21:34:58.523Z
Answers.Text: I'm still confused about paralelization. As I understand teacher forcing as it is used ground truth instead of model prediction, but still sequentialy or? Lets say that we want translate I'm dog to Ich bin hund. As I imagine it is encoder process I'm dog at once and provide representation to decoder part. Decoder takes start token and produce something, than I provide decoder start token and Ich token(teacher forcing, using truth instead of output) and so on. But your description looks like decoder create Ich bin hund in one run. May you guide me what I missing? :)
Answers.UserDisplayName: viceriel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-20T07:42:26.523Z
Answers.Text: @viceriel The crucial bit is that there is a difference between a) training and b) using a trained model for translation after training. During a) the decoder side can be parallelized more than during b). During a) yes, all positions of the target sentence can be processed in parallel.
Answers.UserDisplayName: Mathias Müller
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-29T13:29:27.277Z
Answers.LastActivityDate: 2020-11-12T12:39:55.133Z
Answers.LastEditDate: 2020-11-12T12:39:55.133Z
Answers.UserDisplayName: Mathias Müller
Answers.Body: <p>Can't see that this has been mentioned yet - there are ways to generate text non-sequentially using a non-autoregressive transformer, where you produce the entire response to the context at once.  This typically produces worse accuracy scores because there are interdependencies within the text being produced - a model translating &quot;thank you&quot; could say &quot;vielen danke&quot; or &quot;danke schön&quot; but whereas an autoregressive model can know which word to say next based on previous decoding, a non-autoregressive model can't do this, so also could produce &quot;danke danke&quot; or &quot;vielen schön&quot;.  There is some research that suggests you can close in on the accuracy gap though: <a href="https://arxiv.org/abs/2012.15833" rel="nofollow noreferrer">https://arxiv.org/abs/2012.15833</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-27T11:44:02.713Z
Answers.Text: Wouldn't positional encoding help it produce it in the right order? Or they would merely  help, but not guarantee the right order?
Answers.UserDisplayName: Kari
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-10T15:59:29.073Z
Answers.Text: The problem is that you're decoding everything in parallel, so even if you apply positional encoding at the beginning, the token at position 1 needs to know what the token at position 2 is going to predict or vice versa, because in the above example "danke" would be appropriate both in the first and second position.  So although it is helpful, it doesn't completely solve the issue.
Answers.UserDisplayName: Ben
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-18T08:10:44.180Z
Answers.Text: It seems my rep is too low to edit directly, so: actually "vielen dank" would be correct in German. This means your second example needs to be "danke dank". You could edit to avoid this minor distraction.
Answers.UserDisplayName: Mathias Müller
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-22T22:38:22.330Z
Answers.LastActivityDate: 2021-02-22T22:38:22.330Z
Answers.UserDisplayName: Ben
Answers.Body: <p>Pytorch implemintation of TransformerDecoderLayer doesn't use predicted output of itself (can use or not labels) but predict all sequence at once. BUT if you are using stack of TransformerDecoderLayers (num of Decoders &gt; 1) they use output of previous layer even if tgt sequence (teacher forcing) is used.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-15T16:38:58.030Z
Answers.LastActivityDate: 2023-12-15T16:38:58.030Z
Answers.UserDisplayName: Dima
Answers.Body: <blockquote>
  <p><strong>Can the decoder in a transformer model be parallelized like the</strong>
  <strong>encoder?</strong></p>
  </blockquote>
  <p><strong>Generally NO:</strong></p>
  <p>Your understanding is completely right. In the decoder, the output of each step is fed to the bottom decoder in the next time step, just like an <a href="https://en.wikipedia.org/wiki/Long_short-term_memory" rel="noreferrer">LSTM</a>.</p>
  <p>Also, like in LSTMs, the self-attention layer needs to attend to earlier positions in the output sequence in order to compute the output. Which makes straight parallelisation impossible.</p>
  <p>However, when decoding during training, there is a frequently used procedure which doesn't take the previous output of the model at step t as input at step t+1, but rather takes the ground truth output at step t. This procudure is called 'Teacher Forcing' and makes the decoder parallelised during training. You can read more about it <a href="https://machinelearningmastery.com/teacher-forcing-for-recurrent-neural-networks/" rel="noreferrer">here</a>.</p>
  <p><em>And For detailed explanation of how Transformer works I suggest reading this article: <a href="http://jalammar.github.io/illustrated-transformer/" rel="noreferrer">The Illustrated Transformer</a>.</em></p>
  <blockquote>
  <p><strong>Is using a transformer-decoder better than say an lstm when comparing</strong>
  <strong>purely on the basis of parallelization?</strong></p>
  </blockquote>
  <p><strong>YES:</strong></p>
  <p>Parallelization is the main drawback of <a href="https://en.wikipedia.org/wiki/Recurrent_neural_network" rel="noreferrer">RNNs</a> in general. In a simple way, RNNs have the ability to memorize but not parallelize while <a href="https://en.wikipedia.org/wiki/Convolutional_neural_network" rel="noreferrer">CNNs</a> have the opposite. Transformers are so powerful because they <strong>combine both</strong> parallelization (at least partially) and memorizing.</p>
  <p>In Natural Language Processing for example, where RNNs are used to be so effective, if you take a look at <a href="https://gluebenchmark.com/leaderboard" rel="noreferrer">GLUE leaderboard</a> you will find that most of the world leading algorithms today are Transformer-based (e.g <a href="https://arxiv.org/abs/1810.04805" rel="noreferrer">BERT by GOOGLE</a>, <a href="https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf" rel="noreferrer">GPT by OpenAI</a>..)</p>
  <p><em>For better understanding of why Transformers are better than CNNs I suggest reading this Medium article: <a href="https://towardsdatascience.com/transformers-141e32e69591" rel="noreferrer">How Transformers Work.</a></em></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-29T14:28:54.367Z
Answers.Text: This answer is misleading since it does not mention the fact that the decoder part of a Transformer is parallelizable during training.
Answers.UserDisplayName: Mathias Müller
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-06T13:24:21.170Z
Answers.Text: Thanks for the note @MathiasMüller. However, while the decoder can be parallelized during training using the 'already  known' trick, to my knowledge this will not have the same result because you can replace by a word that isn't the same as the one the decoder will predict. And this will affect the model differently during back-propagation. So my answer concerns the general understanding of transformer decoder and actual ability to parallelize without a trick that will affect the model. Please clarify if I've mistaken smthg.
Answers.UserDisplayName: HLeb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-06T14:03:05.740Z
Answers.Text: No, this is not a trick that changes the training procedure: all implementations of standard Transformers compute all positions in the same layer in parallel (for both encoder and decoder during training, for encoder during translation). This does not affect the model at all: the results for each position are mathematically independent.
Answers.UserDisplayName: Mathias Müller
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-06T15:34:22.497Z
Answers.Text: ... In case you meant that during training, actual predictions by the model are not used to build up the target sequence: this is also not a trick I would say, but a standard procedure called "teacher forcing" that is used in virtually all supervised sequence prediction models.
Answers.UserDisplayName: Mathias Müller
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-14T07:38:16.600Z
Answers.Text: Thanks @MathiasMüller for the clarification. I edited the answer to include that.
Answers.UserDisplayName: HLeb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-04T06:52:09.977Z
Answers.Text: good answer, might be better if you clearly separate between training and inferencing. For inferencing, I guess decoder-only models cannot be parallelised.
Answers.UserDisplayName: Yneedtobeserious
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-24T10:57:08.330Z
Answers.LastActivityDate: 2021-04-14T07:32:02.637Z
Answers.LastEditDate: 2021-04-14T07:32:02.637Z
Answers.UserDisplayName: HLeb
Body: <p>Can the decoder in a transformer model be parallelized like the encoder?</p>
  
  <p>As far as I understand, the encoder has all the tokens in the sequence to compute the self-attention scores. But for a decoder, this is not possible (in both training and testing), as self-attention is calculated based on previous timestep outputs. Even if we consider some techniques, like teacher forcing, where we are concatenating expected output with obtained, this still has a sequential input from the previous timestep. </p>
  
  <p>In this case, apart from the improvement in capturing long-term dependencies, is using a transformer-decoder better than say an LSTM, when comparing purely on the basis of parallelization?</p>
  
Tags: <deep-learning><comparison><long-short-term-memory><sequence-modeling><transformer>
Title: Can the decoder in a transformer model be parallelized like the encoder?
UserDisplayName: shiredude95
ThreadText: # Can the decoder in a transformer model be parallelized like the encoder?
  
  (Asked by: username_0 on 2019-05-23)
  
  Can the decoder in a transformer model be parallelized like the encoder?
  
  As far as I understand, the encoder has all the tokens in the sequence to compute the self-attention scores. But for a decoder, this is not possible (in both training and testing), as self-attention is calculated based on previous timestep outputs. Even if we consider some techniques, like teacher forcing, where we are concatenating expected output with obtained, this still has a sequential input from the previous timestep.
  
  In this case, apart from the improvement in capturing long-term dependencies, is using a transformer-decoder better than say an LSTM, when comparing purely on the basis of parallelization?
  
  **Tags:** deep-learning, comparison, long-short-term-memory, sequence-modeling, transformer
  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_8 on 2019-05-24. Score: 19 | Accepted answer
  
  > **Can the decoder in a transformer model be parallelized like the**
  > **encoder?**
  
  **Generally NO:**
  
  Your understanding is completely right. In the decoder, the output of each step is fed to the bottom decoder in the next time step, just like an [LSTM](https://en.wikipedia.org/wiki/Long_short-term_memory).
  
  Also, like in LSTMs, the self-attention layer needs to attend to earlier positions in the output sequence in order to compute the output. Which makes straight parallelisation impossible.
  
  However, when decoding during training, there is a frequently used procedure which doesn't take the previous output of the model at step t as input at step t+1, but rather takes the ground truth output at step t. This procudure is called 'Teacher Forcing' and makes the decoder parallelised during training. You can read more about it [here](https://machinelearningmastery.com/teacher-forcing-for-recurrent-neural-networks/).
  
  *And For detailed explanation of how Transformer works I suggest reading this article: [The Illustrated Transformer](http://jalammar.github.io/illustrated-transformer/).*
  
  > **Is using a transformer-decoder better than say an lstm when comparing**
  > **purely on the basis of parallelization?**
  
  **YES:**
  
  Parallelization is the main drawback of [RNNs](https://en.wikipedia.org/wiki/Recurrent_neural_network) in general. In a simple way, RNNs have the ability to memorize but not parallelize while [CNNs](https://en.wikipedia.org/wiki/Convolutional_neural_network) have the opposite. Transformers are so powerful because they **combine both** parallelization (at least partially) and memorizing.
  
  In Natural Language Processing for example, where RNNs are used to be so effective, if you take a look at [GLUE leaderboard](https://gluebenchmark.com/leaderboard) you will find that most of the world leading algorithms today are Transformer-based (e.g [BERT by GOOGLE](https://arxiv.org/abs/1810.04805), [GPT by OpenAI](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf)..)
  
  *For better understanding of why Transformers are better than CNNs I suggest reading this Medium article: [How Transformers Work.](https://towardsdatascience.com/transformers-141e32e69591)*
  
  > **Comments:**
  > **username_1**: This answer is misleading since it does not mention the fact that the decoder part of a Transformer is parallelizable during training.  
  > **username_8**: Thanks for the note @username_1. However, while the decoder can be parallelized during training using the 'already  known' trick, to my knowledge this will not have the same result because you can replace by a word that isn't the same as the one the decoder will predict. And this will affect the model differently during back-propagation. So my answer concerns the general understanding of transformer decoder and actual ability to parallelize without a trick that will affect the model. Please clarify if I've mistaken smthg.  
  > **username_1**: No, this is not a trick that changes the training procedure: all implementations of standard Transformers compute all positions in the same layer in parallel (for both encoder and decoder during training, for encoder during translation). This does not affect the model at all: the results for each position are mathematically independent.  
  > **username_1**: ... In case you meant that during training, actual predictions by the model are not used to build up the target sequence: this is also not a trick I would say, but a standard procedure called "teacher forcing" that is used in virtually all supervised sequence prediction models.  
  > **username_8**: Thanks @username_1 for the clarification. I edited the answer to include that.  
  > **username_9**: good answer, might be better if you clearly separate between training and inferencing. For inferencing, I guess decoder-only models cannot be parallelised.  
  
  ---
  
  ### Answer by username_1 on 2020-01-29. Score: 9
  
  **Can the decoder in a transformer model be parallelized like the encoder?**
  
  The correct answer is: computation in a Transformer decoder can be parallelized during training, but not during actual translation (or, in a wider sense, generating output sequences for new input sequences during a testing phase).
  
  **What exactly is parallelized?**
  
  Also, it's worth mentioning that "parallelization" in this case means to compute encoder or decoder states in paralllel **for all positions** of the input sequence. Parallelization over several layers is not possible: the first layer of a multi-layer encoder or decoder still needs to finish computing all positions in parallel before the second layer can start computing.
  
  **Why can the decoder be parallelized position-wise during training?**
  
  For each position in the input sequence, a Transformer decoder produces a decoder state as an output. (The decoder state is then used to eventually predict a token in the target sequence.)
  
  In order to compute one decoder state for a particular position in the sequence of states, the network consumes as inputs: 1) the entire input sequence and 2) **the target words that were generated previously**.
  
  During training, the target words generated previously are known, since they are taken from the target side of our parallel training data. This is the reason why computation can be factored over positions.
  
  During inference (also called "testing", or "translation"), the target words previously generated are predicted by the model, and computing decoder states must be performed sequentially for this reason.
  
  **Comparison to RNN models**
  
  While Transformers can parallelize over input positions during training, an encoder-decoder model based on RNNs cannot parallelize positions. This means that Transformers are generally faster to train, while RNNs are faster for inference.
  
  This observation leads to the nowadays common practice of training Transformer models and then using [sequence-level distillation](https://arxiv.org/abs/1606.07947) to learn an RNN model that mimicks the trained Transformer, for faster inference.
  
  > **Comments:**
  > **username_2**: it seems your definition of "parallelization" seems to be very different from the actual definition of Parallelization. We say MLP and CNN are Parallelizable because the ith output node produces output independent of other output nodes. That doesn't happen in RNN/LSTM/decoder of transformer. Hence the architecture cannot produce parallel outputs, hence not Parallelizable. Though data/batches can be fed to such architectures in parallel. But that is technically "data parallization", not Parallelization.  
  > **username_1**: @username_2 As I explain in my answer, all elements in an output sequence inside a particular layer of a Transformer decoder are produced in parallel **during training, but not during sequence generation**. What is your "actual definition of parallelization"?  
  > **username_2**: isn't sequence generation part of the training process only? and architecture is "parallelizable" if it generates outputs by simple mathematical operations like matmul, max, min, pool, etc. In RNN/decoder of transformer an additional loop is needed for each output word, next output depends on previous output, hence not parallelizable. I will share the reference to this, in some time.  
  > **username_1**: @username_2 "isn't sequence generation part of the training process only?" then what do you call producing a sequence after training is finished? "and architecture is "parallelizable" if it generates outputs by simple mathematical operations like matmul, max, min, pool, etc." - no, that is not an accurate definition of parallelizable. Parallelizable in the context of Transformers here means that a certain operation can be run independently for each item in a sequence. It does not matter at all what kind of mathematical operation is performed.  
  > **username_1**: "RNN/decoder of transformer an additional loop is needed for each output word, next output depends on previous output, hence not parallelizable." - yes, but for a Transformer decoder that is _not_ true during training.  
  > **username_2**: if sequence generation is _not_ part of training then how exactly loss is calculated in the decoder (while training)?  
  > **username_1**: @username_2 What I meant is: yes, you can think of sequence generation to _also_ happen during training, but during training a decoder is usually not fed its own previous predictions when it has to generate the output distribution for the next time step. Here: "during training, but not during sequence generation" I use "sequence generation" in the sense of "producing sequences after training is finished".  
  > **username_2**: quoting directly from the paper "Given z,  the decoder then generates an output sequence of symbols one element at a time. At each step the model is auto-regressive, consuming the previously generated symbols as additional input when generating the next." The paper doesn't differentiate between the training and testing paradigms of the decoder.  
  > **username_1**: @username_2 If you are referring to the paper "Attention is All You Need": even if in this particular sentence, there is no distinction between training and testing, this does not mean that they are identical. In most cases, Transformer decoders are trained with a technique called "teacher forcing" that does not feed the decoder its own previous outputs during training. You are claiming that this is not true without any evidence.  
  > **username_3**: @username_2 "Attention is All You Need" definitely uses teacher forcing in training, even though it's not obvious from the sentence you quoted. For example, take a look at https://github.com/tensorflow/tensor2tensor/issues/695 (I now it doesn't prove that's what the original paper did, but this is the codebase referenced in the paper).  
  > **username_4**: I'm still confused about paralelization. As I understand teacher forcing as it is used ground truth instead of model prediction, but still sequentialy or? Lets say that we want translate I'm dog to Ich bin hund. As I imagine it is encoder process I'm dog at once and provide representation to decoder part. Decoder takes start token and produce something, than I provide decoder start token and Ich token(teacher forcing, using truth instead of output) and so on. But your description looks like decoder create Ich bin hund in one run. May you guide me what I missing? :)  
  > **username_1**: @username_4 The crucial bit is that there is a difference between a) training and b) using a trained model for translation after training. During a) the decoder side can be parallelized more than during b). During a) yes, all positions of the target sentence can be processed in parallel.  
  
  ---
  
  ### Answer by username_5 on 2021-02-22. Score: 2
  
  Can't see that this has been mentioned yet - there are ways to generate text non-sequentially using a non-autoregressive transformer, where you produce the entire response to the context at once. This typically produces worse accuracy scores because there are interdependencies within the text being produced - a model translating "thank you" could say "vielen danke" or "danke schön" but whereas an autoregressive model can know which word to say next based on previous decoding, a non-autoregressive model can't do this, so also could produce "danke danke" or "vielen schön". There is some research that suggests you can close in on the accuracy gap though: <https://arxiv.org/abs/2012.15833>.
  
  > **Comments:**
  > **username_6**: Wouldn't positional encoding help it produce it in the right order? Or they would merely  help, but not guarantee the right order?  
  > **username_5**: The problem is that you're decoding everything in parallel, so even if you apply positional encoding at the beginning, the token at position 1 needs to know what the token at position 2 is going to predict or vice versa, because in the above example "danke" would be appropriate both in the first and second position.  So although it is helpful, it doesn't completely solve the issue.  
  > **username_1**: It seems my rep is too low to edit directly, so: actually "vielen dank" would be correct in German. This means your second example needs to be "danke dank". You could edit to avoid this minor distraction.  
  
  ---
  
  ### Answer by username_7 on 2023-12-15. Score: 0
  
  Pytorch implemintation of TransformerDecoderLayer doesn't use predicted output of itself (can use or not labels) but predict all sequence at once. BUT if you are using stack of TransformerDecoderLayers (num of Decoders > 1) they use output of previous layer even if tgt sequence (teacher forcing) is used.
  
  
  ---
  
  