Answers.Body: <p>During training, the decoder can be trained in parallel (and that's one of its advantage over LSTM) :</p>
  <p>You input <code>&lt;s&gt; I love you</code>, and the decoder learns to produce <code>I love you &lt;/s&gt;</code>. For each token, the model learns to predict the next token (for example given <code>&lt;s&gt;</code> the model learns to predict <code>I</code>). This is possible thanks to attention.</p>
  <hr />
  <p>But at inference time, you are right : the model has to predict the first token, then put this token in the decoder input and predict the second, etc...</p>
  <p>So yes, at inference time a decoder is much slower (because we go through the whole decoder several times : one time for each token) than an encoder (which do a single forward pass).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-09T05:52:45.337Z
Answers.LastActivityDate: 2022-11-09T05:52:45.337Z
Answers.UserDisplayName: Astariul
Body: <p>From my understanding, seq2seq models work by first computing a representation of the input sequence, and feeding this to the decoder. The decoder then predicts each token in the output sequence in an autoregressive manner. In this sense, it's limited to processing one time step at a time, as the next token in the sequence depends on the previous (non parallelizable). This is different from the encoder, as it's able to process many time steps in parallel. Is my understanding correct?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-16T22:19:39.333Z
Comments.Text: Yes, the decoder in a transformer Seq2Seq model is non-parallelizable. Yes, that is correct.
Comments.UserDisplayName: Faizy
Tags: <deep-learning><transformer><seq2seq>
Title: Is the decoder in a transformer Seq2Seq model non parallelizable?
UserDisplayName: Andrew Tang
ThreadText: # Is the decoder in a transformer Seq2Seq model non parallelizable?
  
  (Asked by: username_0 on 2022-10-12)
  
  From my understanding, seq2seq models work by first computing a representation of the input sequence, and feeding this to the decoder. The decoder then predicts each token in the output sequence in an autoregressive manner. In this sense, it's limited to processing one time step at a time, as the next token in the sequence depends on the previous (non parallelizable). This is different from the encoder, as it's able to process many time steps in parallel. Is my understanding correct?
  
  **Tags:** deep-learning, transformer, seq2seq
  
  > **Comments:**
  > **username_1**: Yes, the decoder in a transformer Seq2Seq model is non-parallelizable. Yes, that is correct.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-11-09. Score: 1
  
  During training, the decoder can be trained in parallel (and that's one of its advantage over LSTM) :
  
  You input `<s> I love you`, and the decoder learns to produce `I love you </s>`. For each token, the model learns to predict the next token (for example given `<s>` the model learns to predict `I`). This is possible thanks to attention.
  
  ---
  
  But at inference time, you are right : the model has to predict the first token, then put this token in the decoder input and predict the second, etc...
  
  So yes, at inference time a decoder is much slower (because we go through the whole decoder several times : one time for each token) than an encoder (which do a single forward pass).
  
  
  ---
  
  