Answers.Body: <p>The original encoder+decoder transformer (from <em>attention is all you need</em>) can perform different tasks compared to either the decoder-only transformer or the encoder-only transformer.</p>
  <p>The encoder+decoder transformer was originally proposed for <em>language translation</em> tasks. Where the encoder receives the complete foreign language prompt, and the decoder receives what has already been translated. Handling such a task is not possible for the decoder-only or encoder-only transformer.</p>
  <p>Even though decoder-only architectures can do language translation tasks, the encoder-decoder architecture suits this better. Decoder-only transformers are mainly suited for next-word prediction (ChatGPT etc).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-05T09:57:12.573Z
Answers.LastActivityDate: 2023-11-05T09:57:12.573Z
Answers.UserDisplayName: Robin van Hoorn
Answers.Body: <p>Transformers are based on encoder-decoder architectures for sequence processing, the point of this is to allow different input and output sequence lengths, and soft-attention was initially defined for encoder-decoder architectures for neural machine translation.</p>
  <p>So the Transformer is an extension, from soft-attention to multi-head attention, but also a transformer is an autoregressive model, these ideas combines mean that the encoder consume the input while the decoder layers partially take the encoder output at the same time that the decoder output at step <code>t</code> is given as input at step <code>t+1</code> (this is the autoregressive part).</p>
  <p>So you cannot make a Transformer without encoder-decoder architectures, if not it would lose its capacity to process sequences.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-27T16:46:10.403Z
Answers.Text: Decoder only model can also do the same right
Answers.UserDisplayName: Swastik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-27T20:51:23.267Z
Answers.Text: @Swastik Depends on what you mean "the same".
Answers.UserDisplayName: Dr. Snoopy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-27T21:13:31.733Z
Answers.Text: Snoopy I mean whatever mentioned in your answer / whatever an encoder-decoder model can do, the same can be done using decoder only model also right?
Answers.UserDisplayName: Swastik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-27T23:17:09.117Z
Answers.Text: @Swastik No, not at all, how would you process variable-sized sequences with only a decoder?
Answers.UserDisplayName: Dr. Snoopy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-26T21:11:14.557Z
Answers.LastActivityDate: 2023-10-26T21:11:14.557Z
Answers.UserDisplayName: Dr. Snoopy
Body: <p>In the original Transformers paper why encoder is added when a decoder alone can do what an encoder can do (like multi-head attention, feed-forward NN etc....).</p>
  <p>I mean even a decoder also has the same components that an encoder has</p>
  <p><a href="https://i.sstatic.net/i7Rkm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/i7Rkm.png" alt="enter image description here" /></a></p>
  
Tags: <natural-language-processing><transformer><attention><natural-language-understanding><encoder-decoder>
Title: Why encoders are required in Transformers
UserDisplayName: Swastik
ThreadText: # Why encoders are required in Transformers
  
  (Asked by: username_0 on 2023-10-26)
  
  In the original Transformers paper why encoder is added when a decoder alone can do what an encoder can do (like multi-head attention, feed-forward NN etc....).
  
  I mean even a decoder also has the same components that an encoder has
  
  [![enter image description here](https://i.sstatic.net/i7Rkm.png)](https://i.sstatic.net/i7Rkm.png)
  
  **Tags:** natural-language-processing, transformer, attention, natural-language-understanding, encoder-decoder
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-11-05. Score: 1
  
  The original encoder+decoder transformer (from *attention is all you need*) can perform different tasks compared to either the decoder-only transformer or the encoder-only transformer.
  
  The encoder+decoder transformer was originally proposed for *language translation* tasks. Where the encoder receives the complete foreign language prompt, and the decoder receives what has already been translated. Handling such a task is not possible for the decoder-only or encoder-only transformer.
  
  Even though decoder-only architectures can do language translation tasks, the encoder-decoder architecture suits this better. Decoder-only transformers are mainly suited for next-word prediction (ChatGPT etc).
  
  
  ---
  
  ### Answer by username_2 on 2023-10-26. Score: 1
  
  Transformers are based on encoder-decoder architectures for sequence processing, the point of this is to allow different input and output sequence lengths, and soft-attention was initially defined for encoder-decoder architectures for neural machine translation.
  
  So the Transformer is an extension, from soft-attention to multi-head attention, but also a transformer is an autoregressive model, these ideas combines mean that the encoder consume the input while the decoder layers partially take the encoder output at the same time that the decoder output at step `t` is given as input at step `t+1` (this is the autoregressive part).
  
  So you cannot make a Transformer without encoder-decoder architectures, if not it would lose its capacity to process sequences.
  
  > **Comments:**
  > **username_0**: Decoder only model can also do the same right  
  > **username_2**: @username_0 Depends on what you mean "the same".  
  > **username_0**: username_2 I mean whatever mentioned in your answer / whatever an encoder-decoder model can do, the same can be done using decoder only model also right?  
  > **username_2**: @username_0 No, not at all, how would you process variable-sized sequences with only a decoder?  
  
  ---
  
  