Answers.Body: <p><strong>GPT-2</strong> is a <strong>close copy</strong> of the basic transformer architecture.</p>
  <p>GPT-2 does not require the encoder part of the original transformer architecture as it is <strong>decoder-only</strong>, and <em>there are no encoder attention blocks</em>, so the <strong>decoder is equivalent to the encoder</strong>, except for the <em>MASKING</em> in the multi-head attention block, the decoder is only allowed to glean information from the prior words in the sentence. It works just like a traditional language model as it takes <strong>word vectors</strong> as input and produces <strong>estimates for the probability</strong> of the next word as outputs but it is <strong>auto-regressive</strong> as each token in the sentence has the context of the previous words. Thus GPT-2 works one token at a time.</p>
  <p><strong>BERT</strong>, by contrast, is <strong>not auto-regressive</strong>. It uses the <strong>entire surrounding context</strong> all-at-once. GPT-2 the context vector is <strong>zero-initialized</strong> for the first word embedding.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-31T10:29:36.200Z
Answers.Text: GPT-2 does not require the encoder part of the transformer architecture because the model uses a masked self-attention that can only look at prior tokens. The encoder is not needed because the model does not need to learn the representation of the input sequence.
Answers.UserDisplayName: Faizy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-27T20:55:36.600Z
Answers.LastActivityDate: 2021-03-27T20:55:36.600Z
Answers.UserDisplayName: Faizy
Answers.Body: <p>The cases when we use encoder-decoder architectures are typically when we are mapping one type of sequence to another type of sequence, e.g. translating French to English or in the case of a chatbot taking a dialogue context and producing a response.  In these cases, there are qualitative differences between the inputs and outputs so that it makes sense to use different weights for them.</p>
  <p>In the case of GPT-2, which is trained on continuous text such as Wikipedia articles, if we wanted to use an encoder-decoder architecture, we would have to make arbitrary cutoffs to determine which part will be dealt with by the encoder and which part by the decoder.  In these cases therefore, it is more common to just use the decoder by itself.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-14T16:39:59.597Z
Answers.LastActivityDate: 2021-05-14T16:39:59.597Z
Answers.UserDisplayName: Ben
Body: <p>After looking into transformers, BERT, and GPT-2, from what I understand, GPT-2 essentially uses only the decoder part of the original transformer architecture and uses masked self-attention that can only look at prior tokens.</p>
  <p>Why does GPT-2 not require the encoder part of the original transformer architecture?</p>
  <p><strong>GPT-2 architecture with only decoder layers</strong></p>
  <p><a href="https://i.sstatic.net/Kb8Gq.png" rel="noreferrer"><img src="https://i.sstatic.net/Kb8Gq.png" alt="enter image description here" /></a></p>
  
Tags: <natural-language-processing><transformer><attention><bert><gpt>
Title: Why does GPT-2 Exclude the Transformer Encoder?
UserDisplayName: Athena Wisdom
ThreadText: # Why does GPT-2 Exclude the Transformer Encoder?
  
  (Asked by: username_0 on 2021-03-27)
  
  After looking into transformers, BERT, and GPT-2, from what I understand, GPT-2 essentially uses only the decoder part of the original transformer architecture and uses masked self-attention that can only look at prior tokens.
  
  Why does GPT-2 not require the encoder part of the original transformer architecture?
  
  **GPT-2 architecture with only decoder layers**
  
  [![enter image description here](https://i.sstatic.net/Kb8Gq.png)](https://i.sstatic.net/Kb8Gq.png)
  
  **Tags:** natural-language-processing, transformer, attention, bert, gpt
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-03-27. Score: 16
  
  **GPT-2** is a **close copy** of the basic transformer architecture.
  
  GPT-2 does not require the encoder part of the original transformer architecture as it is **decoder-only**, and *there are no encoder attention blocks*, so the **decoder is equivalent to the encoder**, except for the *MASKING* in the multi-head attention block, the decoder is only allowed to glean information from the prior words in the sentence. It works just like a traditional language model as it takes **word vectors** as input and produces **estimates for the probability** of the next word as outputs but it is **auto-regressive** as each token in the sentence has the context of the previous words. Thus GPT-2 works one token at a time.
  
  **BERT**, by contrast, is **not auto-regressive**. It uses the **entire surrounding context** all-at-once. GPT-2 the context vector is **zero-initialized** for the first word embedding.
  
  > **Comments:**
  > **username_1**: GPT-2 does not require the encoder part of the transformer architecture because the model uses a masked self-attention that can only look at prior tokens. The encoder is not needed because the model does not need to learn the representation of the input sequence.  
  
  ---
  
  ### Answer by username_2 on 2021-05-14. Score: 7
  
  The cases when we use encoder-decoder architectures are typically when we are mapping one type of sequence to another type of sequence, e.g. translating French to English or in the case of a chatbot taking a dialogue context and producing a response. In these cases, there are qualitative differences between the inputs and outputs so that it makes sense to use different weights for them.
  
  In the case of GPT-2, which is trained on continuous text such as Wikipedia articles, if we wanted to use an encoder-decoder architecture, we would have to make arbitrary cutoffs to determine which part will be dealt with by the encoder and which part by the decoder. In these cases therefore, it is more common to just use the decoder by itself.
  
  
  ---
  
  