Answers.Body: <p>Large-scale language models can be classified into three main types: encoder-decoder, causal decoder and prefix decoder. Each has its advantages and is used in different LLMs for specific purposes.</p>
  <ul>
  <li><p>Encoder-Decoder: Based on the vanilla Transformer model, the encoder-decoder architecture consists of two stacks of Transformer blocks - an encoder and a decoder. The encoder uses stacked multi-headed self-attention layers to encode the input sequence and generate latent representations. The decoder performs cross-attention on these representations and generates the target sequence. This architecture is used in translation software such as Google Translate.</p>
  </li>
  <li><p>Causal Decoder: The causal decoder architecture incorporates a one-way attention mask, allowing each input token to pay attention only to past tokens and to itself. Input and output tokens are treated in the same way in the decoder. The GPT series, including GPT-1, GPT-2 and GPT-3, are representative language models built on this architecture.</p>
  </li>
  <li><p>Prefix Decoder : The prefix decoder architecture, also known as the non-causal decoder, modifies the masking mechanism of causal decoders to allow bidirectional attention on prefix tokens and unidirectional attention on generated tokens.</p>
  </li>
  </ul>
  <p>The choice between these architectures depends on the specific task you want to accomplish. For example, if you want to translate text, you can use the encoder-decoder architecture. If you want to generate text autonomously, you can use the causal decoder architecture. Finally, if you want a combination of both, you can use the prefix decoder architecture.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-06T09:16:48.247Z
Answers.Text: To the best of my knowledge, i think causal decoders models are also called [decoder-only transformers](https://ai.stackexchange.com/questions/40179/how-does-the-decoder-only-transformer-architecture-work) and prefix decoder models are often referred to as encoder-only models. [Nomenclature is not yet completely static on this](https://twitter.com/ylecun/status/1651762787373428736).
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-06T09:00:05.770Z
Answers.LastActivityDate: 2023-11-06T09:17:24.487Z
Answers.LastEditDate: 2023-11-06T09:17:24.487Z
Answers.UserDisplayName: Revolucion for Monica
Body: <p>Motivated by this <a href="https://ai.stackexchange.com/questions/27038/why-does-gpt-2-exclude-the-transformer-encoder">post</a> wherein one of the comments mentioned the use-case for encoder-decoder LM. I wanted to know when to use prefix-decoder LM? vis a vis encoder-decoder or causal decoder only architectures?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-09T11:16:21.117Z
Comments.Text: Can you please briefly describe (1-2 sentences) this "prefix decoder" and what the context is?
Comments.UserDisplayName: nbro
Tags: <natural-language-processing><gpt>
Title: what are the applications scenarios for prefix decoder LMs
UserDisplayName: Singh
ThreadText: # what are the applications scenarios for prefix decoder LMs
  
  (Asked by: username_0 on 2023-11-05)
  
  Motivated by this [post](https://ai.stackexchange.com/questions/27038/why-does-gpt-2-exclude-the-transformer-encoder) wherein one of the comments mentioned the use-case for encoder-decoder LM. I wanted to know when to use prefix-decoder LM? vis a vis encoder-decoder or causal decoder only architectures?
  
  **Tags:** natural-language-processing, gpt
  > **Comments:**
  > **username_1**: Can you please briefly describe (1-2 sentences) this "prefix decoder" and what the context is?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-11-06. Score: 1
  
  Large-scale language models can be classified into three main types: encoder-decoder, causal decoder and prefix decoder. Each has its advantages and is used in different LLMs for specific purposes.
  
  * Encoder-Decoder: Based on the vanilla Transformer model, the encoder-decoder architecture consists of two stacks of Transformer blocks - an encoder and a decoder. The encoder uses stacked multi-headed self-attention layers to encode the input sequence and generate latent representations. The decoder performs cross-attention on these representations and generates the target sequence. This architecture is used in translation software such as Google Translate.
  * Causal Decoder: The causal decoder architecture incorporates a one-way attention mask, allowing each input token to pay attention only to past tokens and to itself. Input and output tokens are treated in the same way in the decoder. The GPT series, including GPT-1, GPT-2 and GPT-3, are representative language models built on this architecture.
  * Prefix Decoder : The prefix decoder architecture, also known as the non-causal decoder, modifies the masking mechanism of causal decoders to allow bidirectional attention on prefix tokens and unidirectional attention on generated tokens.
  
  The choice between these architectures depends on the specific task you want to accomplish. For example, if you want to translate text, you can use the encoder-decoder architecture. If you want to generate text autonomously, you can use the causal decoder architecture. Finally, if you want a combination of both, you can use the prefix decoder architecture.
  
  > **Comments:**
  > **username_3**: To the best of my knowledge, i think causal decoders models are also called [decoder-only transformers](https://ai.stackexchange.com/questions/40179/how-does-the-decoder-only-transformer-architecture-work) and prefix decoder models are often referred to as encoder-only models. [Nomenclature is not yet completely static on this](https://twitter.com/ylecun/status/1651762787373428736).  
  
  ---
  
  