Answers.Body: <p>For autoregressive tasks like language modeling, decoder-only models can process long sequences in a straightforward way and avoid the encoder step entirely. The CE loss is calculated for each token prediction across the sequence except the first one in parallel, 4096 predictions from your 4097 token inputs example, and then summed or averaged across all positions to get the total loss to back propagate.</p>
  <p>In encoder-decoder models, the CE loss is calculated for each token in the target sequence <em>conditioned</em> on the input from the encoder, which requires cross-attention. Each token in the target sequence learns to predict the next one based on both the input sequence and preceding target tokens. Therefore encoder-decoder models are less token-efficient for training language models, as they split attention across two sequences (input and target). They are better suited for translation-like tasks where there's an explicit mapping between input and output sequences required.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-14T06:38:55.240Z
Answers.Text: Thanks very much! One thing that's not clear to me is how splitting the attention across two sequences leads to lower token efficiency. Do you have more details on this?
Answers.UserDisplayName: Green grün 绿色 vert зеленый
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-14T07:12:27.250Z
Answers.Text: @Greengrün绿色vertзеленый During the cross-attention phase, each token in the decoder attends to *all* the tokens in the encoder’s self-attended output, where decoder might process some tokens in a way that doesn't maximize their utility. For example, the decoder is focusing on tokens in the encoder that may not carry much new information for a given decoding step, thus consuming more resources for potentially less useful information. Of course it's unavoidable for classic encode-decode models. Hope this clarifies.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-14T05:29:35.753Z
Answers.LastActivityDate: 2024-11-14T05:29:35.753Z
Answers.UserDisplayName: cinch
Body: <p>What is the difference between an encoder-decoder transformer and decoder-only transformer with regard to the loss calculation. Specifically, how does the loss signal differ? And how does this relate to token efficiency?</p>
  <p>As far as I understand, an encoder-decoder model is composed of an encoder and a decoder. The output of the first is fed as queries and keys into a second multi-head attention layer of the decoder part. The input to the encoder is the input sequence. The input to the decoder is the target sequence, shifted right by one, and the output is a probability distribution over the tokens.</p>
  <p>A decoder-only model is basically the same minus the multi-head attention layer that takes the encoder as input. Hence, as far as I understand, the inputs and outputs to the decoder-only model are identical to the encoder-decoder model's decoder.</p>
  <p>When training a decoder-only model, I am given a sequence of, say, 4097 tokens. The first 4096 tokens are my input sequence, the last 4096 tokens my target labels. That way, the model will relate each token to its successor in the sequence. So, when training, will I calculate the loss signal of 4096 tokens at the same time? Shouldn't this be identical to the encoder-decoder model?</p>
  
Tags: <transformer><generative-model><gpt><encoder-decoder>
Title: Difference of encoder-decoder to decoder-only transformers w.r.t. loss
UserDisplayName: Green grün 绿色 vert зеленый
ThreadText: # Difference of encoder-decoder to decoder-only transformers w.r.t. loss
  
  (Asked by: username_0 on 2024-11-13)
  
  What is the difference between an encoder-decoder transformer and decoder-only transformer with regard to the loss calculation. Specifically, how does the loss signal differ? And how does this relate to token efficiency?
  
  As far as I understand, an encoder-decoder model is composed of an encoder and a decoder. The output of the first is fed as queries and keys into a second multi-head attention layer of the decoder part. The input to the encoder is the input sequence. The input to the decoder is the target sequence, shifted right by one, and the output is a probability distribution over the tokens.
  
  A decoder-only model is basically the same minus the multi-head attention layer that takes the encoder as input. Hence, as far as I understand, the inputs and outputs to the decoder-only model are identical to the encoder-decoder model's decoder.
  
  When training a decoder-only model, I am given a sequence of, say, 4097 tokens. The first 4096 tokens are my input sequence, the last 4096 tokens my target labels. That way, the model will relate each token to its successor in the sequence. So, when training, will I calculate the loss signal of 4096 tokens at the same time? Shouldn't this be identical to the encoder-decoder model?
  
  **Tags:** transformer, generative-model, gpt, encoder-decoder
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-11-14. Score: 1 | Accepted answer
  
  For autoregressive tasks like language modeling, decoder-only models can process long sequences in a straightforward way and avoid the encoder step entirely. The CE loss is calculated for each token prediction across the sequence except the first one in parallel, 4096 predictions from your 4097 token inputs example, and then summed or averaged across all positions to get the total loss to back propagate.
  
  In encoder-decoder models, the CE loss is calculated for each token in the target sequence *conditioned* on the input from the encoder, which requires cross-attention. Each token in the target sequence learns to predict the next one based on both the input sequence and preceding target tokens. Therefore encoder-decoder models are less token-efficient for training language models, as they split attention across two sequences (input and target). They are better suited for translation-like tasks where there's an explicit mapping between input and output sequences required.
  
  > **Comments:**
  > **username_0**: Thanks very much! One thing that's not clear to me is how splitting the attention across two sequences leads to lower token efficiency. Do you have more details on this?  
  > **username_1**: @username_0 During the cross-attention phase, each token in the decoder attends to *all* the tokens in the encoder’s self-attended output, where decoder might process some tokens in a way that doesn't maximize their utility. For example, the decoder is focusing on tokens in the encoder that may not carry much new information for a given decoding step, thus consuming more resources for potentially less useful information. Of course it's unavoidable for classic encode-decode models. Hope this clarifies.  
  
  ---
  
  