Body: <p>I have seen a number of ways to train (yes, train, not fine-tune) these models efficiently with batches. I will illustrate these techniques with the following example dataset and context window:</p>
  <pre><code>Context window:
     -----------------
  Data samples:
  1. ###
  2. ################
  3. ####
  4. ##############
  5. ########
  6. #########
  </code></pre>
  <p>Suppose we have a batch size of 2. Our pad token is x</p>
  <h2>First technique: Vanilla Padding</h2>
  <pre><code>Context window:
     -----------------
  batch 1:
  1. ###xxxxxxxxxxxxx
  2. ################
  
  batch 2:
  3. ####xxxxxxxxxx
  4. ##############
  
  batch 3: 
  5. ########x
  6. #########
  </code></pre>
  <h2>Second technique: Bucketed Padding</h2>
  <p>Samples of similar lengths are batched together to minimise the number of pad tokens</p>
  <pre><code>Context window:
     -----------------
  batch 1:
  1. ###x
  3. ####
  
  batch 2:
  2. ################
  4. ##############xx
  
  batch 3: 
  5. ########x
  6. #########
  </code></pre>
  <p>this is <em>uniform length batching</em> described in <a href="https://mccormickml.com/2020/07/29/smart-batching-tutorial/" rel="nofollow noreferrer">this blogpost</a> and referred to as <em>bucketed random sampling</em> in <a href="https://aclanthology.org/2021.findings-acl.74/" rel="nofollow noreferrer">this paper</a>.</p>
  <h2>Third technique: Concatenating samples</h2>
  <p>In this technique, we concatenate samples, separating them with a EOS token (E) until they reach the context length. In this way, we have no padding tokens, and the entire context length is used. The attention mask keeps track of where the EOS tokens occur.</p>
  <pre><code>Context window:
     -----------------
  batch 1:
     ###E############# (1 and part of 2)
  batch 2:
     ###E####E######## (rest of 2, 3 and part of 4)
  batch 3:
     ######E########E# (rest of 4, 5, part of 6)
  batch 4:
     ######## (rest of 6)
  </code></pre>
  <p>This technique is referenced at 2:28 of <a href="https://www.youtube.com/watch?v=ma1TrR7gE7I" rel="nofollow noreferrer">this video</a> from <a href="https://huggingface.co/course/chapter7/6" rel="nofollow noreferrer">this huggingface tutorial</a>.</p>
  <p>With this technique, we reduce the number of batches, and only have to pad the final batch if necessary. However, it is unclear to me whether this is &quot;allowed&quot; for causal language modelling, as it is unclear whether this will cause the causal attention mechanism to attend to tokens from previous samples, only ignoring the EOS token (instead of everything before it)</p>
  <hr />
  <p>Of these 3 techniques, which is the most memory efficient? Which is the most commonly used?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-28T14:30:52.370Z
Comments.Text: @user253751 I don't recall saying that. The attention mask only keeps track of where the EOS token is, not whether to conditionally attend to tokens before it.
  You could indeed concatenate all the samples but eventually you'd run into the max sequence length afforded by the transformer (the context window size, e.g. 1024 tokens) so you'd have to resort to batching.
Comments.UserDisplayName: thesofakillers
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-28T14:36:38.433Z
Comments.Text: indeed it's the point of the attention mask, but it shouldn't be attending to tokens before the EOS token delimiting the previous sample, since those are not technically preceeding the tokens of the current sample.
Comments.UserDisplayName: thesofakillers
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-28T14:41:20.073Z
Comments.Text: The issue is that I would have to make a separate attention mask for each sample to do that, which defeats the purpose of concatenating them in the first place. It's because of this issue that I am unsure whether this technique (suggested by huggingface) is actually valid for CLM training.
Comments.UserDisplayName: thesofakillers
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-28T14:53:32.000Z
Comments.Text: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/144920/discussion-between-thesofakillers-and-user253751).
Comments.UserDisplayName: thesofakillers
Tags: <training><transformer><gpt><language-model><batch-learning>
Title: What's the most efficient way of performing batched training of Causal Language Models?
UserDisplayName: thesofakillers
ThreadText: # What's the most efficient way of performing batched training of Causal Language Models?
  
  (Asked by: username_0 on 2023-03-28)
  
  I have seen a number of ways to train (yes, train, not fine-tune) these models efficiently with batches. I will illustrate these techniques with the following example dataset and context window:
  Context window:
     -----------------
  Data samples:
  1. ###
  2. ################
  3. ####
  4. ##############
  5. ########
  6. #########
  
  Suppose we have a batch size of 2. Our pad token is x
  First technique: Vanilla Padding
  Context window:
     -----------------
  batch 1:
  1. ###xxxxxxxxxxxxx
  2. ################
  
  batch 2:
  3. ####xxxxxxxxxx
  4. ##############
  
  batch 3: 
  5. ########x
  6. #########
  
  Second technique: Bucketed Padding
  Samples of similar lengths are batched together to minimise the number of pad tokens
  Context window:
     -----------------
  batch 1:
  1. ###x
  3. ####
  
  batch 2:
  2. ################
  4. ##############xx
  
  batch 3: 
  5. ########x
  6. #########
  
  this is uniform length batching described in this blogpost and referred to as bucketed random sampling in this paper.
  Third technique: Concatenating samples
  In this technique, we concatenate samples, separating them with a EOS token (E) until they reach the context length. In this way, we have no padding tokens, and the entire context length is used. The attention mask keeps track of where the EOS tokens occur.
  Context window:
     -----------------
  batch 1:
     ###E############# (1 and part of 2)
  batch 2:
     ###E####E######## (rest of 2, 3 and part of 4)
  batch 3:
     ######E########E# (rest of 4, 5, part of 6)
  batch 4:
     ######## (rest of 6)
  
  This technique is referenced at 2:28 of this video from this huggingface tutorial.
  With this technique, we reduce the number of batches, and only have to pad the final batch if necessary. However, it is unclear to me whether this is "allowed" for causal language modelling, as it is unclear whether this will cause the causal attention mechanism to attend to tokens from previous samples, only ignoring the EOS token (instead of everything before it)
  
  Of these 3 techniques, which is the most memory efficient? Which is the most commonly used?
  
  **Tags:** training, transformer, gpt, language-model, batch-learning
  > **Comments:**
  > **username_0**: @user253751 I don't recall saying that. The attention mask only keeps track of where the EOS token is, not whether to conditionally attend to tokens before it. You could indeed concatenate all the samples but eventually you'd run into the max sequence length afforded by the transformer (the context window size, e.g. 1024 tokens) so you'd have to resort to batching.  
  > **username_0**: indeed it's the point of the attention mask, but it shouldn't be attending to tokens before the EOS token delimiting the previous sample, since those are not technically preceeding the tokens of the current sample.  
  > **username_0**: The issue is that I would have to make a separate attention mask for each sample to do that, which defeats the purpose of concatenating them in the first place. It's because of this issue that I am unsure whether this technique (suggested by huggingface) is actually valid for CLM training.  
  > **username_0**: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/144920/discussion-between-thesofakillers-and-user253751).  
  
  ---
  
  ## 0 answers
  
  