Answers.Body: <p>The reason why you want to pad at all is because you want to stuff everything into giant matrix multiplies. GPUs are great at parallelizing these operations. Instead of running e.g., <span class="math-container">$Wx_i + b$</span> 32 times, you can run <span class="math-container">$W X + b$</span> a single time, where each column of <span class="math-container">$X$</span> is a single input.</p>
  <p>So, if you have inputs with different input lengths, you just need to add enough padding such that you can fit all your <span class="math-container">$x_i$</span>s into a single matrix <span class="math-container">$X$</span>. That is, you only need to pad to the maximum length item in your batch.</p>
  <p>The padding tokens are also masked, so that tokens can't attend to them and they don't contribute to the loss. Because of this, adding any additional padding tokens wouldn't really do anything.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-24T16:03:51.067Z
Answers.Text: Sorry but my question is how come different batches have different length. In the example pasted above 2 of the batches had length of 13 and 1 had a length of 14.
Answers.UserDisplayName: PS1
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-24T17:44:07.353Z
Answers.Text: Because you only need to pad to the maximum length sequence in your batch. Some batches will have shorter sentences and some batch will have longer sentences.
Answers.UserDisplayName: Alexander Wan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-24T00:15:22.670Z
Answers.LastActivityDate: 2023-08-24T00:15:22.670Z
Answers.UserDisplayName: Alexander Wan
Body: <p>I understand that all inputs in a batch need to be of the same size. However, it seems BERT/Transformers models can accept batches with different sizes as input.</p>
  <p>How is that possible? I thought we needed to pad all examples in a batch to <code>model.max_input_size</code>, however, it seems HuggingFace does <code>Dynamic Padding</code> that allows sending batches of different lengths (till the time they are smaller than <code>max_input_size</code>)</p>
  <p><a href="https://i.sstatic.net/zC1WS.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zC1WS.png" alt="enter image description here" /></a></p>
  <p>Link: <a href="https://mccormickml.com/2020/07/29/smart-batching-tutorial/" rel="nofollow noreferrer">https://mccormickml.com/2020/07/29/smart-batching-tutorial/</a><br />
  Link2: <a href="https://huggingface.co/learn/nlp-course/en/chapter3/2?fw=pt#dynamic-padding" rel="nofollow noreferrer">https://huggingface.co/learn/nlp-course/en/chapter3/2?fw=pt#dynamic-padding</a></p>
  
Tags: <neural-networks><natural-language-processing><transformer><bert>
Title: How can BERT/Transformer models accept input batches of different sizes?
UserDisplayName: PS1
ThreadText: # How can BERT/Transformer models accept input batches of different sizes?
  
  (Asked by: username_0 on 2023-08-23)
  
  I understand that all inputs in a batch need to be of the same size. However, it seems BERT/Transformers models can accept batches with different sizes as input.
  
  How is that possible? I thought we needed to pad all examples in a batch to `model.max_input_size`, however, it seems HuggingFace does `Dynamic Padding` that allows sending batches of different lengths (till the time they are smaller than `max_input_size`)
  
  [![enter image description here](https://i.sstatic.net/zC1WS.png)](https://i.sstatic.net/zC1WS.png)
  
  Link: <https://mccormickml.com/2020/07/29/smart-batching-tutorial/>  
  Link2: <https://huggingface.co/learn/nlp-course/en/chapter3/2?fw=pt#dynamic-padding>
  
  **Tags:** neural-networks, natural-language-processing, transformer, bert
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-08-24. Score: 1
  
  The reason why you want to pad at all is because you want to stuff everything into giant matrix multiplies. GPUs are great at parallelizing these operations. Instead of running e.g., $Wx\_i + b$ 32 times, you can run $W X + b$ a single time, where each column of $X$ is a single input.
  
  So, if you have inputs with different input lengths, you just need to add enough padding such that you can fit all your $x\_i$s into a single matrix $X$. That is, you only need to pad to the maximum length item in your batch.
  
  The padding tokens are also masked, so that tokens can't attend to them and they don't contribute to the loss. Because of this, adding any additional padding tokens wouldn't really do anything.
  
  > **Comments:**
  > **username_0**: Sorry but my question is how come different batches have different length. In the example pasted above 2 of the batches had length of 13 and 1 had a length of 14.  
  > **username_1**: Because you only need to pad to the maximum length sequence in your batch. Some batches will have shorter sentences and some batch will have longer sentences.  
  
  ---
  
  