Answers.Body: <p>Each sample in each batch is processed separately. Each batch in turn is handled separately. Having consistency between batches is not going to improve the convergence of your model.</p>
  <p>If any, it might result in a less generalized model because your data is <a href="https://datascience.stackexchange.com/questions/24511/why-should-the-data-be-shuffled-for-machine-learning-tasks">not shuffled enough</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-18T09:37:40.387Z
Answers.LastActivityDate: 2023-05-18T09:37:40.387Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>When training a transformer model (I'm using nn.TransformerEncoder from pytorch) is it better to use sequential batches (for example, three sequences <code>[[1,2,3,4], [5,6,7,8], [9,10,11,12]]</code> where next sequence continues where the previous ended)?</p>
  <p>Or can I use non-sequential or randomly picked sequences from my dataset (for example, here are three randomly picked sequences <code>[[1,2,3,4], [56, 57, 58, 59], [22, 23, 24, 25]]</code>)?</p>
  <p>I can make a test myself, but I was just wondering if that's already known.</p>
  
Tags: <transformer><pytorch>
Title: Do batches need to be sequential in Transformer traning?
UserDisplayName: Evgenii
ThreadText: # Do batches need to be sequential in Transformer traning?
  
  (Asked by: username_0 on 2023-05-18)
  
  When training a transformer model (I'm using nn.TransformerEncoder from pytorch) is it better to use sequential batches (for example, three sequences `[[1,2,3,4], [5,6,7,8], [9,10,11,12]]` where next sequence continues where the previous ended)?
  
  Or can I use non-sequential or randomly picked sequences from my dataset (for example, here are three randomly picked sequences `[[1,2,3,4], [56, 57, 58, 59], [22, 23, 24, 25]]`)?
  
  I can make a test myself, but I was just wondering if that's already known.
  
  **Tags:** transformer, pytorch
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-05-18. Score: 2 | Accepted answer
  
  Each sample in each batch is processed separately. Each batch in turn is handled separately. Having consistency between batches is not going to improve the convergence of your model.
  
  If any, it might result in a less generalized model because your data is [not shuffled enough](https://datascience.stackexchange.com/questions/24511/why-should-the-data-be-shuffled-for-machine-learning-tasks).
  
  
  ---
  
  