Answers.Body: <blockquote>
  <p>Instead of shuffling <strong>all</strong> images within one big dataset, i only shuffle <strong>batches</strong>, so that the camera views never get mixed up within a single training batch. Using this, i hope that the properties of the background views are learned more specifically.</p>
  </blockquote>
  <p>The result you want - learning background views better in order to recognise objects against them - does <strong>not</strong> follow from the plan of presenting training minibatches where each batch only has a single background. Similarly, there won't be tradeoff with generalisation, because shuffling the training set is not a technique to improve generalisation in the first place. Instead it is a technique to improve performance towards the neural network reaching convergence. It <em>does</em> impact generalisation in as much as a poorly trained network will also score badly on generalisation.</p>
  <p>What will most likely happen is that training proces will be more erratic, and you may reduce the final effectiveness of the model. The impact of this will depend on the size of minibatches that you use. A minbatch size of 1 will be equivalent to normal stochastic gradient descent, and larger minibatches will perform progressively worse on the measures of training time required and final test scores.</p>
  <p>The neural network will learn to recognise and account for the different types of background without you intervening with changes to the dataset. Attempts to incorporate this knowledge of the expected set of backgrounds into tweaks to the training process are probably not necessary.</p>
  <p>There is a related problem which you may or may not have (not clear from the question). If you need to test your trained model's resilience to changes in background camera settings without needing to retrain when an existing camera is adjusted in some way, or perhaps a new camera is added, then you may have an issue with <em>data leakage</em> between training, cross-validation and test environments. This is a common problem when trying to prevent over-fitting where dataset samples might be correlated in some way due to limited data gathering opportunities. The usual solution is to group the data by the common factor that you want to generalise over (in your case, which camera), and ensure that training data does not overlap with cross-validation or test data for this factor. You would still shuffle the training data as normal, after splitting test/train using this grouping/stratification approach. I cannot say whether you need to do this - if the trained model will only ever be used with the same, unadjusted cameras, and you would retrain before using the model again for any change to camera setup, then it would not be necessary.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-19T11:52:19.960Z
Answers.Text: thank you so much for the detailed response! 
  my observation was nevertheless, that with batch shuffling i received lower values for the loss function, compared to individual shuffling; 0.9 (batch shuffle) compared to 1.9 (individual shuffling) using the same time for training. edit: evalution loss function of course ;)
Answers.UserDisplayName: beinando
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-08T07:13:33.913Z
Answers.LastActivityDate: 2024-09-08T07:13:33.913Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>In Pytorch - DataLoader -  If you set Shuffle to false , it will select the samples as they are stored - Sequential Access</p>
  <p>Problems with Shuffle set to false : Gneralization, Overfitting.
  If the model always sees certain classes first, it might allocate more weight or attention to those classes</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-09T03:34:21.410Z
Answers.LastActivityDate: 2024-08-09T03:39:07.953Z
Answers.LastEditDate: 2024-08-09T03:39:07.953Z
Answers.UserDisplayName: Biku
Body: <p>I am training a YOLO (You only look once) object detector for an application within an industrial environment. Since a fixed setup of cameras is used, the backgrounds of the images are camera-dependent, but do not vary over time (besides positional inaccuracies of the measured parts).</p>
  <p>From a data-science point of view, this means: My training data can be logically grouped into N camera views, each containing a specific background.</p>
  <p>My idea is the following: Instead of shuffling <strong>all</strong> images within one big dataset, i only shuffle <strong>batches</strong>, so that the camera views never get mixed up within a single training batch. Using this, i hope that the properties of the background views are learned more specifically. The tradeoff would be to &quot;sacrifice&quot; some generalization capabilities of the neural network (which is not necessary in this special application) in order to get better performance in this specialized setting.</p>
  <p>How different will the weights be updated using this method? Some literature would also be cool, but i did not find similar problems in the time i researched.</p>
  
Tags: <machine-learning><deep-learning><computer-vision><training-datasets>
Title: Deep Learning training strategy: Avoid shuffling individual training images, instead shuffle batches?
UserDisplayName: beinando
ThreadText: # Deep Learning training strategy: Avoid shuffling individual training images, instead shuffle batches?
  
  (Asked by: username_0 on 2024-08-06)
  
  I am training a YOLO (You only look once) object detector for an application within an industrial environment. Since a fixed setup of cameras is used, the backgrounds of the images are camera-dependent, but do not vary over time (besides positional inaccuracies of the measured parts).
  
  From a data-science point of view, this means: My training data can be logically grouped into N camera views, each containing a specific background.
  
  My idea is the following: Instead of shuffling **all** images within one big dataset, i only shuffle **batches**, so that the camera views never get mixed up within a single training batch. Using this, i hope that the properties of the background views are learned more specifically. The tradeoff would be to "sacrifice" some generalization capabilities of the neural network (which is not necessary in this special application) in order to get better performance in this specialized setting.
  
  How different will the weights be updated using this method? Some literature would also be cool, but i did not find similar problems in the time i researched.
  
  **Tags:** machine-learning, deep-learning, computer-vision, training-datasets
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-09-08. Score: 1
  
  > Instead of shuffling **all** images within one big dataset, i only shuffle **batches**, so that the camera views never get mixed up within a single training batch. Using this, i hope that the properties of the background views are learned more specifically.
  
  The result you want - learning background views better in order to recognise objects against them - does **not** follow from the plan of presenting training minibatches where each batch only has a single background. Similarly, there won't be tradeoff with generalisation, because shuffling the training set is not a technique to improve generalisation in the first place. Instead it is a technique to improve performance towards the neural network reaching convergence. It *does* impact generalisation in as much as a poorly trained network will also score badly on generalisation.
  
  What will most likely happen is that training proces will be more erratic, and you may reduce the final effectiveness of the model. The impact of this will depend on the size of minibatches that you use. A minbatch size of 1 will be equivalent to normal stochastic gradient descent, and larger minibatches will perform progressively worse on the measures of training time required and final test scores.
  
  The neural network will learn to recognise and account for the different types of background without you intervening with changes to the dataset. Attempts to incorporate this knowledge of the expected set of backgrounds into tweaks to the training process are probably not necessary.
  
  There is a related problem which you may or may not have (not clear from the question). If you need to test your trained model's resilience to changes in background camera settings without needing to retrain when an existing camera is adjusted in some way, or perhaps a new camera is added, then you may have an issue with *data leakage* between training, cross-validation and test environments. This is a common problem when trying to prevent over-fitting where dataset samples might be correlated in some way due to limited data gathering opportunities. The usual solution is to group the data by the common factor that you want to generalise over (in your case, which camera), and ensure that training data does not overlap with cross-validation or test data for this factor. You would still shuffle the training data as normal, after splitting test/train using this grouping/stratification approach. I cannot say whether you need to do this - if the trained model will only ever be used with the same, unadjusted cameras, and you would retrain before using the model again for any change to camera setup, then it would not be necessary.
  
  > **Comments:**
  > **username_0**: thank you so much for the detailed response!  my observation was nevertheless, that with batch shuffling i received lower values for the loss function, compared to individual shuffling; 0.9 (batch shuffle) compared to 1.9 (individual shuffling) using the same time for training. edit: evalution loss function of course ;)  
  
  ---
  
  ### Answer by username_2 on 2024-08-09. Score: 0
  
  In Pytorch - DataLoader - If you set Shuffle to false , it will select the samples as they are stored - Sequential Access
  
  Problems with Shuffle set to false : Gneralization, Overfitting.
  If the model always sees certain classes first, it might allocate more weight or attention to those classes
  
  
  ---
  
  