Answers.Body: <p>You can read that this was done in &quot;<a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">Attention is All You Need</a>&quot;, for Transformers: &quot;Sentence pairs were batched together by approximate sequence length.&quot;.</p>
  <p>But, with RNN you don't need any padding at all when using packedSequences, so I don't think your idea will be efficient, if I understand correctly.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-21T14:34:43.033Z
Answers.LastActivityDate: 2023-02-22T23:13:01.610Z
Answers.LastEditDate: 2023-02-22T23:13:01.610Z
Answers.UserDisplayName: EmmaRenauld
Body: <p>I am training an RNN in PyTorch to produce captions for images. It's a pretty standard architecture – the image is processed by a pre-trained InceptionV3 to extract features, the recurrent module processes the words seen so far and then its result is merged with image features and fed through a linear layer to produce a probability distribution over the vocab to predict the next word. A network trained this way can auto-regressively predict a whole caption for an image (it's seeded with an artificial  token at the beginning to have a preceding sequence to start off with).</p>
  <p>To train the network every caption is decomposed like this in the training dataset:</p>
  <pre><code>X, y
  &lt;START&gt;, a
  &lt;START&gt; a, man
  &lt;START&gt; a man, is
  &lt;START&gt; a man is, sitting
  &lt;START&gt; a man is sitting, on
  &lt;START&gt; a man is sitting on, a
  &lt;START&gt; a man is sitting on a, chair
  &lt;START&gt; a man is sitting on a chair, &lt;END&gt;
  </code></pre>
  <p>The result is that there is a lot of input sequence length variation in the training data which is a problem since we would need a lot of padding in batches. I am aware that there are techniques of counteracting this (<code>pack_padded_sequence</code> in PyTorch) but I came up with another idea.</p>
  <p>I am modifying the <code>Sampler</code> object that's used to provide indices for the <code>DataLoader</code> to construct batches in such a way that the <code>Sampler</code> orders all of the example indices by input sequence length, groups it into batch-size chunks, and shuffles the chunks before feeding them to the <code>DataLoader</code>. As a result the sequences in individual batches are almost always the same length – not because of any padding but because of the fact that we ordered the examples by length before grouping them. There will be some batches that will encompass examples on a border of length groups so some padding will be necessary but it will be minimal (just one zero for a group of lengths [5, 5, 5, 6, 6, 6] for example) and the number of such groups would be negligible (~0.002% of the batches in my case).</p>
  <p>My question is – does this method introduce some kind of drawback that I am not aware of? I find it simpler than having completely random batches and padding and packing them. I can't tell however, if it would introduce some kind of bias to the model. My intuition tells me that this should not be problem since the sequence lengths for batches would vary greatly across training (since I am shuffling the index chunks), but I wanted to get a second opinion. I guess the question boils down to -&gt; is sequence length variation <strong>within the batch</strong> important for proper training? Or, for this specific case -&gt; is it important that the net process examples decomposed from a single caption <strong>within the same batch</strong> or close to each other time-wise?</p>
  <p>I'd be happy to train the network with both solutions and check the results myself but it's quite time-expensive and I'd prefer to avoid some stupid stuff if I can.</p>
  
Tags: <natural-language-processing><recurrent-neural-networks><pytorch><training-datasets><batch-learning>
Title: Batching together similar length sequences to avoid padding and packing
UserDisplayName: czypsu
ThreadText: # Batching together similar length sequences to avoid padding and packing
  
  (Asked by: username_0 on 2022-10-09)
  
  I am training an RNN in PyTorch to produce captions for images. It's a pretty standard architecture – the image is processed by a pre-trained InceptionV3 to extract features, the recurrent module processes the words seen so far and then its result is merged with image features and fed through a linear layer to produce a probability distribution over the vocab to predict the next word. A network trained this way can auto-regressively predict a whole caption for an image (it's seeded with an artificial  token at the beginning to have a preceding sequence to start off with).
  To train the network every caption is decomposed like this in the training dataset:
  X, y
  , a
   a, man
   a man, is
   a man is, sitting
   a man is sitting, on
   a man is sitting on, a
   a man is sitting on a, chair
   a man is sitting on a chair, 
  
  The result is that there is a lot of input sequence length variation in the training data which is a problem since we would need a lot of padding in batches. I am aware that there are techniques of counteracting this (pack_padded_sequence in PyTorch) but I came up with another idea.
  I am modifying the Sampler object that's used to provide indices for the DataLoader to construct batches in such a way that the Sampler orders all of the example indices by input sequence length, groups it into batch-size chunks, and shuffles the chunks before feeding them to the DataLoader. As a result the sequences in individual batches are almost always the same length – not because of any padding but because of the fact that we ordered the examples by length before grouping them. There will be some batches that will encompass examples on a border of length groups so some padding will be necessary but it will be minimal (just one zero for a group of lengths [5, 5, 5, 6, 6, 6] for example) and the number of such groups would be negligible (~0.002% of the batches in my case).
  My question is – does this method introduce some kind of drawback that I am not aware of? I find it simpler than having completely random batches and padding and packing them. I can't tell however, if it would introduce some kind of bias to the model. My intuition tells me that this should not be problem since the sequence lengths for batches would vary greatly across training (since I am shuffling the index chunks), but I wanted to get a second opinion. I guess the question boils down to -> is sequence length variation within the batch important for proper training? Or, for this specific case -> is it important that the net process examples decomposed from a single caption within the same batch or close to each other time-wise?
  I'd be happy to train the network with both solutions and check the results myself but it's quite time-expensive and I'd prefer to avoid some stupid stuff if I can.
  
  **Tags:** natural-language-processing, recurrent-neural-networks, pytorch, training-datasets, batch-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-02-21. Score: 1
  
  You can read that this was done in "[Attention is All You Need](https://arxiv.org/pdf/1706.03762.pdf)", for Transformers: "Sentence pairs were batched together by approximate sequence length.".
  
  But, with RNN you don't need any padding at all when using packedSequences, so I don't think your idea will be efficient, if I understand correctly.
  
  
  ---
  
  