As I understand it, in order to ‘mask’ the input to an RNN (e.g. values of zero which simply represent padding of unequal length text to be uniform), one has to sort the sequence from longest to shortest and then this new sorted stacked sequence is sent into the RNN in mini batches.
Credit to : https://github.com/HarshTrivedi/packing-unpacking-pytorch-minimal-tutorial 16
Doesn’t this preclude you from shuffling your mini batches between epochs?
I am presuming that the full training set is packed prior to training. I tried to shuffle this packed data during training and an exception was raised that the data needed to be sorted.
Perhaps a work around is to pack each minibatch separately in the training loop? Not sure the impact this has on efficiency.