With torchtext 0.9.0, BucketIterator was depreciated and DataLoader is encouraged to be used instead, which is great since DataLoader is compatible with DistributedSampler 3 and hence DDP. However, it has a downside of not having the out-of-the-box implementation of having batches of similar length. The migration tutorial 1 recommends using batch_sampler argument of DataLoader to pool together batches of similar length. Unfortunately, the batch_sampler is not compatible with the sampler.
Does anyone have any suggestions/ideas on how to make sure that when using DDP, the batches are of as similar length as possible?