I’m wondering if anyone has tried to do this. When shifting from train to val or vice versa, there is a pause in training as the active DataLoader changes and begins filling up it’s queue of batches (I’m talking particularly about using the torch.utils.data.DataLoader with num_workers > 1). When using large batches, this pause can be significant. Is there some way to have the next dataloader start queuing batches as soon as the previous one is finished buffering; even if the model hasn’t processed all those batches yet?