So the dataloaders are initialized when the iterator is created 1. You could look into creating the iterator right after loading your last batch of data with the other iterator.
The other question is whether it is worth the complexity you’d be introducing with it. As you note, this only affects the first batch of each iteration, so if you have a sizeable number of batches, it’ll contribute at most 1/number of batches to the total run time, and you get better leverage from considerably smaller optimizations that affect each batch (from where/when do I do augmentation or other processing to where/how do I store my data).
I’m saying this because in my experience it is very easy to be mislead by intuition when looking at optimizing the overall runtime. This bit is very visible during training (the progress bars aren’t ticking and all) but what kind of speedup of the overall time can you get from eliminating it?
Best regards
Thomas