Solved by ptrblck in post #2 
              
              
                If your use case allows to manipulate the dataset after a full epoch, I would recommend to perform the manipulations directly on the data inside the Dataset and just recreate the DataLoader. 
Instantiation of DataLoaders should be cheap, so you shouldn’t see any slow down. 
Also, manipulating the da…