I’d also be interested in the answers. As you say, there are GPU exhaustion drops after every epoch that I haven’t been able to get around. What is it that takes time here? The sampler, spinning up new workers or what?
There is only ever a significant performance hit for small dataset where one have to shuffle/resample often. Could I ask how quickly you go through an epoch?



 Carsten_Ditzel:

Then there is the option of saving the data via torch.save() prior to the training process. Would that be beneficial under certain circumstances?


I would say no. The only use case I see is if you want to do pre-processing that takes a lot of time and wouldn’t be necessary every epoch, but can’t be done before training.
Just to be clear, are you having GPU drops after the epochs or also between batches? You are running several workers right?