Sorry, I wasn’t clear enough, I meant it lower it of 3~5 second (~35/37s instead of ~40s if pin_memory is True).
Anyway, I resolved the issue. It wasn’t from the Pytorch code but from the way my dataset worked.
I was reading from disk every batch but in my case, it slowed down the algorithm too much as I was using a small training set (~1200).
Once the dataset loaded in memory, I have actually ~1s per epoch.
Anyway, thank you for your help.