With num_workers=0 I occasionally get an error after a long time which lead me to discover that some of the images were 2048x4096 (original images actually were 256x512, they were just cropped to 224x224). I created a new dataset with the downsampled images to 256x512 and that seemed to fix the issue. Thank you for the help