Thanks, that was helpful. Now I know that this essentially expected behavior. The thread you linked is also very helpful, although so far I haven’t been able to get DistributedSampler to work correctly.
Having looked into it more, I think I’m going to pursue a different solution for now, where I just don’t load the whole dataset into RAM, but instead load batches from the hard drive when necessary. I figured pre-loading all the data into RAM would provide a big speedup, but having thought about it more and based on advice from others, I now think this intuition is not correct, since using multiple workers allows upcoming batches to be pre-loaded into RAM anyway.