Solved by ptrblck in post #2 
              
              
                Did you store the complete dataset in a single tensor? 
If so, I think you might need to load it once and store smaller chunks of the data (and load only certain chunks in each process) or load the data lazily from the beginning.