Solved by ZimoNitrome in post #2 
              
              
                Have you tried different num_workers for the DataLoader? 
num_workers=0 has been the most stable one from personal experience. 
Otherwise I don’t see anything wrong with your code. Maybe the model is large and simply performs very slowly on CPU.