Have you tried different num_workers for the DataLoader?
num_workers=0 has been the most stable one from personal experience.
Otherwise I don’t see anything wrong with your code. Maybe the model is large and simply performs very slowly on CPU.