As it seems to be the same iteration where the nan happens, could it be something in your batch? Have you tried shuffling your data from your dataloader? Also, one thing I noticed is that the output Tensor on iteration 13 is all the same. Could this potentially be an issue?