Well, I assume there’s no NaN in the dataset  I don’t know about this being a Pytorch bug… Maybe you could snapshot the state of the model and the input data just before the gradients go NaN, and see if some particular numbers cause the problem for some reason.