I also had an issue with nan and it was caused by my error function (the one on which I backward) containing torch.sqrt (I wanted to measure my error in Euclidean distance). After removing that part (so now I use classical MSE) it all works without problems.
I have to point out though that I had the very same code/algorithm implemented in TensorFlow, with sqrt included, and it worked without any nan problems.