I would suggest you to look at the gradient at the previous step. In my case, the gradient were exploding in the previous iteration of training due to some normalization step i was doing after the svd. This made the input ‘Nan’, which causes this error when Nan goes into torch.svd.
So, what you can do is check the the value of gradients of variables which are output of svd, if they are becoming very large, You can look for gradients using backward hook. Also look for places, where you are dividing by some variable which can become close to zero (if there are any such variables, just add an epsilon to it)