I might have misunderstood the issue, but in the first post you claim, that you tried to "remove the NaNs in the input to torch.svd()".
Based on your description now it seems that you are suddently getting NaNs in the model output, if you are trying to apply svd?