What range are your inputs using at the moment?
Is the first iteration already creating the NaN outputs or after a couple of updates?
In the latter case, you could add torch.autograd.set_detect_anomaly(True) at the beginning of the script, which would point to the operation, which created the first NaN output.