Hi,
I’m encountering a similar issue.
I’m getting NaN after backproping the first iteration.
The model uses multiple losses.
But, I narrowed down the problem to kl_divergence.
When I’m backpropping torch.mean(self.kl_divergence(dist0,dist1)), I’m getting the following message from the anomaly detection:-
File “/home/schatter/Soumick/Code/DS6/pipeline.py”, line 262, in train
self.scaler.scale(floss[i]).backward(retain_graph=True)
File “/home/schatter/anaconda3/envs/torchMRI17/lib/python3.8/site-packages/torch/tensor.py”, line 221, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File “/home/schatter/anaconda3/envs/torchMRI17/lib/python3.8/site-packages/torch/autograd/init.py”, line 130, in backward
Variable._execution_engine.run_backward(
RuntimeError: Function ‘CudnnConvolutionBackward’ returned nan values in its 0th output.
Without AMP, its working fine.
Its only showing up when I’m using AMP.
Any ideas?
Thanks.