When I use torch.autograd.set_detect_anomaly(True) in train function  after optim step, then it shows the following traces
Sometimes first few batches run smoothly, but it starts suddenly by giving NaN values in the Weights(in Kernels and biases).
I checked the input data and my input data is fine as it doesn’t contain any NaN Vlaue after computing MFCC features
Model contains NaN after optim step
Note: When I tried to replace ReLU with Tanh, it works fine somehow but after some iterations (>50), it starts to give NaN values again. When I tried to analyse the weights, they don’t change. I am confused
loss.backward()
(Pdb) n
Warning: Error detected in NativeBatchNormBackward. Traceback of forward call that caused the error:
  File "Code/main.py", line 178, in <module>
    main()
  File "Code/main.py", line 134, in main
    train_avg_loss, correct_train, total_train = initiate_train_val.train(model, criterion, optim, epoch, train_stutter_dataset, lr, device)
  File "/home/shsheikh/PhD/audio_modality/dysfluencydetectionaudiomodality/Code/train.py", line 89, in train
    output = model(batch)                # Forward Pass
  File "/home/shsheikh/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/shsheikh/PhD/audio_modality/dysfluencydetectionaudiomodality/Code/tdnn_debug.py", line 143, in forward
    input = self.bn8(input)
  File "/home/shsheikh/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/shsheikh/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/nn/modules/batchnorm.py", line 106, in forward
    exponential_average_factor, self.eps)
  File "/home/shsheikh/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/nn/functional.py", line 1923, in batch_norm
    training, momentum, eps, torch.backends.cudnn.enabled
 (print_stack at /opt/conda/conda-bld/pytorch_1591914985702/work/torch/csrc/autograd/python_anomaly_mode.cpp:60)
RuntimeError: Function 'NativeBatchNormBackward' returned nan values in its 0th output.
> /home/shsheikh/PhD/audio_modality/dysfluencydetectionaudiomodality/Code/train.py(97)train()

#The Second one 
Warning: Error detected in LogSoftmaxBackward. Traceback of forward call that caused the error:
  File "Code/main.py", line 178, in <module>
    main()
  File "Code/main.py", line 134, in main
    train_avg_loss, correct_train, total_train = initiate_train_val.train(model, criterion, optim, epoch, train_stutter_dataset, lr, device)
  File "/home/shsheikh/PhD/audio_modality/dysfluencydetectionaudiomodality/Code/train.py", line 92, in train
    loss = criterion(output, label)      #Loss
  File "/home/shsheikh/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/shsheikh/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/nn/modules/loss.py", line 932, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/home/shsheikh/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/nn/functional.py", line 2317, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/shsheikh/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/nn/functional.py", line 1535, in log_softmax
    ret = input.log_softmax(dim)
 (print_stack at /opt/conda/conda-bld/pytorch_1591914985702/work/torch/csrc/autograd/python_anomaly_mode.cpp:60)
Traceback (most recent call last):
  File "Code/main.py", line 178, in <module>
    main()
  File "Code/main.py", line 134, in main
    train_avg_loss, correct_train, total_train = initiate_train_val.train(model, criterion, optim, epoch, train_stutter_dataset, lr, device)
  File "/home/shsheikh/PhD/audio_modality/dysfluencydetectionaudiomodality/Code/train.py", line 97, in train
    loss.backward()
  File "/home/shsheikh/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/shsheikh/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/autograd/__init__.py", line 100, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Function 'LogSoftmaxBackward' returned nan values in its 0th output.