Output for torch.max(abs(param.grad))
Model Parameters Gradients conv1d_tdnn1.kernel tensor(0.0776)
Model Parameters Gradients conv1d_tdnn1.bias tensor(0.0294)
Model Parameters Gradients bn1.weight tensor(0.0034)
Model Parameters Gradients bn1.bias tensor(0.0023)
Model Parameters Gradients conv1d_tdnn2.kernel tensor(0.0044)
Model Parameters Gradients conv1d_tdnn2.bias tensor(0.0018)
Model Parameters Gradients bn2.weight tensor(0.0022)
Model Parameters Gradients bn2.bias tensor(0.0007)
Traceback (most recent call last):
  File "Code/main.py", line 180, in <module>
    main()
  File "Code/main.py", line 136, in main
    train_avg_loss, correct_train, total_train = initiate_train_val.train(model, criterion, optim, epoch, train_stutter_dataset, lr, device)
  File "/home/shsheikh/PhD/audio_modality/dysfluencydetectionaudiomodality/Code/train.py", line 106, in train
    print("Model Parameters Gradients",name,torch.max(abs(param.grad)))
TypeError: bad operand type for abs(): 'NoneType'

Output for loss.data and loss.grad
loss.data: >>>>>>>>>>>>>>>>>>>>>  tensor(1.6281)
loss.grad: >>>>>>>>>>>>>>>>>>>>>  None
Model Parameters Gradients conv1d_tdnn1.kernel tensor(nan)
Model Parameters Gradients conv1d_tdnn1.bias tensor(nan)
Model Parameters Gradients bn1.weight tensor(nan)
Model Parameters Gradients bn1.bias tensor(nan)
Model Parameters Gradients conv1d_tdnn2.kernel tensor(nan)
Model Parameters Gradients conv1d_tdnn2.bias tensor(nan)
Model Parameters Gradients bn2.weight tensor(nan)
Model Parameters Gradients bn2.bias tensor(nan)
Traceback (most recent call last):
  File "Code/main.py", line 180, in <module>
    main()
  File "Code/main.py", line 136, in main
    train_avg_loss, correct_train, total_train = initiate_train_val.train(model, criterion, optim, epoch, train_stutter_dataset, lr, device)
  File "/home/shsheikh/PhD/audio_modality/dysfluencydetectionaudiomodality/Code/train.py", line 108, in train
    print("Model Parameters Gradients",name,torch.max(abs(param.grad)))
TypeError: bad operand type for abs(): 'NoneType'


When I print the same with Tanh, it is still giving the Nan Values
=============TOTAL PARAMETERS================== 660469
Model Parameters Gradients conv1d_tdnn1.kernel tensor(0.)
Model Parameters Gradients conv1d_tdnn1.bias tensor(0.)
Model Parameters Gradients bn1.weight tensor(0.)
Model Parameters Gradients bn1.bias tensor(0.)
Model Parameters Gradients conv1d_tdnn2.kernel tensor(0.)
Model Parameters Gradients conv1d_tdnn2.bias tensor(0.)
Model Parameters Gradients bn2.weight tensor(0.)
Model Parameters Gradients bn2.bias tensor(0.)
Model Parameters Gradients conv1d_tdnn3.kernel tensor(0.)
Model Parameters Gradients conv1d_tdnn3.bias tensor(0.)
Model Parameters Gradients bn3.weight tensor(0.)
Model Parameters Gradients bn3.bias tensor(0.)
Model Parameters Gradients conv1d_tdnn4.kernel tensor(0.)
Model Parameters Gradients conv1d_tdnn4.bias tensor(0.)
Model Parameters Gradients bn4.weight tensor(0.)
Model Parameters Gradients bn4.bias tensor(0.)
Model Parameters Gradients conv1d_tdnn5.kernel tensor(0.)
Model Parameters Gradients conv1d_tdnn5.bias tensor(0.)
Model Parameters Gradients bn5.weight tensor(0.)
Model Parameters Gradients bn5.bias tensor(0.)
Model Parameters Gradients conv1d_tdnn6.kernel tensor(0.)
Model Parameters Gradients conv1d_tdnn6.bias tensor(0.)
Model Parameters Gradients bn6.weight tensor(0.)
Model Parameters Gradients bn6.bias tensor(0.)
Model Parameters Gradients conv1d_tdnn7.kernel tensor(0.)
Model Parameters Gradients conv1d_tdnn7.bias tensor(0.)
Model Parameters Gradients bn7.weight tensor(0.)
Model Parameters Gradients bn7.bias tensor(0.)
Model Parameters Gradients conv1d_tdnn8.kernel tensor(nan)
Model Parameters Gradients conv1d_tdnn8.bias tensor(nan)
Model Parameters Gradients bn8.weight tensor(nan)
Model Parameters Gradients bn8.bias tensor(nan)
Model Parameters Gradients conv1d1.weight tensor(nan)
Model Parameters Gradients conv1d1.bias tensor(nan)
Model Parameters Gradients bn9.weight tensor(nan)
Model Parameters Gradients bn9.bias tensor(nan)
Model Parameters Gradients conv1d3.weight tensor(nan)
Model Parameters Gradients conv1d3.bias tensor(nan)
Model Parameters Gradients bn11.weight tensor(nan)
Model Parameters Gradients bn11.bias tensor(nan)
Model Parameters Gradients conv1d2.weight tensor(nan)
Model Parameters Gradients conv1d2.bias tensor(nan)
Model Parameters Gradients bn10.weight tensor(nan)
Model Parameters Gradients bn10.bias tensor(nan)
Model Parameters Gradients fc1.weight tensor(4.6971e-09)
Model Parameters Gradients fc1.bias tensor(0.0470)
[Gradients ====>]
Kernel and bias after tdnn layers conv Parameter containing:
tensor([[[nan, nan],
         [nan, nan],
         [nan, nan],
         ...,
         [nan, nan],
         [nan, nan],
         [nan, nan]],

        [[nan, nan],
         [nan, nan],
         [nan, nan],
         ...,
         [nan, nan],
         [nan, nan],
         [nan, nan]],

        [[nan, nan],
         [nan, nan],
         [nan, nan],
         ...,
         [nan, nan],
         [nan, nan],
         [nan, nan]],

        ...,

        [[nan, nan],
         [nan, nan],
         [nan, nan],
         ...,
         [nan, nan],
         [nan, nan],
         [nan, nan]],

        [[nan, nan],
         [nan, nan],
         [nan, nan],
         ...,
         [nan, nan],
         [nan, nan],
         [nan, nan]],

        [[nan, nan],
         [nan, nan],
         [nan, nan],
         ...,
         [nan, nan],
         [nan, nan],
         [nan, nan]]], requires_grad=True) Parameter containing:
tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
       requires_grad=True)

@ptrblck