Solved by ptrblck in post #2 
              
              
                You can get all gradients by iterating the parameters after the backward operation: 
for name, param in model.named_parameters():
    print(name, param.grad)

Using this approach you can store the gradient magnitude and plot it for each iteration using e.g. matplotlib. 
To get the learning rate and …