How to loop over all the variable in a nn.module
and register hooks to the gradient?
ypxie
Thanks for your reply. Do you know how to get name  or meaningful identifier of variable?  I did not find a name field in variable.
ypxie
Thanks for your reply. But this seems to only work on layer level? <SCODE>def inves(name=''):
    def f(tensor):
        if np.isnan(torch.mean(tensor).data.cpu().numpy() ):
            print('gradient of {} is'.format(name))
            print(tensor)
            assert 0, 'nan gradient'
    return f

for key, var in Model.varibles().iteritems():
    var.register_hook(inves(key))
<ECODE>
ypxie
update:  I fond using locals() can be a solution.
ypxie
You could try that: <SCODE>def register_nan_checks(model):
    def check_grad(module, grad_input, grad_output):
        if np.isnan(grad_input.data.numpy()):
            print('NaN gradient in ' + type(module).__name__)
    model.apply(lambda module: module.register_backward_hook(check_grad))
<ECODE>
apaszke
ypxie
<SCODE>def register_nan_checks(model):
    def check_grad(module, grad_input, grad_output):
        # print(module) you can add this to see that the hook is called
        if any(np.all(np.isnan(gi.data.numpy())) for gi in grad_input if gi is not None):
            print('NaN gradient in ' + type(module).__name__)
    model.apply(lambda module: module.register_backward_hook(check_grad))
<ECODE> I’ve just checked and it works for me. If I add an additional print, it will show all modules.
apaszke
ypxie
Can you show me the code? It works for me
apaszke
ypxie
apaszke
Thanks, it works now!
ypxie
ypxie
fmassa
ypxie
fmassa
Thanks for your comments. Currently, I have no clue which part goes wrong.
ypxie
Why it’s not suggested to directly use forward, isn’t it just building a graph?
ruotianluo
apaszke
I dont understand what the issue is, isn’t <SCODE>    for W in mdl_sgd.parameters():
        W.data = W.data - eta*W.grad.data
<ECODE> what u do to loop over? (also it seems ur title doesn’t reflect ur real question I suggest to update it, its confusing and unnecessary).
Brando_Miranda