Register_backward_hook on nn.Sequential
<SCODE>import torch
import torch.nn as nn
from torch.autograd import Variable

a = nn.Sequential(nn.Linear(5,3), nn.Tanh(), nn.Linear(3,2))

def hookFunc(module, gradInput, gradOutput):
	print(len(gradInput))
	for v in gradInput:
		print v
a.register_backward_hook(hookFunc)

input = Variable(torch.randn(4,5))
output = a(input)

target = torch.FloatTensor(4,2).fill_(1)
output.backward(target)
<ECODE> <SCODE>Variable containing:
-0.5910 -0.7340 -0.4239
-0.5910 -0.7340 -0.4239
[torch.FloatTensor of size 2x3]

Variable containing:
 4
 4
[torch.FloatTensor of size 2]
<ECODE>
supakjk
Yes, you should never modify any arguments given to the hook in-place. If you want to replace grad input, you can do out-of-place operations on it, and return new values from the hook.
apaszke
supakjk
Like that: <SCODE>x = Variable(torch.randn(5, 5), requires_grad=True)
y = x + 2
y.register_hook(lambda grad: grad * 2)
y.sum().backward()
x.grad # is now filled with 2
<ECODE>
apaszke
But remember about that container hook problem. Do this only on primitive modules or Variables.
apaszke
Thanks!
supakjk
It works in the same way for modules: <SCODE>module.register_backward_hook(lambda module, grad_i, grad_o: grad_i * -1)
<ECODE>
apaszke
supakjk
Qizhe_Xie
That approach worked in my case.
supakjk
WERush
<SCODE>class DDReLU(nn.Module):
    def __init__(self):
        super(DDReLU, self).__init__()
        self.threshold = nn.Parameter(torch.rand(1), requires_grad=True)
        self.register_backward_hook(lambda module, grad_i, grad_o: (grad_i[0], grad_i[1]*0.01))
        #self.threshold.data.fill_(0.1)
        self.ReLU = nn.ReLU(True)

    def forward(self, x):
        print(self.threshold.data[0])
        return self.ReLU(x + self.threshold) - self.threshold
        #return self.ReLU(x) + self.threshold
<ECODE> Is the code above fine to change the relative learning rate of the new parameter? By relative learning rate, I mean: The parameter created has a learning rate that is 0.01 times the one used to the other model’s parameters.
dlmacedo