Model.zero_grad only fill the grad of parameters to 0
ypxie
smth
apaszke
Thanks for the explanations, but if I don’t fill other variable.grad.data to 0, will the grad of parameters that depended on those variables be wrongly estimated?  Keeping tracking of all the variables and setting their grad to 0 properly seems quite error prone.
ypxie
No, why would it matter? Gradient of parameters is not a function of the gradient w.r.t. some other Variable.
apaszke
Do you mean whatever the module is, as long as it’s inherited from base nn.Module class, simply calling the default model.zero_grad() is enough? I mean the error can pass from other Variable to the parameter,  if some Variable’s grad is unintentionally accumulated from the last run, the gradient of parameters may also be wrong.
ypxie
apaszke
<SCODE>    def __init__(self):
        self.out = torch.autograd.Variable(torch.zeros(timestep, batchsize, 
                                                       self.W_decode.size()[1]) , requires_grad=True)

    def forward(self, input, state=None):
        # input is timesetep*N
        batchsize, timestep = input.size()[1], input.size()[0]
        vec_input = input.view(-1)
        emb = torch.index_select(self.W_emb, 0, vec_input).view(timestep,batchsize, -1) # emb = N*ninp
        inp = matmul(emb, self.W_rnn)
        state = torch.autograd.Variable( torch.zeros(inp.size()[1:])) if state is None else state
        for step in range(inp.size()[0]):
            this_input = inp[step]  # N * nhid
            this_input = torch.addmm(this_input, state, self.U_rnn)
            state = F.tanh(this_input  + self.b_rnn.expand_as(this_input) )
            self.out[step] = torch.addmm(self.out[step], state, self.W_decode)
            self.out[step] = F.softmax(self.out[step]  + self.b_decode.expand_as(out[step]) )
        return self.out
<ECODE>
ypxie
<SCODE>def __init__(self):
    self.out = get_new_out()

def forward(self, input):
    ...
    next_out = get_new_out()
    for step in range(inp.size(0)):
        next_out[step] = ... # an expression containing self.out
    self.out = next_out
    return next_out
<ECODE>
apaszke
ypxie
No, you’re not going to feel any difference in speed. CPU allocations are fast and we have a custom CUDA allocator that caches the memory, so it’s also very fast. I don’t know how you define the safety of Module.zero_grad - it does what it’s meant to do, i.e. zero the grad of parameters.
apaszke
Also, as I said, if you don’t reallocate the output your graphs will never be freed and that will blow up the memory. Don’t cache intermediate things too long, PyTorch philosophy is quite different from Lua torch.
apaszke
<SCODE>from torch.autograd import Variable
x = Variable(torch.ones(2, 2), requires_grad = True)
y = x  + 2 
z = y * y * 3
out = z.mean()
out.backward(retain_variables=True)
print(x.grad)
<ECODE> I get <SCODE>Variable containing:
 4.5000  4.5000
 4.5000  4.5000
[torch.FloatTensor of size 2x2]
<ECODE> <SCODE>Variable containing:
 0  0
 0  0
[torch.FloatTensor of size 2x2]
<ECODE>
ypxie
fmassa
ypxie
smth
Many thanks, very helpful!
ypxie
apaszke