Quick Detach() Question
If I call detach() on a variable after calling backward(retain_variables=True) on a variable further along the graph, will that free the buffers of the graph that led to the detach()'d variable? Example: net1 and net2 are modules, x is an input variable. I want to retain the state of net2 for processing later down the line, but I just want to accumulate gradients in net1 for the time being, no need to retain the graph. out1 = net1(x) out2 = net2(out1) loss = some_loss_function(out2) loss.backward(retain_variables=True) out1.detach_() Will this clear the buffers of net1? Thanks, Andy
ajbrock
Hi, <SCODE>out1 = net1(x)

in2 = Variable(out1.data, requires_grad=True)
out2 = net2(in2)

loss = some_loss_function(out2)

loss.backward(retain_variables=True)
out1.backward(in2.grad.data)

# You can call loss.backward again here
# But not out1.backward !
<ECODE>
albanD
Hmm, assuming detach() doesn’t free the graph, your way makes a lot of sense, and I think I could wrap that up into this chunk of code neatly. Thanks! Edit: a slightly more elegant way to do this (still following your same idea) might be to copy out1 to in2 with the out-of-place detach: in2 = out1.detach() I’ll try out both and report back.
ajbrock
If in2 is detached, you can’t call loss.backward at all, since net2(in2) are not taken into account by the computation of the gradient. When a variable is detached, the backward computations will not visit the branches that start from this variable (all the operations done on it).
alexis-jacq
apaszke
<SCODE>import torch
x = torch.randn(10, 5, requires_grad = True)
w = x * x
a = w*w

a.sum().backward()
print(x.grad.sum())

w = x * x
a = w * w
w.detach_()

a.sum().backward()
print(x.grad.sum())
<ECODE> in this case, the output is: <SCODE>tensor(27.7943)
tensor(55.5887)
<ECODE>
111319
albanD
Thanks for replay. But why sometimes inplace operator cause a error in backward and sometimes not?
111319
Which error do you sometimes see and sometimes not?
albanD
Ou, I mean there are some inplace operators will cause an  error in backward, and some other operators will not. Back to the original question, is there a method that can detach a sub graph after a forward?
111319
No there isn’t I’m afraid. We don’t allow modifying the graph.
albanD