Dummy code:

v = bottom(x)
v.retain_grad()
opt = t.optim.SGD([v],1e-3)
for i in range(10):
opt.zero_grad()
top(v).backward([v])
opt.step()

(I’m only optimizing over v here nothing else)
This kind of setup produces the error: can’t optimize a non-leaf variable. which is correct as v is connected from bottom to the top module. However I do not see why this is a valid reason for not being allowed to optimize over v.