I’m splitting my network into two parts, top and bottom. Let’s call the resulting activation map v=bottom(x). I would like to optimize over L=top(v) treating the activation map v as weights. By default PyTorch does not retain gradients for activation maps, so I used v.retain_grad() in order to preserve the gradient. However when I optimize over the top part of the network I get the error: can’t optimize a non-leaf variable.
However this is exactly what I would like to do, why is PyTorch stopping me? And what can I do about it?