One thing that we didn’t quite clear up from earlier was how exactly to .detach() the parameters of a neural network. Would I have to create a copy of the neural network somehow? I think as soon as we clear that up, I could implement something like what you first proposed. Or were you suggesting detaching the output of the network instead of it’s parameters?
I should add that, as a newcomer to Torch, I’m not exactly sure what .detach() does. My interpretation of the docs is that it acts something like tf.stop_gradient in Tensorflow, preventing gradients from flowing through the variable. But if that is the case, then what does calling x.detach().requires_grad_() do? Does this essentially create a copy of the x node in the graph with all the same in- and out-edges? And what is the performance benefit to doing this?