No the grads of the part of the net not in subnet are not zero. Actually, new_loss takes all the grad values from all the params but calc grad of only those which occur in subnet.
But yeah right now the only difference is just the learning rate. But I might want to use an altogether different optimizer (SGD) for it. But for now I can just use your method. Does it work for any optimizer?
Also torch.autograd.grad() does not accumulate the grads in the .grad attribute of the input variables. But as you see I want them to accumulate or how else would I use them?