I wonder whether there is a significant computation cost for those parameters I am not interested.
I have not got a time to dig into automatic differentiation.
For now torch.autograd.grad is not working in this situation.
If I set only_inputs = True (default), the gradient is not accumulated for all parameters and loss does not decrease.
If I set only_inputs = False, the gradient of parameters that I do not put into optimizer are also computed and accumulated because they are all leaf-nodes and have requires_grad = True.
For now I assume that I have to wait pytorch update attach()(or other similar method) to enable dynmaically optimizing selected features.
[https://github.com/pytorch/pytorch/pull/6561 8]