Thanks for the reply. But can it have a more elegant solution? By doing above solution, I have to do two forward and two backward computation for the gradient modified sub-network, which is very inefficient. Can I have a way to control how the autograd engine do backward propagation?