I want to calc grads using one loss Variable (lets say, loss1) for only a part of the network and accumulate it in the .grad attribute of the respective variables that come in that sub-network.
The rest of the network does not take grads from that loss Variable (loss1) but it has grads from some other loss Variable (loss2).
torch.autograd.grad() does not accumulate the grads in the .grad attribute of the input variables. How do I make them accumulate?