I think the naive solution will be the following :
First, you don’t sum the losses, but instead do this
optimizer.zero_grad()
loss1.backward()
### do something with these gradients, like
## print(self.Network[numlayer].weight.grad)
optimizer.zero_grad()
loss2.backward()
## do something with these gradients
## ...

Since we do backpropagation 2 times in the same step, it can slow down the step, but I’m not sure about that since we compute gradients separately, like, in out case d(loss)/dW = d(loss_1 + loss_2)/dW = d(loss_1)/dW + d(loss_2)/dW => autograd engine will compute these gradients separately too and the only overhead we’ll get is optimizer.zero_grad()?
Though, I’m not sure how to handle the case when you need to have access to both gradients at once
: thinking :