Solved by spanev in post #2 
              
              
                Hi @harshj94, 
In PyTorch, gradients are accumulating by default. 
So if you simply have this before optimizer.step: 
Loss1.backward()
Loss2.backward()

with no optimizer.zero_grad()  in between, 
you will end up with x.grad containing dLoss1/dx + dLoss2/dx. 
Also when you create the optimzer you ha…