silvester:

grad(loss) = grad(loss0) + grad(loss1)


As you said here, gradients are independent I suppose.
So, I am not sure how to debug this. I do not know if the advanced optimizers (Like Adam) consider the parameters as independent entities or if they perform any operations to normalize the gradients somehow.
Did you try with simple SGD?