Hi, I tried torch.allclose(grad1+grad2+grad3, grad4) and it returns True. That means the two gradient matrices are element-wise equal within a tolerance. The printed values of them are the same but they just do not equal each other.
So, how to explain this? Why the addition of gradient_1 + gradient_2 + gradient_3 is slightly different from using loss = loss1 + loss2 + loss3 to do backpropagation at a time? shouldn’t they differentiate on the same graph three times and accumulate (add) gradients to be gradient_4?