Pytorch seems to only aggregate (accumulate) gradients on the same layer.
optimizer.zero_grad()
loss1.backward(retain_graph=True)
gradient_1 = get_layer_gradient(model, layer_name='common_layer')
optimizer.zero_grad()
loss2.backward(retain_graph=True)
gradient_2 = get_layer_gradient(model, layer_name='common_layer')
optimizer.zero_grad()
loss3.backward(retain_graph=True)
gradient_3 = get_layer_gradient(model, layer_name='common_layer')

optimizer.zero_grad()
loss = loss1 + loss2 + loss3
loss.backward()
gradient_4 = get_layer_gradient(model, layer_name='common_layer')

I found that gradient_1 + gradient_2 + gradient_3 != gradient_4. Why?
How can I get the exact three gradients by the three losses, which should add up to gradient_4 ?
Feel free to discuss!