tumble-weed:

.grad.data.zero_()


Thanks a lot for your answer.
I made these changes according to your reply.
A.grad.data.zero_()
task1_loss.backward(retain_graph=True)
A_task1_grad = A.grad.clone()
A.grad.data.zero_()
task2_loss.backward(retain_graph=True)
A_task2_grad = A.grad