Hello,
You need to explicitly reset the gradient between each backward pass with optimizer.zero_grad() or model.zero_grad(), else you are doing gradient accumulation. That’s the expected behavior and it has nothing to do with retain_graph. Are you sure you need retain_graph=True? Why do you need it?
retain_graph can be used, among other things, to backward multiple times the same loss, or to compute a backward pass on a loss computed on some gradient (for example, to compute gradient penalty in WGAN-GP models). In most case you won’t need it, and you don’t need it for gradient accumulation.
Best regards,
Thomas