If you don’t want the gradients to accumulate, then I’d recommend using the functional interface (torch.autograd.grad). Use this, compute the real loss you want to optimize based on the gradients, and run backward from this. There’s no other way to avoid the gradients from getting mixed up.