Hello,
I am exploring a RL agent and want to keep gradients of previous iterations in the following way:
grad_current_iter = grad_from_loss_current_iter + grad_from_loss_iter_-1 + grad_from_loss_iter_-2 …
reset every N iter
Would it be just enough to deactivate optimizer.zero_grad():
loss = obtain_loss()
if self.iteration % 10:
    optimizer.zero_grad()
loss.backward()
self.optimizer.step()