it stores the gradient after each backward pass. so shouldn’t be out of sync. it hooks after each forward pass. You can double check by setting the parameters to be zero, you should get zero grads for that iteration.