The gradients will be accumulated for all batches in your train_loader, while the optimizer performs a step after accum steps.
Usually you would zero out the gradients after calling optimizer.step().
It’s interesting to hear it’s working. Did you compare it with other approaches (e.g. zeroing out the gradients after optimizer.step())?