I don’t believe gradient accumulation can cause the problem. It does take some extra memory to maintain the gradients and intermediate values, but this memory should be reused in the next iteration of the model.
Just to make sure, I repeated the test with your change and see that the memory usage still increases.