The code looks generally alright. Since the updates per iteration seem to work, could you run a test by eg. reusing the same sample 10x in the gradient accumulation loop and check the parameter update afterwards. Based on your code snippet the update should be equal to a single update step (without gradient accumulation).