Often you would divide the loss by the accumulation_steps so this point seems to be indeed missing in the linked post.



 duddal:

Also, should I still use gradient accumulation if I have BatchNorm2D Layers in my network?


You could try to use it, but note that the smaller the actual batch size is the more noise the running stats in batchnorm layers could get, which is not specific to gradient accumulation but a general limitation of small batch sizes in combination with batchnorm layers.