Well it actually depend on the loss you are using but most of the time with use a CrossEntropy loss averaged over the samples in the batch (so it’s mostly independant of the batch size).
The natural extension of that to gradient accumulation is to average over the accumulation.
I wrote a bit about that some time ago here: https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255 56