Solved by Seo in post #2 
              
              
                I arrived at the conclusion that there is no need to re-adjust the loss value since averaging gradients will produce an effect that is the same as using the global batch size. 
This can be proven computing the loss for each local batch. This loss is sum(l)/batch_size, where l is the list containing …