Solved by albanD in post #4 
              
              
                If you’re using momentum and/or weight decay (or another optimizer) it won’t work. 
For example SGD+weight decay (with learning rate lr and weight_decay wd will do the following update: w = w - lr * (dL/dw + wd*w). You can see that scaling your loss: dL/dw -> 1/n*dL/dw will not have the same effect …