The loss is averaged by default using a criterion, e.g. nn.CrossEntropyLoss or nn.MSELoss. This would make the gradients independent regarding the batch size. You could change this behavior by setting the reduction argument to e.g. 'sum'.