Well, it depends on the loss function you use right? All autograd does is just to calculate the gradient, it has no notion of batching, and I don’t see how it can have different behavior with different batching mechanism. If you look at the doc of the loss function here (http://pytorch.org/docs/master/nn.html#torch.nn.functional.nll_loss 747), you will notice that size_average=True in your case. It means that the loss is averaged across the batch.