I use Alexnet and MNIST dataset, loss is BCEWithLogitsLoss, reduction is SUM, no optimizer operations. I use the same data samples to calculate loss and gradient of two same model. Use 32 samples to get loss and loss.backward(), the sum of loss is 248.0272. Then I split these samples into multi batches and use them to calculate loss and loss.backward() sequentially, the sum of loss is 245.8620. These two loss without reduction is still different on each class.