Thanks for your reply! BN and Dropout can change the output of the layer, so I get different loss value. In my observation code, the sum of loss is used to measure the loss and a full batch and the loss across the small batches, no gradient contribution. You remind me that opt.zero_grad() is used for avoiding accumulating grads!. In my experiment now,  the absolute error of grads is caused by the limit of floating point precision. When the reduction of BCEWithLogitsLoss is mean, the average error is around e-10, and the error is around e-8 when the reduction is sum.