Merry Christmas everyone!
Over the past weeks I struggled with a similar problem and invested a lot of nerves and coputation time into it. Things that I found out and that might help others:

If you calculate a batch-wise mean loss (e.g. MSE) be careful how to calculate the mean of all batches. If your batches are not all the same size (use drop_last in the DataLoader to make sure your last batch is as well), you need to weight your mean.
If your in eval() mode, be careful if you print any summarys with external libraries like torch-summary or torchsummary. There is / was a bug concerning the order of the summary and eval() statements, I believe it’s fixed now in torch-summary.

(I worked with Resnet18 and ResNeXt50)