Since you’ve isolated a possible reason for the difference in losses to the last batchnorm layer, it would be interesting to see the stats (min, max, mean, std) of the output of this particular bn layer using 1) a training batch 2) swithcing to eval() and using a validation batch.