as a check, set the model in the validation script in train mode (net.train() ) instead of net.eval(). If the loss does NOT go up, then the problem is most likely batchNorm. This happens more than anyone would think.