Usually this should not be the case.
It might happen, for the reason I explained in my first answer:



 ptrblck:

If you call your model with some test samples using a learning rate of 0, the running stats in each BatchNorm layer will be updated using the test set. This might yield better results, if your data domains of training and test differ.


However, if you don’t have any BatchNorm layers, it would be interesting so see a small code snippet to help debugging this issue.