Using model.eval sets all nn.Modules to evaluation, i.e. some layers like BatchNorm and Dropout change their operation.
BatchNorm layers for example don’t calculate the batch statistics anymore and thus don’t update the running stats. They just apply the already calculated running stats on the test sample.
If you call your model with some test samples using a learning rate of 0, the running stats in each BatchNorm layer will be updated using the test set. This might yield better results, if your data domains of training and test differ.
However, I wouldn’t recommend this approach for a model using some kind of accuracy metrics, since it’s data leaking in my opinion. It’s not really bad for a generative model in my opinion, but that’s probably another topic. Maybe you could play around with the momentum argument of BatchNorm to counter this effect.
Also, what is your batch size? It is usually advised to use a batch size of >=64.
If you have a smaller batch size, you might want to have a look at nn.GroupNorm (in master) or nn.InstanceNorm.