Even if you don’t do optimizer.step, if the pretrained model has a batch norm layer, those values will get updated when you are evaluating. If the pre-training was done on a large data, best not to change. Also if the model has a dropout layer, the predictions will be a bit random. Hence it is better to evaluate only after model.eval()