If you do not use  torch.no_grad() , gradients in pretrained model will be updated and the result will be overfitted on your test set.

I’m not sure I completely follow. If I am using a pretrained model and make predictions without torch.no_grad(), then even if the gradients are computed, no weights are actually being updated. There is no overfitting happening because there is no loss defined when doing just the inference, and no gradient descent being done.