Asa-Nisi-Masa:

There is no overfitting happening because there is no loss defined when doing just the inference, and no gradient descent being done.


Oh my bad, I considered same code for training as your original post was about swaping between model.eval and model.train. But using torch.no_grad will save a lot of time and memory!